Rosenverse

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Journeying toward AI-assisted documentation in healthcare

Gold
Wednesday, June 5, 2024 • Designing with AI 2024
Share the love for this talk
Journeying toward AI-assisted documentation in healthcare
Speakers: Jennifer Kong
Link:

Summary

Documentation technology is the foundation of modern healthcare delivery. Convoluted, redundant, and excessive documentation is a pervasive problem that causes inefficiency in all aspects of the industry. At IncludedHealth, we are developing an AI-assisted documentation that summarizes and documents conversations between patients and their care providers. A care provider can push one button and have their entire patient encounter captured in a succinct and standardized format. Upon a pilot launch, the results were staggering. Within 6 months, we demonstrated a 64% reduction in time per encounter! However, despite our promising results, there still remain challenges specific to the demands of the healthcare domain. As our team continues to develop solutions to meet these challenges, we gain even more clarity on what it takes to design a human-backed, AI-powered healthcare system. Takeaways From this session, you can expect to learn the following: Developing AI design in healthcare requires close collaboration between end users and your data science team Piloting GenAI solutions may be more effective than traditional prototyping Trading accuracy for efficiency is a barrier to adopting GenAI tools in healthcare GenAI design in healthcare requires establishing critical boundaries as well as a good understanding of cognitive processing Other factors to consider when designing AI solutions for service-based industries are understanding how training might be impacted, the importance of standardization vs. personalization of data output and the need for more autonomy and control elements due to consequences of unpredictable output errors

Key Insights

  • Generative AI can reduce healthcare documentation time by over 60% but doesn't eliminate the need for manual editing and human review.

  • Focusing AI tools initially on low-risk use cases like chat encounter summaries mitigates potential clinical harm.

  • Large Language Models (LLMs) excel at summarizing text but struggle with capturing exact clinical details and non-verbal cues.

  • Designing AI tools requires managing user expectations to prevent disappointment and frustration over imperfect outputs.

  • A simple one-button UI with options to regenerate and manually edit notes improves user workflow efficiency and error recovery.

  • Implicit metrics such as the 'edit rate'—the fraction of user edits on AI-generated text—help monitor AI output quality unobtrusively.

  • Traditional prototyping methods are less effective for AI; incremental live pilots provide critical learning about AI's unpredictable outputs.

  • Operational factors like quality assurance metrics impact user attitudes when AI-generated notes lower scores despite time savings.

  • Human-centered collaboration between designers, data scientists, and users is essential to shape AI capabilities and discover unforeseen problems.

  • AI models must remain static (non-learning) in healthcare to comply with regulations, creating challenges for ongoing improvements.

Notable Quotes

"We saw documentation time reduced by 64% after six months of using the AI tool."

"Documentation is important for liability and useful for patient handoffs, so notes must be concise yet detailed."

"Healthcare is not a silver bullet for AI; a lot of context comes from non-verbal cues where generative AI doesn’t apply."

"The unpredictable black box nature of LLMs means we had to zoom in on small problems first before seeing the bigger picture."

"We emphasized a simple one-button workflow with manual editing and regeneration to handle inevitable AI errors."

"The edit rate, our metric of human-added characters over total characters, tracks AI output quality without burdening users."

"Users were initially excited but six months later expressed frustration and uncertainty about the tool’s usefulness."

"We believe cognitive biases like frequency bias and expectation bias affected how users perceived AI errors over time."

"Our AI tool doesn’t learn from ongoing use due to regulatory constraints, which led to user frustration when performance didn’t improve."

"People are the plot twist in this journey — despite promising AI, only human connection reveals new problems and solutions."

Ask the Rosenbot
Dorelle Rabinowitz
The Magic Word is Trust
2018 • Enterprise Experience 2018
Gold
Kristin Skinner
8 Types of Measures in Design Operations
2020 • DesignOps Community
Alicia D. Johnson
Disasters and the 21st Century
2021 • Civic Design 2021
Gold
Dane DeSutter
What co-speech gestures reveal about users’ thinking during interviews
2023 • Rosenfeld Community
Zen Ren
Taking Inspiration from Instructional Design for Research
2022 • Advancing Research 2022
Gold
Neil Barrie
Widening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
2024 • Advancing Research 2024
Gold
Amy Paris
Delivering Equity: Government Services for All Ages, Languages, Sexual Orientations, and Gender Identities
2021 • Civic Design 2021
Gold
Angelos Arnis
State of DesignOps: Learnings from the 2021 Global Report
2021 • DesignOps Summit 2021
Gold
Joerg Beringer
Scaling User Research with AI: Continuous Discovery of User Needs in Minutes
2025 • Designing with AI 2025
Gold
Sharon Banh
Reimagining research: What does the field need to grow? [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Jon Fukuda
Theme One Intro
2022 • DesignOps Summit 2022
Gold
Llewyn Paine
[Demo] Deploying AI doppelgangers to de-identify user research recordings
2024 • Designing with AI 2024
Gold
John Maeda
About Design Organizations
2019 • DesignOps Community
Jose Coronado
People First - Design at JP Morgan
2021 • Design at Scale 2021
Gold
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold
Jacqui Frey
Panel Discussion: Integrating DesignOps
2018 • DesignOps Summit 2018
Gold

More Videos

Xenia Adjoubei

"The use of dovetail helps organize diverse information without losing the author’s voice or connection to original data."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"Service design stops being just a support function; it becomes the strategic driver that transforms organizational performance."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"You can’t put the toothpaste back in the tube once biased AI harms your brand or users."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"Not everyone has the same capacity to aspire to a future; this is critical when working with marginalized communities."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"The Climate Dictionary was built in a couple hours over a weekend, focused on function over form."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"I recently observed a director of engineering talking to a content strategist about what they were both seeing because they were both in the actual physical room at the same time."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"Conflict actually strengthens relationships if we choose to engage with it."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"This tool radically automates user research steps so that no facilitation is needed, it’s actually automatic."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"Zoom fatigue is real, people are not necessarily keen to go into another Zoom meeting even if they think it’s valuable."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021