Journeying toward AI-assisted documentation in healthcare
Summary
Documentation technology is the foundation of modern healthcare delivery. Convoluted, redundant, and excessive documentation is a pervasive problem that causes inefficiency in all aspects of the industry. At IncludedHealth, we are developing an AI-assisted documentation that summarizes and documents conversations between patients and their care providers. A care provider can push one button and have their entire patient encounter captured in a succinct and standardized format. Upon a pilot launch, the results were staggering. Within 6 months, we demonstrated a 64% reduction in time per encounter! However, despite our promising results, there still remain challenges specific to the demands of the healthcare domain. As our team continues to develop solutions to meet these challenges, we gain even more clarity on what it takes to design a human-backed, AI-powered healthcare system. Takeaways From this session, you can expect to learn the following: Developing AI design in healthcare requires close collaboration between end users and your data science team Piloting GenAI solutions may be more effective than traditional prototyping Trading accuracy for efficiency is a barrier to adopting GenAI tools in healthcare GenAI design in healthcare requires establishing critical boundaries as well as a good understanding of cognitive processing Other factors to consider when designing AI solutions for service-based industries are understanding how training might be impacted, the importance of standardization vs. personalization of data output and the need for more autonomy and control elements due to consequences of unpredictable output errors
Key Insights
-
•
Generative AI can reduce healthcare documentation time by over 60% but doesn't eliminate the need for manual editing and human review.
-
•
Focusing AI tools initially on low-risk use cases like chat encounter summaries mitigates potential clinical harm.
-
•
Large Language Models (LLMs) excel at summarizing text but struggle with capturing exact clinical details and non-verbal cues.
-
•
Designing AI tools requires managing user expectations to prevent disappointment and frustration over imperfect outputs.
-
•
A simple one-button UI with options to regenerate and manually edit notes improves user workflow efficiency and error recovery.
-
•
Implicit metrics such as the 'edit rate'—the fraction of user edits on AI-generated text—help monitor AI output quality unobtrusively.
-
•
Traditional prototyping methods are less effective for AI; incremental live pilots provide critical learning about AI's unpredictable outputs.
-
•
Operational factors like quality assurance metrics impact user attitudes when AI-generated notes lower scores despite time savings.
-
•
Human-centered collaboration between designers, data scientists, and users is essential to shape AI capabilities and discover unforeseen problems.
-
•
AI models must remain static (non-learning) in healthcare to comply with regulations, creating challenges for ongoing improvements.
Notable Quotes
"We saw documentation time reduced by 64% after six months of using the AI tool."
"Documentation is important for liability and useful for patient handoffs, so notes must be concise yet detailed."
"Healthcare is not a silver bullet for AI; a lot of context comes from non-verbal cues where generative AI doesn’t apply."
"The unpredictable black box nature of LLMs means we had to zoom in on small problems first before seeing the bigger picture."
"We emphasized a simple one-button workflow with manual editing and regeneration to handle inevitable AI errors."
"The edit rate, our metric of human-added characters over total characters, tracks AI output quality without burdening users."
"Users were initially excited but six months later expressed frustration and uncertainty about the tool’s usefulness."
"We believe cognitive biases like frequency bias and expectation bias affected how users perceived AI errors over time."
"Our AI tool doesn’t learn from ongoing use due to regulatory constraints, which led to user frustration when performance didn’t improve."
"People are the plot twist in this journey — despite promising AI, only human connection reveals new problems and solutions."
Or choose a question:
More Videos
"Tags are a balance; project tags are wild and flexible, global tags must be controlled to avoid chaos."
Taylor Jennings Joe Nelson Alex KnollRepository Retrospective: Learnings from Introducing a Central Place for UX Research
March 9, 2022
"Not everyone has the same capacity to aspire to a future; this is critical when working with marginalized communities."
Nicole AleongFuture Orientations to Everyday Life: Futures Anthropology as a Methodology
March 26, 2024
"These are all attempts by large companies to figure out new ways to deliver their products or services or to create new businesses outside of what they’re currently working on."
Jeff GothelfInnovation Studios: the Engines of Enterprise Experimentation
May 14, 2015
"Meeting designers where they spend their time, like integrating accessibility plugins into Figma, is the most effective way to make progress."
Saara Kamppari-MillerDesignOps for Inclusive Design and Accessibility
May 26, 2022
"Having questions about team happiness broken into person, team and product helps clear up confusion in large organizations."
Tess DixonC'mon Get Happy
September 29, 2021
"Soft skills often are a multiplier on your hard skills—they’re just as valuable, if not more so."
Liam ThurstonWhy Your Design Team Is Quitting, And How To Fix It
June 10, 2022
"Careers as ladders are what HR wants, but careers aren’t linear; they should be a jungle gym."
Ian SwinsonDesigning and Driving UX Careers
June 8, 2016
"At DTA, distributed teams and lack of local diversity meant everyone was out in the field a lot to get diverse input."
Leisa ReicheltOpening Keynote: Operating in Context
November 7, 2018
"At Sage Sure, we build paved roads so people can choose to drive on them or carve their own path."
Rachael Greene Alison DavisBuilding a Design Ops Practice that Really Works (Most of the Time)
October 2, 2025
Latest Books All books
Dig deeper with the Rosenbot
What role can AI play in improving patient education and discharge instructions for better comprehension?
What are the main categories of survey tools and how do they differ in use cases?
How do research operations teams manage the tension between consolidating tools and needing specialized capabilities?