Summary
Documentation technology is the foundation of modern healthcare delivery. Convoluted, redundant, and excessive documentation is a pervasive problem that causes inefficiency in all aspects of the industry. At IncludedHealth, we are developing an AI-assisted documentation that summarizes and documents conversations between patients and their care providers. A care provider can push one button and have their entire patient encounter captured in a succinct and standardized format. Upon a pilot launch, the results were staggering. Within 6 months, we demonstrated a 64% reduction in time per encounter! However, despite our promising results, there still remain challenges specific to the demands of the healthcare domain. As our team continues to develop solutions to meet these challenges, we gain even more clarity on what it takes to design a human-backed, AI-powered healthcare system. Takeaways From this session, you can expect to learn the following: Developing AI design in healthcare requires close collaboration between end users and your data science team Piloting GenAI solutions may be more effective than traditional prototyping Trading accuracy for efficiency is a barrier to adopting GenAI tools in healthcare GenAI design in healthcare requires establishing critical boundaries as well as a good understanding of cognitive processing Other factors to consider when designing AI solutions for service-based industries are understanding how training might be impacted, the importance of standardization vs. personalization of data output and the need for more autonomy and control elements due to consequences of unpredictable output errors
Key Insights
-
•
Generative AI can reduce healthcare documentation time by up to 64% in low-risk chat encounter scenarios.
-
•
Limiting AI applications to verbal and text-based interactions reduces risk compared to video or phone encounters where non-verbal cues matter.
-
•
LLMs excel at summarization but struggle with capturing exact medical details and unspoken actions.
-
•
Balancing model accuracy and latency is critical to maintain business value through time savings.
-
•
Incremental, imperfect pilot releases provide better learnings than traditional iterative prototyping with AI tools.
-
•
Implicit user feedback mechanisms, like measuring the edit rate of AI-generated notes, help assess output quality without disrupting workflows.
-
•
User excitement about AI tools can decline due to cognitive biases such as novelty wearing off, frequency bias toward errors, and expectation bias.
-
•
Operational metrics, like note quality affecting performance reviews, can shape user attitudes toward AI tools more than raw efficiency gains.
-
•
Educating users consistently on AI’s augmentative role and setting realistic expectations improves tool adoption and satisfaction.
-
•
Human-centered design involving early collaboration between designers, data scientists, researchers, and quality assurance is essential for effective AI integration in healthcare.
Notable Quotes
"Physicians can spend up to two hours on documentation for every one hour of patient interaction."
"We released several incremental but imperfect pilot solutions to inform usability and strategy rather than relying on typical prototyping."
"The current LLM models are great at summarizing, but they’re not so great at capturing exact details."
"AI is not a comprehensive silver bullet solution; we limited our scope to capturing notes for verbal interactions only."
"Our UI focused on saving time, making the workflow one button and enabling manual edits and regeneration for error recovery."
"We created an edit rate metric—the fraction of human-added characters—to measure how much human editing was needed."
"Users initially were excited about AI, but six months later, there was more dissatisfaction and frustration despite quantitative time savings."
"Errors that were funny at first became annoying, leading to frequency bias because users felt errors were more frequent than before."
"Employees were graded on note quality, and their scores declined by about 10% after using the AI tool, impacting morale."
"The plot twist was that AI was causing new problems impacting performance, morale, and satisfaction, showing people are the real key."
Or choose a question:
More Videos
"Teaching product managers heuristics took 15 minutes, and now they use that shared language to articulate design."
Standardizing Product Merits for Leaders, Designers, and Everyone
June 15, 2018
"There’s no such thing as one-size-fits-all methods; what works in one part of the map doesn’t work in another."
Simon WardleyMaps and Topographical Intelligence (Videoconference)
January 31, 2019
"Many smartwatch designs assume male hands as default, excluding smaller wrists and diverse body types."
Sandra CamachoCreating More Bias-Proof Designs
January 22, 2025
"We’re all capable of creating and perpetuating toxic work relationships."
Darian DavisLessons from a Toxic Work Relationship
January 8, 2024
"I challenge everyone to create a strategic AI integration plan that goes beyond everyday tasks to achieve career and company goals."
Fisayo Osilaja[Demo] The AI edge: From researcher to strategist
June 4, 2024
"Working with CEOs like Mark Templeton is like Dancing with the Stars — an interpretive dance of translating fuzzy ideas."
Uday GajendarThe Wicked Craft of Enterprise UX
May 13, 2015
"What’s so difficult about designing a login? On the surface, it’s simple, but the real challenge was cultural alignment across business units."
Davis Neable Guy SegalHow to Drive a Design Project When you Don’t Have a Design Team
June 10, 2021
"TripAdvisor is really nine major business units each thinking like sub companies instead of one end-to-end user experience."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"At Qubits, you can customize or copy standard design processes like Lean UX and add or remove activities easily."
Aurobinda Pradhan Shashank DeshpandeIntroduction to Collaborative DesignOps using Cubyts
September 9, 2022
