Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

Coffee with Lou #4: Taking a Peek Under the Rosenbot's Hood (Videoconference)
Friday, June 14, 2024 • Rosenfeld Community
Share the love for this talk
Coffee with Lou #4: Taking a Peek Under the Rosenbot's Hood (Videoconference)
Speakers: Peter Van Dijck and Louis Rosenfeld
Link:

Summary

What happens when you cross an eager librarian, a happy puppy, and 800 UX experts? You get the Rosenbot—Rosenfeld's new GPT-4 level chatbot, trained on our books and hundreds of hours of conference and community call recordings. What went into creating the Rosenbot? Lou is joined by SimplyPut's Peter van Dijck, an old friend from the IA community and the chief architect of the Rosenbot. If you're beginning your journey into developing generative AI products, you'll want to join Lou and Peter to learn from their lessons, ask questions, and share your own thoughts on AI's role in making curated content more useful and impactful.

Key Insights

  • Building effective AI chatbots requires substantial investment in evaluation—often 30-50% of total project time and budget—to ensure quality, safety, and usefulness.

  • The Rosenbot is trained on highly curated Rosenfeld Media content, avoiding outdated or contradictory material common to most organizational knowledge bases.

  • The chatbot is explicitly designed to credit the human authors and experts behind the content to maintain transparency and respect for intellectual property.

  • Rather than summarizing entire books, the Rosenbot delivers insights linked to sources, encouraging users to explore original works and supporting book sales.

  • Conversational memory is simulated by summarizing prior dialogue and feeding it back into the model, as current large language models lack intrinsic memory.

  • Evaluation requires both end-user feedback and expert review since users might be satisfied with inaccurate or incomplete answers.

  • There is a persistent challenge in balancing the presentation of evergreen knowledge with content currency given constantly evolving methodologies and societal norms.

  • AI can surface relevant historical context, but nuanced interpretation and linking of evolving methodologies likely require human curation or 'knowledge historians'.

  • Expanding the interface beyond the traditional conversational model with interactive widgets can enhance user interaction, yet is still underexplored.

  • Analytics on chatbot queries can reveal trends in users’ interests and help authors understand how their work is being referenced and applied.

Notable Quotes

"You can’t read it all or watch it all yourself, but now we have technology that can make that useful—like a community memory for our content."

"A very large percentage of the project should be focused on evaluation, sometimes 30 to 50 percent of time and budget."

"The large language model itself does not have memory, so we summarize previous dialogue and feed it back to simulate memory."

"We’ve trained it not to summarize books to avoid cannibalizing book sales; it’s a tool to promote discovery and connection to content."

"Evaluations need both end users and experts because an answer that makes a user happy can be objectively wrong."

"The content we provide is all curated and high quality, unlike typical intranet or company content that’s often outdated or conflicting."

"This product is not perfect but might be the best there is for UX and product content at this point—about 60% there, aiming for 80%."

"Methodologies and societal understandings change, so the system needs to surface changes over time, connecting older ideas to newer ones."

"The idea of a conversational interface shouldn’t constrain us; adding widgets or interactive elements is definitely possible."

"If you don’t jump in and build this stuff, all you’ll do is read articles that don’t give you the real understanding."

Ask the Rosenbot
Nora Tejeda
Scaling Design Capabilities at BBVA Through a Self-service Design Model
2021 • Design at Scale 2021
Gold
Sam Proulx
SUS: A System Unusable for Twenty Percent of the Population
2021 • DesignOps Summit 2021
Gold
Andreas Huebner
What Is It Like To Be Part of The UX Team at Compass?
2021 • Advancing Research 2021
Gold
Allison Sanders
Operating with Purpose
2024 • DesignOps Summit 2020
Gold
John Maeda
Making Sense of Enterprise UX
2016 • Enterprise UX 2016
Gold
Peter Merholz
The Mysterious Case of the Missing UX Career Path (Videoconference)
2022 • DesignOps Community
Lily Aduana
5 Reasons to Bring Your Recruiting in-House (and How To Do It)
2021 • Advancing Research 2021
Gold
Kristin Skinner
Group Activity: A Deep Dive Into Value and Outcomes
2019 • DesignOps Summit 2019
Gold
Meredith Black
Scaling Design Culture
2017 • DesignOps Summit 2017
Gold
Emilia Åström
Unlock Your Team’s Intelligence with Collaboration Design
2022 • Design at Scale 2022
Gold
Peter Merholz
Customer-Centered Design Organizations
2017 • Enterprise Experience 2017
Gold
Brianna Sylver
Lead With Purpose
2020 • Advancing Research 2020
Gold
Meaghan Waters
Lack of Product Thinking will Doom Your Legacy Modernization
2021 • Design at Scale 2021
Gold
Jorge Arango
Exploding the Notebook: How to Unlock the Power of Linked Notes (2nd of 3 seminars) (Videoconference)
2024 • Rosenfeld Community
DesignOps Community Sensing Session (Videoconference)
2021 • DesignOps Community
Caitlyn Hampton
Compass 101: Growing Your Career In A Startup World
2021 • Design at Scale 2021
Gold

More Videos

Jemma Ahmed

"If you miss any one of access, accuracy, insight generation, and engagement, it’s not truly democratization."

Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher Nash

Democratization: Working with it, not against it [Advancing Research Community Workshop Series]

July 24, 2024

Nina Jurcic

"Having a design system is not the end goal; it enables radical cost cuts, reduces tech debt, and improves workflows."

Nina Jurcic

The Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change

October 3, 2023

Nathan Curtis

"I try to position the system as an enabling force for designers to do their work better and succeed better."

Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn Ressel

Discussion

June 9, 2016

Saara Kamppari-Miller

"If we are working on something that does not ladder up to our KSPs, then we should not be working on it."

Saara Kamppari-Miller Nicole Bergstrom Shashi Jain

Key Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”

November 13, 2024

Malini Rao

"Don’t forget the critical change agent persona — adoption and success was determined by the admin or practitioner users."

Malini Rao

Lessons Learned from a 4-year Product Re-platforming Journey

June 9, 2021

Mackenzie Cockram

"Over 4,000 users volunteered to look at the beta version, giving us excellent data at scale."

Mackenzie Cockram Sara Branco Cunha Ian Franklin

Integrating Qualitative and Quantitative Research from Discovery to Live

December 16, 2022

Bria Alexander

"Day one of Design at Scale is going to be about ensuring craft on a massive scale."

Bria Alexander

Opening Remarks

June 9, 2021

Jackie Ho

"Many product teams have never come face to face with their revenue numbers before."

Jackie Ho

Lead Effectively While Preserving Team Autonomy with Growth Boards

January 8, 2024

Dan Hill

"The street used to be for public life; then cars took over and changed it profoundly."

Dan Hill

Designing for the infrastructures of everyday life

June 4, 2024