Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI for Information Architects: Are the robots coming for our jobs?

Thursday, November 21, 2024 • Rosenfeld Community
Share the love for this talk
AI for Information Architects: Are the robots coming for our jobs?
Speakers: Karen McGrane and Jeff Eaton
Link:

Summary

Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.

Key Insights

  • AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.

  • Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.

  • Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.

  • Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.

  • Different AI models interpret the same content differently; model choice significantly affects results.

  • Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.

  • Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.

  • Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.

  • Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.

  • Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.

Notable Quotes

"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."

"You have to ask specific, very precise questions, not big general ones, to get useful AI results."

"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."

"Natural language processing is basically about figuring out how to math words."

"Different AI models have wildly different views of the same content; the model choice makes a big difference."

"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."

"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."

"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."

"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."

"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."

Ask the Rosenbot
Scott Plewes
Why Isn't Your UX Approach Going Viral?: A Mathematical Model
2023 • Advancing Research 2023
Gold
Alex Hurworth
Designing a Contact Tracing App for Universal Access
2020 • DesignOps Summit 2020
Gold
Savannah Carlin
Don't botch the bot: Designing interactions for AI
2024 • Designing with AI 2024
Gold
Peter Levin
Solve a Problem Here, Transform a Strategy There: Research as an Occasion for Expanding Organizational Possibility
2024 • Advancing Research 2024
Gold
Giff Constable
Financial fluency for product leaders: AMA with Giff Constable
2024 • Rosenfeld Community
Surya Vanka
Unleashing Swarm Creativity to Solve Enterprise Challenges
2021 • Design at Scale 2021
Gold
Dr. Karl Jeffries
The Science of Creativity for DesignOps
2024 • DesignOps Summit 2020
Gold
Gina Mendolia
Coordinated collaboration: a Service Design & DesignOps love story
2025 • Advancing Service Design 2025
Gold
Nathan Curtis
Discussion
2016 • Enterprise UX 2016
Gold
Bria Alexander
Opening Remarks
2021 • DesignOps Summit 2021
Gold
Jeff Ephraim Bander
Eye Tracking Gamechanger: Why Smartphone Eye Tracking will Revolutionize Your UX Research
2022 • Advancing Research 2022
Gold
Darian Davis
Lessons from a Toxic Work Relationship
2024 • Enterprise Experience 2020
Gold
Briana Thomas
When Design Ops Comes in H.O.T. : A Tale of a Transformed Design Org
2021 • DesignOps Summit 2021
Gold
Craig Villamor
Design Systems for Ethical Design
2023 • Enterprise Community
Dan Willis
Enterprise Storytelling Sessions
2019 • Enterprise Experience 2019
Gold
Prayag Narula
How to Empower Your Designers to Do Good Research – And Why You Want To
2022 • Design at Scale 2022
Gold

More Videos

Xenia Adjoubei

"The Researcher in Residence program invites members of affected communities to use our tools over a funded three-month period pro bono."

Xenia Adjoubei Sean Bruce

Empowering Communities Through the Researcher in Residence Program

March 29, 2023

Tamara Kartoziia

"The focus isn’t on head to head competition, but on specialization and differentiation in the Italian market."

Tamara Kartoziia

Think global, adapt local: how service design accelerated B2B market entry by 6 months

November 20, 2025

Jay Bustamante

"Embrace the role of party planner with your expertise to shape ethical AI innovation."

Jay Bustamante

Navigating the Ethical Frontier: DesignOps Strategies for Responsible AI Innovation

October 2, 2023

Nicole Aleong

"Destiny is the belief that a certain future outcome is predetermined regardless of the path."

Nicole Aleong

Future Orientations to Everyday Life: Futures Anthropology as a Methodology

March 26, 2024

Nick Lewis

"Using generative AI for design risks losing the human creativity that makes design special."

Nick Lewis

Designing and building low-carbon websites independently

November 18, 2025

Vicky Teinaki

"By intentionally connecting the user outcomes we’re looking for with a business outcome via a structured hypothesis, designers can start considering the wider business impact."

Vicky Teinaki Michele Marut Tim Parmee

Short Take #3: UX/Product Lessons from Your Industry Peers

December 6, 2022

Laura Weiss

"The alternate to engaging with conflict is suffering quietly or accumulating conflict debt."

Laura Weiss

Turn Down the Heat: 3 Ways to Handle Conflict in the Moment

November 20, 2024

Joerg Beringer

"Most product teams don't really have user research outputs or they come too late and disconnected from design."

Joerg Beringer Thomas Geis

Scaling User Research with AI: Continuous Discovery of User Needs in Minutes

June 10, 2025

Shelby Switzer

"It’s okay to be goofy and have fun—creating space for awkwardness and playfulness helps engagement."

Shelby Switzer

Making Space for Community Knowledge-sharing in a Distributed World

December 10, 2021