Rosenverse

Log in or create a free Rosenverse account to watch this video.

Log in Create free account

100s of community videos are available to free members. Conference talks are generally available to Gold members.

AI for Information Architects: Are the robots coming for our jobs?

Thursday, November 21, 2024 • Rosenfeld Community
Share the love for this talk
AI for Information Architects: Are the robots coming for our jobs?
Speakers: Karen McGrane and Jeff Eaton
Link:

Summary

Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.

Key Insights

  • AI’s vast hype often overshadows its practical limitations in nuanced tasks like taxonomy creation and content categorization.

  • Natural language processing excels at tokenizing and quantifying text but struggles with capturing deep contextual meaning and nuance.

  • Machine learning models rely heavily on quality training data and do not possess reasoning abilities; they identify patterns without true understanding.

  • Text embeddings convert documents into numerical vectors enabling proximity-based semantic search and clustering, offering fast but approximate classification.

  • Different AI models interpret the same content differently; model choice significantly affects results.

  • Large language models (LLMs) like chat GPT are very flexible but require explicit, narrowly defined prompts and guardrails to avoid off-task or fabricated outputs.

  • Accurate AI categorization of complex content is far from perfect; the best models achieved only about 47% accuracy in the Reddit post experiment.

  • Using AI at scale entails significant computational cost, time, and environmental impact that must be factored into decisions.

  • Information architects remain essential to breaking down complex tasks into clear, testable questions for AI and validating results.

  • Successful enterprise AI implementations, like Microsoft Learn, rely heavily on existing IA and content structure infrastructures rather than AI alone.

Notable Quotes

"What is AI gonna do for us? Are we afraid the robots will take our jobs? AI is a tool, not a replacement."

"You have to ask specific, very precise questions, not big general ones, to get useful AI results."

"Trying to map all possible connotations and context in language is mind-bogglingly computationally expensive."

"Natural language processing is basically about figuring out how to math words."

"Different AI models have wildly different views of the same content; the model choice makes a big difference."

"The biggest challenge with LLMs was keeping them on task instead of writing their own Reddit posts."

"The top LLM we tested got only 47% accuracy categorizing Reddit posts, which is sobering."

"Running AI categorization on 2000 posts took 38 hours, $19, and as much carbon as driving a 1998 Chevy Malibu from Philly to Brooklyn."

"The real value of AI in IA is as a collaborative tool requiring significant human support, not a magic knob to turn."

"AI-powered enterprise tools sit on top of the classic IA infrastructure; without it, AI can’t scale effectively."

Ask the Rosenbot
Joseph Meersman
Sweating the Pixel: Scaling Quality through Critique
2021 • Design at Scale 2021
Gold
Bryce Benton
[Demo] AI-powered UX enhancement: Aligning GitHub documentation with USWDS at Austin Public Library
2024 • Designing with AI 2024
Gold
Megan Blocker
A Selectively Scrappy Approach to ResearchOps
2018 • DesignOps Summit 2018
Gold
Amy Jiménez Márquez
The Atypical UX Manager Path
2020 • Enterprise Community
Dave Gray
Group Activity: Making Sense of DesignOps
2017 • DesignOps Summit 2017
Gold
Ryan Rumsey
Business Influence Without Losing Your Soul
2021 • Enterprise Community
Sharon Banh
Reimagining research: What does the field need to grow? [Advancing Research Community Workshop Series]
2024 • Advancing Research Community
Sam Proulx
Accessibility: An Opportunity to Innovate
2022 • Civic Design 2022
Gold
Onur Kocan
Understanding the Strategy for Civic Design in a Complex City: Istanbul
2022 • Civic Design 2022
Gold
Mariah Hay
BUILD: Discussion
2018 • Enterprise Experience 2018
Gold
Monty Hammontree
The Future of UX Research
2020 • Advancing Research Community
Theresa Neil
Designing for Wellness: Specializing in Healthcare
2024 • Rosenfeld Community
Taylor Jennings
Repository Retrospective: Learnings from Introducing a Central Place for UX Research
2022 • Advancing Research 2022
Gold
Fredrik Matheson
First-time users, longtime strategies: Why Parkinson’s Law is making you less effective at work – and how to design a fix.
2016 • Enterprise UX 2016
Gold
Billy Carlson
Ideation tips for Product Managers
2022 • Design in Product 2022
Gold
Abby Covert
Panel: Collaboration Tools
2017 • DesignOps Summit 2017
Gold

More Videos

Joanna Vodopivec

"Inviting other teams into research sessions lets us triangulate findings by combining interviews, observations, and focus groups."

Joanna Vodopivec Prabhas Pokharel

One Research Team for All - Influence Without Authority

March 9, 2022

Louis Rosenfeld

"You have to build relationships with legal and political leadership to figure out what you can actually change."

Louis Rosenfeld Lashanda Hodge Senongo Akpem Chris Hodowanec

Becoming a Civic Designer: Making the Move from Private to Public Sector

November 17, 2022

Bria Alexander

"We are back and we are right on time. Eight o'clock on the dot."

Bria Alexander

Day 3 Welcome

September 25, 2024

Uday Gajendar

"Sponsor sessions are so great that people want recordings. They’re not sales pitches."

Uday Gajendar Louis Rosenfeld

Day 2 Welcome

June 5, 2024

Sam Proulx

"More ways to contact support—chat, email, phone—are essential because different disabilities require different options."

Sam Proulx

Online Shopping: Designing an Accessible Experience

June 7, 2023

Russ Unger

"Manhole cover questions belong in the past. Focus on real skills and relevant experience instead."

Russ Unger

Onboarding: The Ecosystem, not the Afterthought

November 7, 2017

Catherine Dubut

"We could impact the end-to-end customer service and customer experience through our most underutilized asset: our store employees."

Catherine Dubut

Bridging Physical and Digital Spaces: Approaches to Retail Service Design

March 18, 2021

Josh Clark

"Assistance interfaces combine open-ended dialogue with the ability to do tasks on your behalf."

Josh Clark Veronika Kindred

Sentient Design: New Postures for AI-Mediated Experiences (2nd of 3 seminars)

January 29, 2025

Dantley Davis

"There were things I controlled and things I didn’t, and I needed to plan for both."

Dantley Davis

Leadership & Diversity—A Fireside Chat with Dantley Davis

September 17, 2020