Log in or create a free Rosenverse account to watch this video.
Log in Create free account100s of community videos are available to free members. Conference talks are generally available to Gold members.
This video is featured in the Ed's Test playlist playlist.
Summary
Artificial Intelligence (AI) tools like large language models present opportunities — and risks — for people working with digital content. Can AI help with tedious tasks, like encoding and categorizing documents, or rewriting text snippets? Will AI be so good at these tasks that Information Architects (IAs) are no longer needed? In this session, Jeff and Karen provide an overview of what "AI" means for IAs, explaining the differences between natural language processing, machine learning, and large language models. They also dig into a real-world example of using different systems to categorize web content from Reddit. IAs will come away from the session reassured that the robots pose no threat to their jobs.
Key Insights
-
•
Natural language processing is about mathematically modeling words and their relationships but struggles with context and nuance.
-
•
Machine learning models detect statistical patterns but do not truly understand content or reason about it.
-
•
Large language models like ChatGPT work by predicting what text should follow based on massive internet-scale training data.
-
•
Tokenization breaks text into meaningful units, filtering out filler words to create useful representations for AI.
-
•
Choosing the right AI model and customizing prompts is more important than using the 'biggest' or most popular model.
-
•
Using AI for content categorization requires iterative refinement, precise instructions, and ongoing validation.
-
•
AI tools often generate plausible but overlapping categories, which complicates achieving mutually exclusive, exhaustive taxonomies.
-
•
Cost, time, and carbon footprint are significant considerations for scaling AI workflows, making simpler tools sometimes preferable.
-
•
Visualization techniques like Sankey diagrams and proximity charts help interpret AI categorization results and spot issues.
-
•
Human expertise in information architecture is more vital than ever to effectively harness AI in complex content tasks.
Notable Quotes
"AI is not going to save us. It is a tool that we use with significant human support and understanding."
"Trying to map all the possible connotations and contextual interrelationships between words is mindbogglingly computationally expensive."
"The risk with LLMs is that they invent new categories or start writing Reddit posts instead of just categorizing."
"The best large language models we tested only achieved about 47% accuracy in categorizing Reddit posts."
"Breaking down tasks into one thing at a time dramatically improves AI performance and reliability."
"Choosing a tool designed for your specific problem matters way more than just picking the smartest AI available."
"We ended up using garbage ID numbers for categories to keep the LLM from inventing new ones."
"It wasn’t obvious the difference between AI generating category ideas and validating categories until we tried both."
"Humans still excel at mental modeling of topics and nuance that AI struggles to replicate automatically."
"Foundational IA and metadata infrastructure are the meat and potatoes supporting scalable AI-driven experiences."
Or choose a question:
More Videos
"All data has bias, problems, and limitations; there is no perfectly clean data, only varying degrees of quality."
Jemma Ahmed Steve Carrod Chris Geison Dr. Shadi Janansefat Christopher NashDemocratization: Working with it, not against it [Advancing Research Community Workshop Series]
July 24, 2024
"We ran a design system bootcamp to rotate engineers and prevent bottlenecks and burnout."
Nina JurcicThe Design System Rollercoaster: From Enabler and Bottleneck to Catalyst for Change
October 3, 2023
"If your organization is healthy, the haters pretty quickly get marginalized when everyone else is excited about the design system."
Nathan Curtis Nalini P. Kotamraju Jack Moffett Dawn ResselDiscussion
June 9, 2016
"Measurements are always a conversation; they should not be a covenant imposed on you, especially when you’re trying something new."
Saara Kamppari-Miller Nicole Bergstrom Shashi JainKey Metrics: Comparing Three Letter Acronym Metrics That Include the Word “Key”
November 13, 2024
"The re-platforming journey is transformative not just for the product, but for the people and teams involved."
Malini RaoLessons Learned from a 4-year Product Re-platforming Journey
June 9, 2021
"We are not collecting vanity analytics here, this data is doing real work informing design decisions."
Mackenzie Cockram Sara Branco Cunha Ian FranklinIntegrating Qualitative and Quantitative Research from Discovery to Live
December 16, 2022
"Sponsor sessions are not sales pitches but truly high-quality content that you won’t want to miss."
Bria AlexanderOpening Remarks
June 9, 2021
"Many product teams have never come face to face with their revenue numbers before."
Jackie HoLead Effectively While Preserving Team Autonomy with Growth Boards
January 8, 2024
"Soft eyes means zooming between the particular tree and the whole forest to see the bigger picture."
Dan HillDesigning for the infrastructures of everyday life
June 4, 2024