[Demo] How to re-categorize content at scale using LLMs
Summary
Large Language Models (LLMs) are to language as spreadsheets are to numbers: tools for modeling, exploration, and development. Among their many capabilities, LLMs can alleviate chores related to the design and implementation of information architectures. But doing so requires venturing beyond chat-based interfaces. In this brief demonstration, we'll see how to use OpenAI's API and a few open source command line tools to re-categorize content in a 1,000+ page website. The techniques demonstrated can be extended to other common content organization tasks.
Key Insights
-
•
Manual retagging of 1,200 blog posts would take about 10 hours, but leveraging GPT-4 reduced active human time to about 2 hours.
-
•
Using GPT-4 via command line and shell scripts enables automated tagging outside typical chat interfaces.
-
•
An organically grown taxonomy over 20 years contained unclear acronyms and inconsistent tag forms that GPT initially struggled with.
-
•
Cleaning and standardizing the taxonomy before prompting GPT is critical for effective AI assistance.
-
•
A review step of AI-suggested tags in CSV format allows human correction to avoid hallucinations entering production.
-
•
GPT-4 can propose new and useful tags outside the original taxonomy, enriching content classification.
-
•
The four-step GRU framework (Gather, Review, Update, Wrap up) balances automation with human oversight.
-
•
Storing blog content as markdown files simplifies integrating AI workflows via scripting and file manipulation.
-
•
The approach is adaptable and scalable to other CMS platforms by replacing scripting with API calls.
-
•
Taxonomies should use clear, unambiguous terms to improve both human and AI understanding.
Notable Quotes
"Some of the older content has discoverability problems, which is typical with blogs."
"Doing this tagging manually would have taken me around 10 hours of mind-numbing work."
"I’m actually using GPT-4, but not via the chat interface—I'm calling it from the Mac’s command line."
"I had to clean the taxonomy up because GPT wouldn’t know what to do with acronyms like TAOI."
"I save the proposed tags to a CSV file so I can preview and edit them before applying the changes."
"A middle review step prevents hallucinations from making it into the production site."
"GPT-4 functioned as an assistant not just in retagging but also in improving the taxonomy itself."
"The entire process took about three hours from start to finish, about a fifth of the manual time."
"Use clear and obvious terms in taxonomies—unusual acronyms won’t make sense to GPT or others."
"You need to review proposed changes before committing them to production, otherwise errors sneak in."
Or choose a question:
More Videos
"If you need to cross a dangerous river, you don’t paddle straight into it; you use the opposing force of the river to move forward."
John CutlerOxbows, Rivers, and Estuaries: How to navigate the currents of change (without burning out)
December 3, 2024
"A common toxic behavior is glory seeking, like presenting work as your own when it was a team effort."
Darian DavisLessons from a Toxic Work Relationship
January 8, 2024
"You don’t have to sit through two hours a day; we’ll spend about 15 minutes on setup then you’ll pair up and do the exercise."
Dave GrayGroup Activity: Making Sense of DesignOps
November 7, 2017
"The subtle art of taking care of yourself and giving a f in ways that generally encourage coherent behaviors."
John CutlerThe Alignment Trap
November 29, 2023
"Just because you can color code something doesn’t mean you should."
Abby CovertStuck? Diagrams Help
October 27, 2022
"Side-pav proposals answer what’s going on, why it matters now, what you propose, what action to take, and what benefit comes from it."
Mark InterranteCollaboration Flows in Product Development
June 9, 2017
"The future will be struggle; we need to face that forward, not deny it."
Devon PowersImagining Better Futures
March 9, 2022
"Unbiased qualitative research does not exist. The best we can do is acknowledge and minimize biases."
Prayag NarulaHow to Empower Your Designers to Do Good Research – And Why You Want To
June 10, 2022
"You can’t just be a designer adapting UX workflows assuming everybody else’s workflow stays the same; everything is changing with AI."
Peter Van DijckHands on AI #3: Claude Code for UX people
October 22, 2025
