Latent Scope: Finding structure in unstructured data
This video is featured in the Designing with AI 2025 playlist and 1 more.
Summary
Latent Scope is an open-source tool I built that streamlines a process of embedding text, mapping it to 2D, clustering the data points on the map and summarizing those clusters with an LLM. Once the process is done on a dataset structure emerges from the unstructured text, allowing us to get a sense of patterns in the survey answers. Themes like “the time it takes” to develop data visualization pop out, as do “the importance of good design.” While people don’t use the same language to describe these themes, they show up as clusters in the tool thanks to the power of embedding models.
https://github.com/enjalot/latent-scope
Key Insights
-
•
AI embeddings transform unstructured data like text or sketches into high-dimensional vectors capturing hidden semantic patterns.
-
•
Dimensionality reduction algorithms map these high-dimensional embeddings into 2D clusters, making patterns visually accessible.
-
•
Free text survey responses, often too large to analyze manually, can be organized and explored effectively through embedding and clustering.
-
•
Google Cloud user journey data revealed surprising usage patterns, like enterprise and beginner tools being combined unexpectedly.
-
•
Simple rule-based filters for harmful prompts in generative AI image models can be bypassed by subtle prompt variations; embedding-based classifiers are more effective.
-
•
Latent Scope is an open source tool enabling non-technical users to embed, cluster, label, and visualize unstructured text data locally.
-
•
Local runs of embedding and clustering models on modest hardware are practical, preserving data privacy and sensitive user information.
-
•
Embedding models trained on multilingual data can cluster semantically similar texts across diverse languages in one shared space.
-
•
Breaking long text into smaller chunks can make embeddings more manageable and improve similarity comparisons.
-
•
Hands-on exploration of embedding models via platforms like Hugging Face helps users internalize concepts of semantic similarity and pattern discovery.
Notable Quotes
"People just don't get that the design and the process is fundamental."
"Similar inputs will produce similar high dimensional numbers."
"Dimensionality reduction algorithms take data points in high dimensional space and put them close together in 2D if they're similar."
"We found patterns that product teams didn’t expect or even want to look for."
"Simple word-list based filters can easily be tricked by misspellings or slight variations in prompts."
"What if you didn’t know there were important questions you should be asking in your data?"
"Latent Scope lets you quickly explore hundreds or thousands of free text responses to find clusters and patterns."
"You don’t need special hardware; these open source models can run locally on an M1 MacBook or a gaming machine."
"Multilingual embedding models can cluster similar meanings across languages in a shared latent space."
"Downloading and playing with local open source models gives a different experience than using faceless APIs mediated through interfaces."
Or choose a question:
More Videos
"We have do’s and don’ts, but approaching it from a rules-based perspective just reduces buy-in."
Dave Malouf Amy ThibodeauPanel: Design Systems and Documentation
November 7, 2017
"One of the most important questions before starting is what tools do I have to work with."
John DonmoyerShipping your code generation experiments to production
June 11, 2025
"Novice users can try things out thanks to automation, but experts can focus on the challenging and creative pieces."
Clara Kliman-SilverUX Futures: The Role of Artificial Intelligence in Design
June 7, 2023
"The post-PC era has had more impact on enterprise UX and computing than any other factor in all history."
Mark TempletonCreating a Legacy: the ultimate experience
June 9, 2017
"Every parent we talked to had some variation of the question: this exists, can I use it right now? Where was this when I needed to apply?"
Sarah GallimoreInspire Progress with Artifacts from the Future
November 18, 2022
"When you magnify prototypes, they may look blurry, but that’s something to fix in the final website design."
Sam ProulxPrototype Reviews, People With Disabilities, and You
December 8, 2021
"Our monthly voice of the customer reports we call Stories in Dovetail; we share them with the team every month."
Anna Nguyen Emily BroganWhy Our Voice of the Customer is Better Than Yours
March 10, 2022
"Unmoderated tools can achieve 100% completion rates and rapid insights if designed thoughtfully."
Anna Poznyakov Richa PrajapatiGet The Most Out Of Stakeholder Collaboration—and Maximize Your Research Impact
March 12, 2021
"I don’t exactly even know who I am if not a researcher."
Nalini P. KotamrajuTwo Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"
March 10, 2021