Rosenverse
Don’t call it AI: Turn words into numbers with quantitative ethnography

This video is only accessible to Gold members. Log in or register for a free Gold Trial Account to watch.

Log in Register

Most conference talks are accessible to Gold members, while community videos are generally available to all logged-in members.

Don’t call it AI: Turn words into numbers with quantitative ethnography

Gold
Wednesday, March 11, 2026 • Advancing Research 2026
Share the love for this talk
Don’t call it AI: Turn words into numbers with quantitative ethnography
Speakers: Vitorio Miliano
Link:

Summary

Quantitative ethnography is the niche subfield you’ve never heard of, but it’s one you’ve been increasingly pressured to practice in over the past couple of years. It’s the math that turns words into numbers underlying generative AI, and LLMs have been getting in between you and a radically new approach to working with verbatims, transcripts, and other texts.

Business stakeholders are always pushing for greater efficiency, faster turnarounds. Qualitative researchers are always looking for more contact with users, and greater engagement with findings and reporting. Quantitative ethnography (and epistemic network analysis) offers a compromise: by trading structure and semantics for human sensemaking in the analysis part of research, perhaps both groups can get what they want.

I’ve had the opportunity to conduct quantitative ethnographic analyses in enterprise studies involving dozens of products, and impacting hundreds of thousands of end-users. Stakeholders were willing to accept a different kind of analysis, and engage more deeply with the process, in exchange for quicker answers.

In this talk, I’ll share how quantitative ethnography differs from qualitative ethnography, the tradeoffs you’ll have to make, and the kinds of results you can expect. This isn’t a tools talk, but you won’t need to do any math, either. I’ll close with a look into the near future, one where you can talk with as many users as will take your call with effectively zero additional analysis work; where you can have the analysis running live during your session, and have the user participate in the sensemaking process on-the-fly; and the dream of every product manager, one where stakeholders can have dashboards of evidence updated live as users talk.

Key Insights

  • •

    Quantitative ethnography unifies qualitative ethnographic methods with quantitative statistical validation, avoiding typical mixed-methods back-and-forth.

  • •

    Formalizing coding rules in a detailed code book is essential to scale qualitative insights and enable automation.

  • •

    Defining mechanistic signifiers, such as keywords or phrase rules, is necessary to automate qualitative coding effectively.

  • •

    Intra-sample statistical analysis uses each coded line as a data point rather than each respondent, enabling meaningful stats from small sample sizes.

  • •

    Partnering with data scientists is critical because quantitative ethnography requires specialized, adjusted statistical methods that differ from conventional ones.

  • •

    Researchers must regularly validate coding accuracy and statistical assumptions over time, a process called closing the interpretive loop.

  • •

    Quantitative ethnography can scale from a handful of interviews to thousands of verbatim responses, maintaining rigor at all scales.

  • •

    Epistemic network analysis helps identify and quantify relationships between qualitative codes within the text data.

  • •

    Large language models can automate parts of quantitative ethnography but require sacrificing some control over code definitions and initial synthesis.

  • •

    Quantitative ethnography opens the possibility for near-real-time insights by automating coding and saturation metrics during ongoing data collection.

Notable Quotes

"Business stakeholders push researchers for faster turnarounds and numbers, often favoring surveys over deep interviews."

"Quantitative ethnography isn’t mixed methods; it’s a unified method using both qualitative theory and quantitative validation."

"If you can’t come up with a rule for something, you can’t code it."

"Each coded line is a data point, which enables statistical power even with small numbers of respondents."

"Partner with data scientists to pick and adjust statistical tests because quantitative ethnography requires new assumptions."

"Closing the interpretive loop means regularly checking that your coding and stats hold up as new data arrives."

"Epistemic network analysis reveals meaningful connections between codes, suggesting but not proving why ideas cluster."

"Large language models cluster text using semantic relationships rather than shared vocabulary like traditional QDA."

"Using generative AI math lets you skip stats, but you lose control over what codes start your synthesis."

"If rules and stats update in real time, you could know when saturation is reached as data streams in."

Ask the Rosenbot
Cheryl Platz
Merging Improv with Design
2019 • Enterprise Community
Greg Petroff
Everything is About to Change: Software as Material
2016 • Enterprise UX 2016
Gold
Bill Scott
Lean Engineering: Engineering for Learning and Experimentation in the Enterprise
2015 • Enterprise UX 2015
Gold
Lona Moore
Scaling Design Beyond Designers
2021 • Design at Scale 2021
Gold
Dave Gray
Connection, Community, and the Future of Work
2026 • Rosenfeld Community
Patrizia Bertini
DesignOps + KPIs = Measure your Impact!
2024 • DesignOps Summit 2020
Gold
Joanna Vodopivec
One Research Team for All - Influence Without Authority
2022 • Advancing Research 2022
Gold
Luz Bratcher
This Is a Talk for Tired People
2022 • Design at Scale 2022
Gold
Max Gadney
Assessing UX jobs for impact in climate
2024 • Climate UX Interest Group
Nora Tejeda
Scaling Design Capabilities at BBVA Through a Self-service Design Model
2021 • Design at Scale 2021
Gold
Bethany Brown
Rewiring operations with service design and AI
2025 • Advancing Service Design 2025
Gold
Rima Campbell
Increase Productivity and Drive Business Impact
2024 • DesignOps Summit 2024
Gold
Dave Hoffer
UX Job Search AMA #2 with Joanne Weaver and Dave Hoffer
2025 • Rosenfeld Community
Aletheia Delivre
New Shapes and Emerging Identities for Design Ops
2025 • DesignOps Summit 2025
Gold
Nalini Kotamraju
Research After UX
2024 • Advancing Research 2024
Gold
Yolanda Rankin
Black Feminist Epistemology as a Critical Framework for Equitable Design
2021 • Advancing Research 2021
Gold

More Videos

Jon Fukuda

"Different ops functions have different incentives; aligning those incentives could make leaps and bounds in collaboration."

Jon Fukuda Jake Burghardt Jose Coronado Natalie Dunbar Denise Tilles

All the Ops: Successful cross-functional collaboration

September 10, 2025

Alana Washington

"Anglos' State of Design Ops report is a really incredible baseline about where we are today."

Alana Washington

Theme 3 Intro

October 1, 2021

Andrew Michael

"Combining the what we see in data with the why we get from user research is where true insights live."

Andrew Michael

Building a Product Insights Team

March 10, 2022

Nalini P. Kotamraju

"Research run rampant — anyone feeling empowered to do research can sometimes cause quality challenges."

Nalini P. Kotamraju

Two Jobs in One: Being a “Leader who is a Researcher” and a “Researcher who is a Leader"

March 10, 2021

Prayag Narula

"Drivers continued the practice of hanging out in physical base camp spots despite platforms saying they wanted to get rid of them."

Prayag Narula Rida Qadri

HCI 2.0: Humanity Deserves the Attention that UX Research has to Offer

March 28, 2023

Yunyan Li

"Rapid low-fidelity prototypes let us get quick feedback without stakeholders focusing on minor visual details."

Yunyan Li Anna Le Jen Kim

UX Best Practices

June 11, 2021

Shipra Kayan

"Our one metric that mattered was maximizing the number of VOС-tagged tickets solved – showing customer feedback made it to the roadmap."

Shipra Kayan

How we Built a VoC (Voice of the Customer) Practice at Upwork from the Ground Up

September 30, 2021

Dorelle Rabinowitz

"Kill them with kindness. Be professional no matter how you’re feeling."

Dorelle Rabinowitz

The Magic Word is Trust

June 15, 2018

Sarah Brooks

"Anticipating and acting on risk turns uncertainty into inspiration for informed action."

Sarah Brooks

Theme 3 Intro

December 10, 2021