Testing and Experimentation Tools
This video is featured in the UX metrics & analytics playlist.
Summary
Join us for a deep dive into testing and experimentation tools at the UXR Tools Summit. Design for Impact author Erin Weigel will lead a broad discussion on the category, and will also be joined by UXtweak, who will reveal their vision for the future of experimentation, showcase innovative capabilities. Ideal for UX, design, and market researchers eager to explore how emerging tools are transforming decision-making, optimization, and evidence-based product design.
Key Insights
-
•
AI-generated follow-up questions in usability tests do not uncover novel user issues as effectively as human moderators.
-
•
Participants tend to get frustrated with AI moderators due to repetitive or obvious questioning.
-
•
Context is crucial for effective research moderation, which AI currently lacks.
-
•
Synthentic participants are useful for brainstorming but cannot replace real user research.
-
•
Democratization of research increases access but risks diluting research quality without expert involvement.
-
•
AI features that save researchers time by handling repetitive tasks are promising and more practical.
-
•
AI moderation should operate as a human-in-the-loop assistant, not a full replacement.
-
•
Bias in AI and lack of diversity representation remains a critical challenge currently unaddressed.
-
•
UX Tweak focuses on researcher-centered tools, expert support, and end-to-end UX research platform.
-
•
There is healthy skepticism about AI fully simulating unpredictable human behavior in usability testing.
Notable Quotes
"Testing tools can help us bring the body language of the internet in front of us to see what people are doing."
"AI cannot substitute the research expertise achieved through multiple interviews and synthesis by humans."
"There was no meaningful difference between AI follow-up questions and static researcher questions in uncovering usability issues."
"AI follow-up questions led to frustrated participants because it asked highly obvious questions and ignored detailed context."
"If all tools try to be for everyone, they risk serving no one properly."
"AI should be a force multiplier handling repetitive tasks, not a replacement for human judgment."
"Synthetic participants shouldn't be used as a methodology but are great for cold starting research."
"Human moderation will always have a place because humans are unpredictable and random."
"We never train AI models on your data, and all AI features are strictly opt-in."
"There is a huge space for improvement in AI features for UX research if these have any future."
Or choose a question:
More Videos
"You don’t feel like it’s a huge corporate burden company, nor a crazy fast startup. It feels authentic and committed."
Kurdin Bazaz Liz Rytting Alex KarrCulture, DIBS & Recruiting
June 10, 2021
"Triangulation is key — compare multiple metrics to understand why scores move the way they do."
Jack MoffettUX Metrics That Matter and The Future of our Design at Scale Conference: A Community Conversation
September 22, 2022
"The shooter used census data to target the location to inflict the most damage on Black people."
Kyria Stephens Marlon KernerPower to Heal: Civic Design in the Aftermath of Tragedy
November 17, 2022
"Teams with different thought worlds should start with who they are, what their roles are, and their passions inside and outside work."
Lin NieWhen Thought-worlds Collide: Collaborating Between Research and Practice
March 10, 2021
"Context retention is tough; Siri and Google Assistant completely lose prior contexts after one action."
Greg NudelmanDesigning Conversational Interfaces
November 14, 2019
"We can over-index on capability and opportunity, but motivation is critical for non-researchers to engage in research activities."
Trisha TerharEmpathizing with the Empowered: Non-Researcher Responses to Democratization
March 10, 2022
"We absolutely need feedback loops so engineers and designers can influence the design system and keep it relevant."
Charles Lee Jennie YipBuilding a New Home for the Atlassian Design System
October 22, 2020
"You don’t have to have a moral to the story; just capture it for now and maybe the lesson comes later."
Steve Portigal Susan Simon-Daniels Tamara Hale Randolph Duke IIWar Stories LIVE! Q&A-Discussion
March 30, 2020
"I’m still early on in measuring accessibility impact but tying early bug fixes to ROI is promising."
Deanna Washington Bria Alexander Jon Fukuda Saara Kamppari-Miller Farid SabitovConnecting the Ops: Plenary Panel and Closing Circle
September 9, 2022
Latest Books All books
Dig deeper with the Rosenbot
How can research and operations teams collaboratively evaluate new UXR tools for usability and operational fit?
What role do small wins play in building credibility and trust for UX in healthcare environments?
How does systems thinking help identify leverage points for improving rural maternal health?