Summary
In this panel, Jeff recounted the story of "Ask Alexis," a promising text-based advice service developed by a small entrepreneurial team that ultimately failed because the part-time commitment could not scale. Melissa shared a parallel experience at Intuit, where a payroll app was initially successful but killed after strategic concerns about revenue and product lineup led to misguided iterations without experimental methods. Both reflected on the emotional weight of these failures and the importance of continual iteration and alignment between product vision and business goals. Another contributor shared insights from Netflix, where the most data-backed design decision contradicted expert UX intuition, illustrating the need for cultural acceptance of being wrong and learning from data. The discussion also covered how to balance strong opinions and expertise with vulnerability and openness to change, as well as screening for people who embrace constraints and hypothesis-driven work. Bill and others emphasized prototyping techniques ranging from quick throwaway tests to evolutionary prototypes close to production, guided by the question of what needs to be learned next. Alyssa highlighted challenges around experimentation in large enterprises, stressing creative approaches to overcome internal policies and the need to measure sustained behavioral change through cohort analysis and iterative experiments. The panel concluded with practical advice about gaining advocacy for experimentation by speaking stakeholders' language and building footholds through small, successful pilots.
Key Insights
-
•
Product ideas conceived as side projects often fail due to lack of full-time commitment and proper scaling resources.
-
•
Successful experimentation requires continuous iteration beyond initial wins to integrate with business strategy.
-
•
Expert intuition and strong UX beliefs can be contradicted by user data, highlighting the importance of testing all options.
-
•
A culture that accepts being wrong and learns from data is critical for long-term product success.
-
•
Hiring for roles that embrace constraints and think in hypotheses fosters better experimentation.
-
•
Prototyping can vary from quick throwaway models to near-production code depending on the learning goal.
-
•
Testing for sustained behavior change demands longer timelines and careful cohort analysis.
-
•
Enterprise experimentation must navigate strict internal policies by building small successful internal experiments and alliances.
-
•
Overscoping and legacy infrastructure dependencies can hinder innovation labs if success is measured by features rather than business impact.
-
•
Effective evangelism for experimentation requires tailoring messaging to different organizational roles and finding internal champions.
Notable Quotes
"The level of commitment we needed to make the Ask Alexis service successful wasn’t possible given our consultants’ full-time jobs."
"Product development is a journey; you can’t stop at the high point when you’ve nailed it."
"People are lazier than they think and not as smart or hardworking as they believe."
"Just because you’ve been right in the past doesn’t guarantee future success."
"We hired people who speak product, think in hypotheses, and love constraints."
"The question with prototyping is, what do you need to learn, and what’s the least work required to learn that?"
"You have to set up your experiment framework to be able to measure sustained behavioral change if that’s the goal."
"Success breeds success, so you need to find allies and partners inside organizations to get experimentation footholds."
"If you change workflows even for a small group but don’t coordinate with affected teams, you’ll quickly get pushback."
"When you evangelize experimentation, you have to understand who you’re convincing and speak their language."
Or choose a question:
More Videos
"Sometimes you’re the leader, sometimes you’re at the end, sometimes you’re trying to get other folks to come in."
Jennifer KanyamibwaCreating the Blueprint: Growing and Building Design Teams
November 8, 2018
"We want to create clarity because lack of clarity is why we struggle with research repositories."
Brigette Metzler Dana ChrisfieldResearch Repositories: A global project by the ResearchOps Community (Videoconference)
August 27, 2020
"Project delays are frequently due to organizational and cultural issues rather than actions of the project team."
Carl TurnerYou Can Do This: Understand and Solve Organizational Problems to Jumpstart a Dead Project
March 28, 2023
"People appreciate when you acknowledge your limitations in blending into an organization."
John Mortimer Milan Guenther Lucy Ellis Patrick QuattlebaumPanel Discussion
December 3, 2024
"Information overload with checklists on top of checklists does not create a great onboarding experience."
Dante GuintuHow to Crush the Talent Crunch
September 8, 2022
"Principle is an idea or value that gives unity and organization to a product, system, service, and human experience."
Richard BuchananCreativity and Principles in the Flourishing Enterprise
June 15, 2018
"Normally the theme leader would be talking about sleep deprivation because we’re overstimulated and excited by the talks."
Dan WillisTheme 3: Intro
January 8, 2024
"Failure is only the beginning."
Dan WardFailure Friday #1 with Dan Ward
February 7, 2025
"We don't just need to do a better job of articulating our value. We need to become more valuable."
Chris GeisonTheme Two Intro
March 28, 2023