Summary
In this panel, Jeff recounted the story of "Ask Alexis," a promising text-based advice service developed by a small entrepreneurial team that ultimately failed because the part-time commitment could not scale. Melissa shared a parallel experience at Intuit, where a payroll app was initially successful but killed after strategic concerns about revenue and product lineup led to misguided iterations without experimental methods. Both reflected on the emotional weight of these failures and the importance of continual iteration and alignment between product vision and business goals. Another contributor shared insights from Netflix, where the most data-backed design decision contradicted expert UX intuition, illustrating the need for cultural acceptance of being wrong and learning from data. The discussion also covered how to balance strong opinions and expertise with vulnerability and openness to change, as well as screening for people who embrace constraints and hypothesis-driven work. Bill and others emphasized prototyping techniques ranging from quick throwaway tests to evolutionary prototypes close to production, guided by the question of what needs to be learned next. Alyssa highlighted challenges around experimentation in large enterprises, stressing creative approaches to overcome internal policies and the need to measure sustained behavioral change through cohort analysis and iterative experiments. The panel concluded with practical advice about gaining advocacy for experimentation by speaking stakeholders' language and building footholds through small, successful pilots.
Key Insights
-
•
Product ideas conceived as side projects often fail due to lack of full-time commitment and proper scaling resources.
-
•
Successful experimentation requires continuous iteration beyond initial wins to integrate with business strategy.
-
•
Expert intuition and strong UX beliefs can be contradicted by user data, highlighting the importance of testing all options.
-
•
A culture that accepts being wrong and learns from data is critical for long-term product success.
-
•
Hiring for roles that embrace constraints and think in hypotheses fosters better experimentation.
-
•
Prototyping can vary from quick throwaway models to near-production code depending on the learning goal.
-
•
Testing for sustained behavior change demands longer timelines and careful cohort analysis.
-
•
Enterprise experimentation must navigate strict internal policies by building small successful internal experiments and alliances.
-
•
Overscoping and legacy infrastructure dependencies can hinder innovation labs if success is measured by features rather than business impact.
-
•
Effective evangelism for experimentation requires tailoring messaging to different organizational roles and finding internal champions.
Notable Quotes
"The level of commitment we needed to make the Ask Alexis service successful wasn’t possible given our consultants’ full-time jobs."
"Product development is a journey; you can’t stop at the high point when you’ve nailed it."
"People are lazier than they think and not as smart or hardworking as they believe."
"Just because you’ve been right in the past doesn’t guarantee future success."
"We hired people who speak product, think in hypotheses, and love constraints."
"The question with prototyping is, what do you need to learn, and what’s the least work required to learn that?"
"You have to set up your experiment framework to be able to measure sustained behavioral change if that’s the goal."
"Success breeds success, so you need to find allies and partners inside organizations to get experimentation footholds."
"If you change workflows even for a small group but don’t coordinate with affected teams, you’ll quickly get pushback."
"When you evangelize experimentation, you have to understand who you’re convincing and speak their language."
Or choose a question:
More Videos
"Each species lost is a thread untethered from the web of life that supports us all."
Alex Hurworth Bonnie John Fahd Arshad Antoine MarinDesigning a Contact Tracing App for Universal Access
October 23, 2020
"Investing in new practitioners is mutually rewarding; we learn from their fresh perspectives and reassess what we know ourselves."
Laine Riley Prokay Lisa GordonCarving a Path for Early Career DesignOps Practitioners
September 9, 2022
"Launching a design system is not a sprint, there’s no end, it’s always a continuous process."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"The most important value delivered to customers is always the qualitative value—the emotional and experiential impact."
Nathan ShedroffDouble Your Mileage: Use Your Research Strategically
March 31, 2020
"Mobile live captions can caption any sounds around the user in real time, making it a powerful accessibility tool."
Sam ProulxMobile Accessibility: Why Moving Accessibility Beyond the Desktop is Critical in a Mobile-first World
November 17, 2022
"Documentation and making findings visible even without a dedicated research librarian helps maintain program momentum."
Feleesha SterlingBuilding a Rapid Research Program (Videoconference)
May 18, 2023
"You get to those iconic signature actions when product excellence meets cultural tensions and what people need."
Neil BarrieWidening the Aperture: The Case for Taking a Broader Lens to the Dialogue between Products and Culture
March 25, 2024
"The real question is what is it about delivering the value of design at scale."
John DevanneyThe Design Management Office
November 6, 2017
"It’s not about the specific tool, but about how you expose people to the insights in a way that excites them."
Katy MogalBut Do Your Insights Scale?
March 12, 2021