Summary
In this panel, Jeff recounted the story of "Ask Alexis," a promising text-based advice service developed by a small entrepreneurial team that ultimately failed because the part-time commitment could not scale. Melissa shared a parallel experience at Intuit, where a payroll app was initially successful but killed after strategic concerns about revenue and product lineup led to misguided iterations without experimental methods. Both reflected on the emotional weight of these failures and the importance of continual iteration and alignment between product vision and business goals. Another contributor shared insights from Netflix, where the most data-backed design decision contradicted expert UX intuition, illustrating the need for cultural acceptance of being wrong and learning from data. The discussion also covered how to balance strong opinions and expertise with vulnerability and openness to change, as well as screening for people who embrace constraints and hypothesis-driven work. Bill and others emphasized prototyping techniques ranging from quick throwaway tests to evolutionary prototypes close to production, guided by the question of what needs to be learned next. Alyssa highlighted challenges around experimentation in large enterprises, stressing creative approaches to overcome internal policies and the need to measure sustained behavioral change through cohort analysis and iterative experiments. The panel concluded with practical advice about gaining advocacy for experimentation by speaking stakeholders' language and building footholds through small, successful pilots.
Key Insights
-
•
Product ideas conceived as side projects often fail due to lack of full-time commitment and proper scaling resources.
-
•
Successful experimentation requires continuous iteration beyond initial wins to integrate with business strategy.
-
•
Expert intuition and strong UX beliefs can be contradicted by user data, highlighting the importance of testing all options.
-
•
A culture that accepts being wrong and learns from data is critical for long-term product success.
-
•
Hiring for roles that embrace constraints and think in hypotheses fosters better experimentation.
-
•
Prototyping can vary from quick throwaway models to near-production code depending on the learning goal.
-
•
Testing for sustained behavior change demands longer timelines and careful cohort analysis.
-
•
Enterprise experimentation must navigate strict internal policies by building small successful internal experiments and alliances.
-
•
Overscoping and legacy infrastructure dependencies can hinder innovation labs if success is measured by features rather than business impact.
-
•
Effective evangelism for experimentation requires tailoring messaging to different organizational roles and finding internal champions.
Notable Quotes
"The level of commitment we needed to make the Ask Alexis service successful wasn’t possible given our consultants’ full-time jobs."
"Product development is a journey; you can’t stop at the high point when you’ve nailed it."
"People are lazier than they think and not as smart or hardworking as they believe."
"Just because you’ve been right in the past doesn’t guarantee future success."
"We hired people who speak product, think in hypotheses, and love constraints."
"The question with prototyping is, what do you need to learn, and what’s the least work required to learn that?"
"You have to set up your experiment framework to be able to measure sustained behavioral change if that’s the goal."
"Success breeds success, so you need to find allies and partners inside organizations to get experimentation footholds."
"If you change workflows even for a small group but don’t coordinate with affected teams, you’ll quickly get pushback."
"When you evangelize experimentation, you have to understand who you’re convincing and speak their language."
Or choose a question:
More Videos
"When teams lose a sense of agency, they act strange, cutting corners and racing to get something out the door."
Standardizing Product Merits for Leaders, Designers, and Everyone
June 15, 2018
"If I don’t have a map, I can’t see patterns or apply context-specific gameplay."
Simon WardleyMaps and Topographical Intelligence (Videoconference)
January 31, 2019
"There’s a gap between intentions and impact; humility is needed to close it."
Sandra CamachoCreating More Bias-Proof Designs
January 22, 2025
"I felt on edge around Jeff and physically depleted after our interactions."
Darian DavisLessons from a Toxic Work Relationship
January 8, 2024
"Using ChatGPT allowed me to elevate from a researcher to a strategist by increasing my visibility and trust in the company."
Fisayo Osilaja[Demo] The AI edge: From researcher to strategist
June 4, 2024
"Empowering maker culture invites everyone to be part of a collective endeavor far beyond any individual."
Uday GajendarThe Wicked Craft of Enterprise UX
May 13, 2015
"We formed a small core task force empowered to make decisions to avoid delays from formal reviews."
Davis Neable Guy SegalHow to Drive a Design Project When you Don’t Have a Design Team
June 10, 2021
"People were very adverse to changes because a small 0.5% conversion increase meant millions in revenue."
Eniola OluwoleLessons From the DesignOps Journey of the World's Largest Travel Site
October 24, 2019
"Product managers feel design is a black box; they want visibility, transparency, and accountability on timelines."
Aurobinda Pradhan Shashank DeshpandeIntroduction to Collaborative DesignOps using Cubyts
September 9, 2022
