Summary
In this panel, Jeff recounted the story of "Ask Alexis," a promising text-based advice service developed by a small entrepreneurial team that ultimately failed because the part-time commitment could not scale. Melissa shared a parallel experience at Intuit, where a payroll app was initially successful but killed after strategic concerns about revenue and product lineup led to misguided iterations without experimental methods. Both reflected on the emotional weight of these failures and the importance of continual iteration and alignment between product vision and business goals. Another contributor shared insights from Netflix, where the most data-backed design decision contradicted expert UX intuition, illustrating the need for cultural acceptance of being wrong and learning from data. The discussion also covered how to balance strong opinions and expertise with vulnerability and openness to change, as well as screening for people who embrace constraints and hypothesis-driven work. Bill and others emphasized prototyping techniques ranging from quick throwaway tests to evolutionary prototypes close to production, guided by the question of what needs to be learned next. Alyssa highlighted challenges around experimentation in large enterprises, stressing creative approaches to overcome internal policies and the need to measure sustained behavioral change through cohort analysis and iterative experiments. The panel concluded with practical advice about gaining advocacy for experimentation by speaking stakeholders' language and building footholds through small, successful pilots.
Key Insights
-
•
Product ideas conceived as side projects often fail due to lack of full-time commitment and proper scaling resources.
-
•
Successful experimentation requires continuous iteration beyond initial wins to integrate with business strategy.
-
•
Expert intuition and strong UX beliefs can be contradicted by user data, highlighting the importance of testing all options.
-
•
A culture that accepts being wrong and learns from data is critical for long-term product success.
-
•
Hiring for roles that embrace constraints and think in hypotheses fosters better experimentation.
-
•
Prototyping can vary from quick throwaway models to near-production code depending on the learning goal.
-
•
Testing for sustained behavior change demands longer timelines and careful cohort analysis.
-
•
Enterprise experimentation must navigate strict internal policies by building small successful internal experiments and alliances.
-
•
Overscoping and legacy infrastructure dependencies can hinder innovation labs if success is measured by features rather than business impact.
-
•
Effective evangelism for experimentation requires tailoring messaging to different organizational roles and finding internal champions.
Notable Quotes
"The level of commitment we needed to make the Ask Alexis service successful wasn’t possible given our consultants’ full-time jobs."
"Product development is a journey; you can’t stop at the high point when you’ve nailed it."
"People are lazier than they think and not as smart or hardworking as they believe."
"Just because you’ve been right in the past doesn’t guarantee future success."
"We hired people who speak product, think in hypotheses, and love constraints."
"The question with prototyping is, what do you need to learn, and what’s the least work required to learn that?"
"You have to set up your experiment framework to be able to measure sustained behavioral change if that’s the goal."
"Success breeds success, so you need to find allies and partners inside organizations to get experimentation footholds."
"If you change workflows even for a small group but don’t coordinate with affected teams, you’ll quickly get pushback."
"When you evangelize experimentation, you have to understand who you’re convincing and speak their language."
Or choose a question:
More Videos
"You need strong regional players with negotiation skills to navigate the split loyalties in global teams."
Adam Cutler Karen Pascoe Ian Swinson Susan WorthmanDiscussion
June 8, 2016
"Knowing your subject matter as a UX leader is just as important as knowing your craft."
Peter MerholzThe Trials and Tribulations of Directors of UX (Videoconference)
July 13, 2023
"If you haven’t designed who your teams are and who your players are, expecting people to comply with standards won’t work."
Lisa WelchmanCleaning Up Our Mess: Digital Governance for Designers
June 14, 2018
"We need to rethink how our cities are designed and function."
Vincent BrathwaiteOpener: Past, Present, and Future—Closing the Racial Divide in Design Teams
October 22, 2020
"The greatest sign of success for a teacher is just to be able to say the children are now working as if I did not exist."
Brenna FallonLearning Over Outcomes
October 24, 2019
"Working from home during the pandemic is hard because it’s fun only when you can actually leave your home."
Tricia WangSpatial Collapse: Designing for Emergent Culture
January 8, 2024
"Marital status mattered because car buying decisions often involve family members, not just the individual."
Edgar Anzaldua MorenoUsing Research to Determine Unique Value Proposition
March 11, 2021
"We want to build teams with diverse skill sets so we can create a full picture during the knowledge creation phase."
Designing Systems at Scale
November 7, 2018
"Most product teams work linearly, but systems thinking captures the real-world complexity of moving forward and sometimes stepping back."
Erin WeigelGet Your Whole Team Testing to Design for Impact
July 24, 2024
