Summary
In this panel, Jeff recounted the story of "Ask Alexis," a promising text-based advice service developed by a small entrepreneurial team that ultimately failed because the part-time commitment could not scale. Melissa shared a parallel experience at Intuit, where a payroll app was initially successful but killed after strategic concerns about revenue and product lineup led to misguided iterations without experimental methods. Both reflected on the emotional weight of these failures and the importance of continual iteration and alignment between product vision and business goals. Another contributor shared insights from Netflix, where the most data-backed design decision contradicted expert UX intuition, illustrating the need for cultural acceptance of being wrong and learning from data. The discussion also covered how to balance strong opinions and expertise with vulnerability and openness to change, as well as screening for people who embrace constraints and hypothesis-driven work. Bill and others emphasized prototyping techniques ranging from quick throwaway tests to evolutionary prototypes close to production, guided by the question of what needs to be learned next. Alyssa highlighted challenges around experimentation in large enterprises, stressing creative approaches to overcome internal policies and the need to measure sustained behavioral change through cohort analysis and iterative experiments. The panel concluded with practical advice about gaining advocacy for experimentation by speaking stakeholders' language and building footholds through small, successful pilots.
Key Insights
-
•
Product ideas conceived as side projects often fail due to lack of full-time commitment and proper scaling resources.
-
•
Successful experimentation requires continuous iteration beyond initial wins to integrate with business strategy.
-
•
Expert intuition and strong UX beliefs can be contradicted by user data, highlighting the importance of testing all options.
-
•
A culture that accepts being wrong and learns from data is critical for long-term product success.
-
•
Hiring for roles that embrace constraints and think in hypotheses fosters better experimentation.
-
•
Prototyping can vary from quick throwaway models to near-production code depending on the learning goal.
-
•
Testing for sustained behavior change demands longer timelines and careful cohort analysis.
-
•
Enterprise experimentation must navigate strict internal policies by building small successful internal experiments and alliances.
-
•
Overscoping and legacy infrastructure dependencies can hinder innovation labs if success is measured by features rather than business impact.
-
•
Effective evangelism for experimentation requires tailoring messaging to different organizational roles and finding internal champions.
Notable Quotes
"The level of commitment we needed to make the Ask Alexis service successful wasn’t possible given our consultants’ full-time jobs."
"Product development is a journey; you can’t stop at the high point when you’ve nailed it."
"People are lazier than they think and not as smart or hardworking as they believe."
"Just because you’ve been right in the past doesn’t guarantee future success."
"We hired people who speak product, think in hypotheses, and love constraints."
"The question with prototyping is, what do you need to learn, and what’s the least work required to learn that?"
"You have to set up your experiment framework to be able to measure sustained behavioral change if that’s the goal."
"Success breeds success, so you need to find allies and partners inside organizations to get experimentation footholds."
"If you change workflows even for a small group but don’t coordinate with affected teams, you’ll quickly get pushback."
"When you evangelize experimentation, you have to understand who you’re convincing and speak their language."
Dig deeper—ask the Rosenbot:
More Videos
"Join the design ops Slack channels to interact closely with speakers and peers as if you were sitting next to them in person."
Bria Alexander Louis RosenfeldWelcome
September 8, 2022
"A truly disruptive idea will often fail the mass-market test, so talk to non-customers as well as existing ones."
Mike OrenWhy Pharmaceutical's Research Model Should Replace Design Thinking
March 28, 2023
"We should think of AI as our ancestor encountered fire: managed well it’s a force for good, deployed recklessly it will burn us."
Jodi ForlizziDesign and AI innovation
June 5, 2024
"Design ops is changing was our first theme, and today it’s all about practical design ops."
Bria AlexanderOpening Remarks
October 3, 2023
"People remember how you made them feel more than what you said."
Michael WeirMixed Methods and Behavioural Science (Videoconference)
May 26, 2023
"If we don’t brand ourselves, others will, and others could be wrong."
Molly FargotsteinMultipurpose Communication & UX Research Marketing (Videoconference)
September 12, 2019
"The Silicon Valley dream is not the reality we are living in Latin America in terms of technology, culture, resources, mindset, and people."
Verónica Urzúa Jorge MontielThe B-side of the Research Impact
March 12, 2021
"You can’t collaborate without or communicate upfront shared context."
Husani OakleyTheme Two Intro
June 6, 2023
"Unlike tools like XD or Figma, we don’t just show specs, we ship actual components with hooks and accessibility built-in."
George Abraham Stefan IvanovDesign Systems To-Go: Reimagining Developer Handoff, and Introducing App Builder (Part 2)
October 1, 2021
