Maximizing Mobile App Experimentation: Moving Beyond A/B Testing
Two weeks ago, I had the privilege of hosting a webinar with Jorden, Senior Product Manager from Booking.com, where he shared some eye-opening insights into mobile app experimentation. Booking.com is well-known for prioritizing A/B testing to enhance user experience and has produced many insightful articles on the topic.
One of the burning questions that emerged from the discussion was, "Where should we start when experimenting on a mobile app?" In this short blog post, we'll delve into this question, debunk the myth that A/B testing is the only experimentation method, and explore practical strategies for maximizing experimentation in mobile app development, and extending these strategies to product development on a broader scale.
While A/B testing is a powerful tool in the experimentation toolkit, it's essential to recognize that experimentation goes beyond it. Experimentation encompasses everything we do to ensure the successful release of new features and enhancements for our users.
Lessons from Spotify: Unveiling the Diversity of Experiments
A couple of weeks ago, Spotify surprised the tech community by revealing that 75% of their experiments are not A/B tests. This revelation challenges the conventional wisdom surrounding A/B testing and prompts us to ask: What are these alternative experiments, and why are they so valuable to Product Teams.
Leveraging Feature Flags
As a Product Manager, investing in feature flags is no longer a luxury but a necessity if you care about understanding how your features impact critical adoption and retention metrics, and ultimately, business success.
Indeed, feature flags provide the invaluable capability to release new features in your mobile app with the ability to deactivate them instantly with a kill switch if they underperform. Shipping a feature into your mobile app without feature flags is akin to driving a new car without a seat belt — it's risky and potentially disastrous.
Why is that so critical for Product Managers overseeing a mobile app? Because fixing a buggy feature on a mobile app is more challenging; it requires submitting a new version to the app store, awaiting approval, and relying on users to download the new version, a process that can take several days.
The good news is that the additional development costs are close to zero. Typically, implementing feature flags requires nothing more than adding a simple IF/ELSE code segment.
Embracing Progressive Rollout and Targeted Delivery
Progressive rollout and targeted delivery serve as crucial stepping stones towards full-fledged experimentation. It's not surprising that they account for 75% of the experiments run by Spotify. These methods empower Product Managers to gradually release a feature to a subset of users. Through this process, you can carefully measure the impact of the feature on key metrics, while simultaneously gathering valuable insights that shape future iterations and experimentation strategies.
This strategic approach not only minimizes risk but also maximizes the potential for success by ensuring that features are thoroughly tested and optimized before reaching a wider audience.
By utilizing progressive rollout and targeted delivery, you can effectively demonstrate the value of these approaches to your managers. Subsequently, you can confidently proceed to introduce your first A/B tests, armed with the data-driven insights gained from these initial rollouts.
Selective A/B Testing
Jim Barksdale, former CEO of Netscape, famously stated:
“If we have data, let’s look at data.”
“If all we have are opinions, let’s go with mine.”
Allow me to rephrase this in my own words:
“If we have clear evidence, let’s not run an A/B test.”
“If all we have are opinions, let’s proceed with an A/B test.”
Not every idea / hypothesis necessitates A/B testing. Reserve this method for situations where uncertainty surrounds a feature's potential success or when it aligns with strategic objectives. Conducting A/B tests for every product change, as if you lack understanding of your end user, isn't always necessary. Instead, consider leveraging qualitative methods like usability tests and user interviews, which can offer nuanced insights into user behavior and preferences.
It's important to recognize that A/B testing is just one tool in the experimentation toolkit and shouldn't be viewed as a fail-safe strategy against risk. Sometimes, prioritizing speed over exhaustive learning is necessary. Therefore, focus on conducting meaningful A/B tests and allocate your time and resources accordingly. Balancing quantitative and qualitative methods is essential for effectively validating ideas and facilitating successful product development.
Conclusion
In 2024, A/B Testing Experimentation must be at the core of mobile app development, fueling innovation and enhancing user satisfaction. By adopting a comprehensive approach to experimentation and effectively balancing feature flagging, progressive and targeted delivery, and A/B testing, product teams can unleash the complete potential of their product development. Each plays a distinct role in ensuring that changes are introduced in a controlled manner to mitigate risks.