Championing experimentation in product development.

Parker Stanley, Senior Product Manager in Washington, D.C.
26 June 2020

Baby Boomers are slowly taking a back seat, it’s tech-savvy Millennials and Gen Z’ers who are now becoming the primary target audience as they come into their prime spending years. Their combined spending power is estimated to be nearly $3 trillion in 2020. But as these new generations are slowly taking over, the customer expectations, preferences, and behaviors are shifting dramatically. After all – they’ve been growing up surrounded by tech, apps, and the customer experience-led giants like Amazon or Apple. 

The purchasing trends are changing – the experience of buying is now just as important as the actual goods or services themselves. So, when we define a digital product, we’re not just talking about products in the traditional sense, but rather the entire end-to-end experience – from the moment customers first encounter the brand up until they receive, use, and finish using their product or service. What does this increased focus on the holistic experience mean for your business? And how can you make sure the experience you’re delivering works? The answer is experimentation.

Product Manager: a champion of experimentation

For a long time in the digital space, the Product Manager’s role was to fully understand their customers and develop a product strategy that aims to fulfill users’ core wants and needs. While this purpose has remained relatively stable, the how part has evolved radically as experimentation and optimization became an integral part of the product life cycle.

Now, we can determine user preferences through the use of experimentation – all in real-time. In this context, we’re referring to testing done on live web experiences by presenting page variations to different groups of site visitors. This can manifest itself in many ways, from granular changes like evaluating forty different shades of blue within a user interface to wholesale changes such as manipulating the order of steps in a checkout flow. Regardless of the scope, effective experimentation can provide us with a wealth of information.

It’s our responsibility as Product Managers to champion these efforts, as there are several benefits we can gain from experimentation:

  1. Avoiding building the “wrong” product

  2. Iterating and optimizing

These two core benefits can be realized by integrating experimentation into different stages of the product development cycle. Where and how we add experimentation into our process, however, can have a significant impact. Let’s dive in. 

How to avoid building the “wrong” product

A cardinal sin of product management is to build features or products without understanding whether users want, need, and would use it first. If we give into temptation, our teams simply become feature factories. Cranking out endless amounts of capabilities that are never really used and that provide little to no user value. It’s important to remember that customers don’t care about the features themselves but rather how the features benefit them and how they improve their experience with the product. 

To avoid this, we can experiment early, prior to launch. At this stage, we can determine if a feature is worth building at all. One example of this is a “painted door” test. This can be accomplished in many ways, but the simplest execution is by adding a call to action into a new flow or feature that didn’t exist before. On interaction, the user is then presented with a message confirming their interest. That interaction can be used to gain an understanding as to whether or not this would be a feature users will care about. 

One word of caution for the apprising Product Manager. Embracing experimentation at this stage in product development can be extremely challenging for organizations, as more often than not these experiments are not successful in the traditional sense. Here, we should look to reframe what ‘failure’ means and instead view these failed experiments as a method to learn more about our users and provide direction for the product roadmap.

Incremental iteration

So, we’ve determined we have a product or feature that our users want. What's next? Once we’ve delivered a product to our users, we have to look to the future – towards improvement and optimization.

Now, we can add experiments into the delivery process as we release increments of our product. At this stage, our focus is on quality and iteration. With each increment, we can explore new concepts and identify what to prioritize to provide users with the greatest amount of value. Successful Product Managers are those who constantly refine and adapt their product. Experiments focused on optimization can springboard teams to do just that.

Without experimentation, we’re limited in our ability to truly evaluate the benefit of a change to our current experience. Relying on methods such as pre/post analysis can be misleading due to uncontrollable factors outside of the scope of the change such as market conditions, promotions, customer preferences, and behaviors. An experiment can mitigate these variable conditions. The ability to gather real-time feedback removes guesswork and gives us immediate insight into the best path forward for our users.

Through iterative experimentation, we can also gain insights into the customer journey as a whole. For example, we ran a multivariate experiment with a recent client, where we introduced five different potential next step actions on a particular product page. This experiment allowed us to determine which action was a typical user’s most logical next step in their journey. This, in turn, led us to re-evaluate our shopping funnel holistically. By applying this to your own digital experience, you’re able to develop additional hypotheses or carry these learnings into future development efforts.

Industry leaders such as Microsoft and Google have embraced this shift and now run upwards of 10,000 digital experiments annually. somehow manages a whopping 25,000. These companies understand the value that experimentation can bring to their organization and products.

If we want to succeed in the digital marketplace, the role experimentation plays is crucial. By integrating experimentation into our workflows and our products, we can develop a true customer-centric culture – a tried and true playbook for success.