How to build your product analytics stack.

Nicola Bertazzoni, Senior Product Manager
10 September 2018

Winning in a product-led world

Some of the best digital companies have grown from zero to billions of users with very little marketing investment. Companies high in the S&P 500 such as Facebook, LinkedIn and Amazon have instead built their success by obsessively focusing on the user experience and creating the best product on the market. They revolutionised the traditional model of launching and growing a business by implementing a continuous product evolution process that allows them to constantly improve engagement, conversion and virality metrics. 

Businesses across all industries can learn from this. Through rapid technology innovation and an ever-changing competitive landscape, it’s now essential for businesses to continuously strive for the best product experience possible or customers will seek a better offering elsewhere. Implementing an agile, fast-paced product development process allows companies to quickly adapt to user behaviours and systematically respond to the demands of the market in real time.

One crucial factor in achieving this is building up a strong and suitable set of analytic tools that can feed the product development process with meaningful user behaviour insights.

The product analytics stack

How do you make sure you are shipping the right new features? How do you confirm they are moving the needle on relevant metrics? How do you understand what users really want? How do you increase the pace at which you test and release without compromising quality? 

Your product analytics stack is a collection of tools that work together to answer these questions. The goal is not just to help your team understand what happened in the past, but to provide the evidence to guide them in improving the future experience. Selecting the right product analytics stack for your circumstances is an essential prerequisite to implementing rapid and effective product development.

We have identified six components that we believe are needed in a comprehensive stack for modern product teams:

Core stack:

  1. Data pipeline and warehousing (e.g., Snowplow, Tealium)

  2. Product experience analytics (e.g. Mixpanel, Amplitude, Heap)

  3. A/B testing and feature toggle (e.g. Optimizely, Adobe Target, Oracle Maxymiser)

Extended stack:

  1. User guidance and engagement (e.g., Urban Airship,

  2. Heatmaps and session replay (e.g. Appsee, Clicktale, Sessionstack, Hotjar)

  3. User feedback and surveys (e.g. Usabilla, Wootric, Delighted) 

In the last couple of years many different tools have emerged in each of these areas, so with a little bit of research you can certainly find a solution that is tailored to your company’s specific needs, scale and maturity. Some initial suggestions are listed above. 

Let’s have a quick look at each layer of the stack:

#1 Data pipeline and warehousing

Think about your analytics stack as something dynamic rather than static. The tools you pick now may need to evolve as your company develops in size, processes, culture and goals.

Unfortunately some tools and technologies can limit your future flexibility and ability to scale. For example, as your reporting needs become more sophisticated, your teams may face barriers such as having to learn a vendor’s proprietary query language, or just not being able to perform a certain analysis because data is structured the wrong way. In addition, fees for some off-the-shelf tools may increase rapidly, their pricing structure being very attractive at low volumes but very expensive once your product achieves scale.

Most problems (other than cost) come down to the fact that each tool collects and structures data in different ways, making it difficult to switch between platforms when you need to without losing some of your data. This is why we recommend decoupling data collection from your other analytics tools and selecting a foundation layer that ensures comprehensive data gathering and portability without creating lock-ins.

The ideal foundation layer will allow you to define data pipelines to collect events across any device and channel (web, mobile, email, ads, push, IoT, even call centres, etc.). It will structure the data in the most appropriate way and allow you to integrate other internal sources if needed, such as CRM systems, and also third party sources (ad networks, market data, etc.).

#2 Product experience analytics

This is the core component of your product analytics stack. Traditional marketing solutions like Google and Adobe Analytics are just not fit for purpose for an effective, rapid process on continuous product improvement. Luckily, there are many modern event-driven analytics platforms that allow you to track pretty much everything that happens within your product. 

In general, all product experience analytics tools offer out-of-the-box reports to cover key areas such as funnels, cohorts, conversion goals, user segments, flows and paths. However, they differ in particular features that may be important to you in understanding user behaviour or in advanced reporting capabilities. Some off-the-shelf tools combine a good set of predefined reports with a simple intuitive interface, allowing product managers to use them easily from day one. This can be important, as one of the hardest tasks is adopting tools that actually get used across the entire team. Choosing such a solution can democratise access to product data and so help decentralise decision making, empowering the people who are closer to customers. 

There are also off-the-shelf platforms with more sophisticated and flexible reporting capabilities, sometimes coupled with predictive / machine learning features that help teams identify emerging user behaviour patterns and segments. These are ideal for highly data-driven industries and products.

Of course, in companies with dedicated teams of analysts and data scientists you can go all the way and adopt a visualisation tool like Tableau or Looker. This option offers various advantages and in general is a more sound long-term strategy for high growth companies. However, you’re committing to a far higher investment of internal resource.

Overall, the key point is to consider the balance of need, effort and capability and find the right point on this scale for your business at its current stage of development.

#3 A/B testing and feature toggle

This is one of the methodologies at the heart of user-focused product development. When solving a new challenge product teams may develop different variations of a solution and face questions such as “Which version best addresses the user need and moves the needle?”, or simply “Will this change have a good or bad impact on conversion?”.

With an A/B testing tool teams can set up tests, roll out the feature for a limited but statistically relevant subset of users, and study how those users behave compared to a control group. They will be able to use data to settle debates and identify the best option based on evidence, rather than HiPPOs.

#4 User guidance and engagement

One of the most interesting directions in the industry is the shift from passively tracking user behaviour to actively trying to influence it. This layer of the stack empowers product teams to take action fast to solve user problems, without even changing the product.

Let’s say the team has identified a specific group of users that struggle to complete a certain task, or a cohort at risk of churn. With a guidance and engagement tool they can immediately implement a solution targeting these users, choosing among a set of options and channels such as push notifications, in-app messages, feature walkthroughs and popups. These tools can also deploy nuanced automated workflows of messages and notifications triggered by specific events, aiming to enhance user experience and conversion.

#5 Heatmaps and session replay

Behavioural reports consisting of numeric data are not always enough to understand the root cause of user experience problems. Qualitative user testing sessions help collect deeper insight, but they don’t allow you to replicate context and usage patterns at scale. To solve this problem - and in general whenever the user experience is complex - a heatmap tool comes in really handy, providing visual insight that helps decode exactly where things are going wrong. 

In addition, some tools allow you to record user sessions and use algorithms to classify them, identifying the most interesting ones for you to review so you get to the heart of issues quickly. (Recordings may be limited to journeys where sensitive data is not exposed.) 

#6 User feedback and surveys

Modern user feedback solutions cover a wide set of qualitative and quantitative techniques to understand user sentiment, perceptions, pain points and friction, and are valuable in identifying improvement opportunities.

These tools can collect the voice of the customer, NPS, feedback and reviews at scale throughout all your digital experience touch-points. These metrics are a great addition to the product analytics mix and can help identify problems with current journeys and guide you  towards stellar user experiences.

Scaling your stack

While we believe an effective product analytics strategy should consider all the layers discussed above - and include at least the core layers - companies at different stages of maturity will need to prioritise different layers and will have different requirements for tools within them in order to reach their short to medium term goals.

On the one hand, companies at the beginning of their analytics journey may pick less complex tools that offer a core set of features and an easy UX/UI to encourage teams to access them regularly and use them in meaningful ways. All-in-one tools that integrate various features across different layers may be appropriate and help reduce the bills.

On the other hand, companies with a more mature data-driven culture will get real value from advanced tools that give more control over the way they store, aggregate and analyse data, picking a foundation layer such as Snowplow and best of breed solutions in each layer of the stack they need. Examples could be Amplitude, Mixpanel or Heap for product experience, Hotjar or Crazyegg for heatmaps, Usabilla for user feedback, etc.

To conclude, the stack needs to evolve with your company, growing in depth of capability as you progress with scale to increasingly complex user experiences and subtle judgements of product value. At each stage of your product journey, it’s important to make sure the tools you rely on can provide the answers to your product team’s most pressing needs.