UX Metrics Program

Change management • Program Management • Strategy

Context

What This Case Study Demonstrates

  • Using data from experiments to support change management

  • Hypothesis-driven design

  • Effective UX metrics

  • Program strategy, design, implementation, and management

Problem

  • Loopio had been using NPS as a gauge of customer satisfaction

  • NPS at Loopio has never elicited any more than 1% response rate, consistently elicited less than 1% qualitative feedback, and the qualitative feedback elicited is consistently vague and non-actionable: e.g. “It’s great” or “It’s terrible”

  • My hypothesis was that the low response rates, low volume of qual feedback, and low specificity in the qual feedback is a result of some inherent flaws with the questions that NPS asks:

    • How likely are you to recommend this product?

    • How can we improve?

    • What is the main reason behind your rating?

  • The first question is based on the flawed assumption that the notion of recommendation is relevant to users of B2B software, while the second and third questions aren’t specific enough to elicit high quality responses.

The Strategy

Assumption Testing

With the PMF survey as an alternative, I directed my Senior UX Researcher to validate the following assumptions:

  1. NPS questions are too vague, leading to poor response outcomes

  2. PMF questions are more specific than NPS, so they should elicit better response outcomes

  3. Business stakeholders are attached to NPS, which is all they’ve known, so they might be reluctant to lose it or replace it with something they’re unfamiliar with

My Role

  • Program strategist and manager

Dates

  • Q1’2022 to Q1’2023

Opportunity

  • Is there a replacement for NPS that nets more qual feedback and more actionable feedback?

  • The lack of actionable data from the NPS survey led the PM team to ignore the scores

  • To make any replacement for NPS worthwhile, it was imperative that the PM team buys into that replacement

  • I intentionally partnered with the Director of Product Management to discuss alternatives, and he suggested the Product Market Fit (PMF) survey*, which asks 3 questions:

    • How disappointed would you be if you could no longer use this product?

    • What’s the main benefit you derive from it?

    • How can we improve it for you?

*Caveat: both NPS and PMF are self-reported customer satisfaction surveys, so both are inherently plagued by bias; the real value of PMF is not in the score but in the qualitative feedback, which have been consistently more specific and hence actionable than NPS.

The Challenge

My challenge was two-fold:

  1. Could PMF be an effective alternative to NPS?

  2. How would stakeholders react to a change?

The path to answering the challenge consisted was 2-pronged:

  • Testing my assumptions

  • Strategically managing change with our stakeholders.

Change Management Strategy

Bringing stakeholders along by sharing the results of our assumption validation work

  • By bringing them along for the learning journey, they’re invested in the problem space and potential solutions.

Methodology

Alternate NPS and PMF survey to A/B test the performance of each survey

  • Comparing the results of the A/B tests will help us test our assumption that PMF will elicit more responses and more actionable qualitative feedback

  • Run the test over 2 quarters and report on each quarter’s outcomes to expose stakeholders PMF data in contrast with NPS data

  • 2 quarters’ worth of data will also help normalize the response rates to counteract any novelty effect from a new survey as well as help normalize the responses.

Results

Assumptions 1 and 2:

PMF will elicit higher response rates, more qualitative feedback, more actionable qualitative feedback

  • PMF consistently outperformed NPS over 2 quarters

  • Response rates to PMF were consistently 3x of NPS response rates

  • PMF qualitative feedback volume consistently 800% more than NPS

  • Quality of PMF qualitative feedback was consistently more specific and actionable than NPS

Assumption 3:

Assumption 3: stakeholders will be reluctant to give up NPS for PMF

  • The A/B test approach offered an objective, fact-based reporting of the results of the 2 survey methodologies

  • The approach presented the results of the 2 methodologies over 2 quarters, giving stakeholders time to get used to the idea of something other than NPS

  • The data spoke for itself: the PMF results was clearly more compelling

  • Ultimately, instead of UX Research having to advocate for replacing PMF with NPS, stakeholders asked us to deprecate NPS in favour of PMF