This is the second of a series of articles. Follow-up articles will be released semiweekly. If you’re curious about the next one, or just want to have a chat, follow me on Twitter. š
In the first article of this series, we took a look at how to set a good goal. Starting out with a clear and well-defined goal is important: It helps us stay on track, makes sure we’re all pulling in the same direction, and serves as the anchor for our efforts. But in order to know whether our actions really contribute towards reaching our goal, we need to make our progress measurable.
Define Your KPI(s) Ahead Of Development
I’ve seen a lot of Product Managers skip this step during the planning phase, only to struggle later when trying to evaluate the impact of a product change. We need to know how to measure our progress towards our goal before we start development so that we can:
- start with a deep understanding of how our planned change will affect our product if everything goes well
- validate that our progress is measurable with the tools we have at hand
- avoid shifting the goal posts later
The last point seems rather obvious, but is important nevertheless: It’s tempting to change your goal after the fact if it becomes clear that you can’t reach (or measure) it. I’ll describe various approaches to handling under-performing product changes in the next post.
Focus On The User
Let’s continue with the user story from the previous post:
AS A customer looking to rent a van I WANT TO immediately see what license each van requires SO THAT I can easily tell which ones Iām allowed to drive.
You can tell we at least did an OK job writing this story because it answers the fundamental question to finding the right KPIs: “What will change for the user?“
In this case, it’s clear that we want our users to more easily find vans they’re allowed to drive.
There’s a tendency to think about the effects on the product before considering the effects on the user (if at all), but it’s important that we don’t get this backwards. Optimizing a KPI without understanding and targeting the desired change in user behavior leads to us focusing on vanity KPIs instead of choosing ones that indicate a real improvement in product performance.
Actionable KPIs Over Vanity KPIs
Vanity KPIs are dangerous: They lead us to make important decisions based on insights that look good, but are meaningless. If we stay focused on the user, we can avoid falling into this trap. In our van example, a (sadly) commonly used vanity KPI could look like this:
The number of weekly van rentals increases by X.
This example shows why vanity KPIs are so easily accepted: Of course we want to sell more van rentals, that’s the entire point of our business! But it’s not the primary point of our planned change. What we really want to achieve is this:
The ratio of van category page visitors leaving the page without interaction (bounce rate) decreases by X%.
This is a good start, but we may want to be more even more precise. Instead of measuring how many users leave the page without interaction, we could also measure how many do interact with it in our desired way:
The ratio of van category page visitors clicking on a product tile increases by X%.
Why are both of these KPIs better than our vanity example? Here some of the most important aspects to look out for:
They are influenced by fewer external factors
In our example, how many users ultimately decide to rent a van is influenced by many factors unrelated to our planned change, e.g. price, availability, traffic quality and seasonal effects. Our negative example above would be heavily affected by all of these, possibly to a point where we can’t tell if an improvement is due to our product change.
They can’t be achieved by increasing cost
Vanity KPIs can oftentimes be influenced by throwing money at them. If you haven’t already exhausted the potential of your primary marketing channels, increasing your marketing spend would be an easy way to increase your total sales volume. But it would benefit neither the customer, nor the business, as you didn’t solve the problem preventing your confused customers from finding the vans they need. So you’d continue to drag along a share of confused users who churn quickly and may never return due to their bad experience.
They are tied closely to the user behavior change
The further removed from the desired user behavior your KPI is, the fuzzier your learnings will be. In our example, our primary goal is not for our users to rent a van, it is for our users to find a van they are interested in. We would, of course, love if they ended up signing up for a rental, but our planned change has very limited influence on that. Measuring the ratio of users clicking a product tile is much closer to the desired behavior change (“I found something I’m interested in!”) compared to a vanity KPI measuring the end of our funnel.
Note: Ratios vs. Absolute Numbers
This is more of a learning from the first three aspects, not so much a rule in itself. You may have noticed that our vanity KPI targets an absolute number (# of van rentals per month), while the other two examples target a ratio. There’s a good reason for this: Absolute numbers rarely make for good feature KPIs. In fact, they rarely make for great business KPIs when isolated: Having millions of visitors doesn’t make for a successful business if none of them buy anything, click ads or create revenue in any other way.
Whenever we are tempted to pick an absolute number as our target KPI, a small alarm bell shold go off and we should ask ourselves if this is really the best way to measure the impact of our planned product change.
Now that we know how to find the right indicators and avoid vanity KPIs, we can focus on actually estimating the anticipated change in our target KPIs. The next article in this series will be focusing on that, as well as data consistency and common estimation mistakes.
How do you approach finding the right KPIs for a product change? Reach out for a chat over a (virtual) cup of coffee. š
This is the second of a series of articles. If you’re curious about the next one, or just want to have a chat, follow me on Twitter.
Do you like or dislike this post? Please give me feedback via the thumbs up or thumbs down button: