Improving your data strategy by getting rid of vanity metrics

We’ve all seen headlines or reports that scream about ‘300% user growth’, or ‘100 million downloads’. While it definitely has impact, and seems impressive on the surface - think about it in more detail. What is it really saying?

Take user growth for example. That 300% seems a lot less impressive if you started out with 5 users, but more impressive if you started out with 500,000. Ratios can be a useful measure, but only if given context (i.e. you have an idea of what the starting point was).

But what about that 100 million downloads, you say? It seems like solid metric, telling you how many have downloaded and used your app. But consider this - downloads does not equal users. 75% of users abandon an app after using it only once. Out of those 100 million downloads, how many users are you actually retaining? With cumulative figures like this, the number can only go up, which doesn’t tell you anything about the current situation.

These are called vanity metrics, and are not a measure of real growth and revenue.

What is a vanity metric?

A vanity metric is something that is fairly easily manipulated, and therefore not a good measure of true indicators of success such as engagement or revenue/profit. A lot of businesses fall victim to these due to not understanding what the underlying role of data is and using default ‘easy’ metrics, while also wanting to focus on what looks good.

However, the true mark of a vanity metric is that it is not actionable. You cannot use this data to decide what actions you should take to improve your KPIs.

Let’s take the example of users. ‘How can users be a vanity metric? They’re our customers!’ I hear you say. But that’s the point - users are not customers. Let’s demonstrate with an example.

Looking at the graph below, you can see that the Wiredcraft website had a spike in traffic on December 10. ‘Awesome’, you think - let’s do more of whatever we did on that day and keep building up our user numbers.

However, this metric in itself does not tell us anything about the source of the traffic. Was it something we did - an email newsletter, a WeChat article? Were we linked to by another website? Was there a spike in searches for a random keyword that we just happen to rank highly for? Without having a way to replicate this spike, we can’t hope to grow our users out of it.

But more importantly - these users are not customers. In order for a user to become a customer and create value, they need to complete your goals (i.e. convert). Until then, they are just site visitors. Case in point - we had no corresponding spike in goal completions on December 10, despite the 200% increase in users. In fact, looking at our goal completions, you wouldn’t even know there was a spike in users on that date.

We could have spent time and resources looking into the source of this traffic, trying to build on it and increase our daily average users, but it would have been wasted effort. The spike in traffic is just a distraction from looking into what’s truly important.

To summarize, vanity metrics are:

• easy to measure, but do not actually say anything on a deeper level
• easily manipulated without a lot of effort
• not true indicators of how your business is doing
• incapable of indicating where you should focus efforts

Whereas true metrics are actionable, helping you to inform your strategies and make data-driven decisions to help grow your business.

How can you avoid vanity metrics?

The basic vanity metrics are things like users, pageviews, and downloads. Now you know not to fall prey to those, but there are many more that are just as insidious.

The best way to avoid these is by having a solid understanding of what your goals are and what metrics will help you to measure your progress towards these. We work with our clients to build their data strategy using a measurement plan, which breaks down objectives into goals and metrics so that we have an objective view of success.

Going through this exercise will show you the metrics that will:

• help build strategies that will benefit the business and help you achieve your goals
• help evaluate whether your actions have had a positive effect on the business
• have a significant impact on the business if there is upwards or downwards movement

Much more actionable than a vanity metric, understanding the purpose behind a true metric and how it linked to solid business outcomes will help you to focus your efforts and resources on where they will have the most impact.

Transform your metrics

Here’s a cheat sheet of some common examples of vanity metrics and a more insightful counterpart that could be used. Obviously, depending on your business model, some might be more relevant than others, but hopefully this helps you to reconsider the way you think about data!