Game developers rely on metrics that help them make the right decisions to eventually drive business.
But do the gaming industry’s standard KPIs really provide the right metrics?
In his popular post, Eric Ries defined the term ‘Vanity Metrics’ and exposedthe fundamental rules of analytics that they break. Ries used the KPI “hits” as an example: it tracks technical events instead of people; it provides gross numbers rather than per-customer (one hit each from a million people is different than a million hits from one person); and it lacks built-in causality. In other words, Hits tell you what happened without revealing the cause.
Let’s now take a look at our top-line industry KPIs, such as CPI, CTR, DAU/MAU, day-1/7/30 retention and minutes played/day. These are all non-actionable indicators, which are useless unless complemented by actionable analysis. Yet, most game marketers settle for these metrics and never go beyond them.
What Happens When We Rely on Vanity KPI’s?
We recently spoke to a mobile game publisher, which found that day-1 retention in its newly launched game was 20% – well below its benchmark of 50%. What happened next was pretty predictable: Product teams blamed Marketing for the low numbers, accusing them of acquiring low-value users. Marketing, meanwhile, blamed Product for bad game design. And both teams blamed developers for creating a buggy product that drove players away. This is a classic example of vanity-KPI chaos. The publisher identified a genuine retention problem, but it did so using a KPI which has no built-in causality. This KPI did not reveal WHY the numbers were low, so responsible teams ended up pointing the finger at each other instead of looking for an effective way to fix the problem and drive new business.
Interestingly enough, when metrics exceeded the benchmark, each team was certain that the numbers were directly attributed to its latest feature design, effective campaign, or flawlessly coded app… A similar case is presented in another Eric Ries post.
Why Do Game Developers Still Use Vanity KPIs?
- They are ubiquitous and convenient benchmarks. As industry standards, they look great on management dashboards and investor presentations. They also provide an initial, quick indication about whether things are on the right track.
- They come pre-configured with off-the-shelf analytics solutions, so marketers can effortlessly showcase reports. Asking more in-depth questions requires more work, a deeper understanding of numbers. It is also frequently unsupported at all by the game itself.
So How Can We be More Actionable?
First of all, look across your organization: the problem can be in marketing, product, code or all of them combined. Looking broadly is key. Next, plan tests and events to validate your assumptions. Segment and slice rigorously to narrow down results. Use deep analysis features like cohorts, paths and granular segmentation to zoom in on the cause of your problem.
Here’s some of what we’re working on with the company described above:
Marketing: could retention correlate with player location? Use cohorts to break down retention by geo-location. You may find that retention is great in the US, however you’ve only acquired a handful of American players because of their higher cost, so their value is masked.
Product: use paths and funnels to answer product questions: where do players churn? How many tutorial steps do your players complete? What is the abandonment rate vs resources? Do players abandon the game when they are low on resources, or when full? This could indicate an unbalanced economy. Do players abandon the game following a difficult battle or after a specific level?
Code/performance: Add custom events that track performance issues like loading time between levels and frame rate in each part of the game. Catch crashes and try to correlate these with churn. We’ve seen cases where kids playing on a low-end phone they got from parents, abandoned a game at a certain level because the frame rate decreased substantially.
The Optimization Lifecycle
Always bear in mind that the best analytics tools will allow you to ask deep question about WHY things are happening in your game. We aren’t opposed to vanity KPIs. However, on their own they are only a snapshot of the situation and do not help optimize a game.