Keywords

1 Introduction

Use of big data and analytics has become pervasive in the video game industry [1]. The adoption of analytics has been driven by the fast development of cost-effective solutions that enable basic analytics even for start-up sized game developers. The second driver of game analytics is the diffusion of the freemium business model that has created a need to accurately measure, predict, and intervene player behaviorFootnote 1. For example, Supercell, the company behind the top-grossing freemium games Clash of Clans and Boom Beach, generated revenue of 1.8 billion USD from in-game purchasing in 2014Footnote 2.

The term freemium is a combination of words “free” and “premium” [2] that describes a business model in which a service or a product is offered a for free but a premium is charged for advanced features, functionality or related products and services. Game analytics refers to applying analytics and big data in the gaming context [3]. Game analytics can be used to improve the players’ gaming experience, or to maximize in-game purchases by tweaking various aspects of the game [4, 5]. For example, the time taken for a specific task, the cost of a specific item or the power of a specific weapon [6, 7] can be optimized with game analytics.

Game analytics is a new field with limited research coverage [8]. Prior research has focused on describing methods of data gathering and analyzing [3] as well as the role of analytics in game development [9, 10]. The limited amount of prior literature can be partly explained with the fact that game developers consider analytics confidential and are thus often reluctant to share information on their analytics processes [11].

As a result, the business aspects of game analytics are not well understood. Consequently, this study seeks to investigate why and how do small and medium-sized freemium game developers use game analytics? To answer the research question we have conducted a set of in-depth interviews among freemium game developers. Our results show that game analytics is used to (1) assist design, (2) to reduce the risks associated with launching new games, and (3) to communicate with investors and publishers. The study advances the research on the business value of game analytics.

2 Literature Review

The Freemium Model. The freemium business model is extensively employed with digital products or services such as software, games, and web services. As a result, companies employing the freemium model strive to monetize their user base by striving to convert the users of the free version into paying customers [12–14].

Freemium games typically employ micro-transactions and advertising as the main monetization strategies [15]. Micro-transactions refer to buying virtual items or services that can be used and have value only inside a specific gaming environment [cf. 13]. Second, the players can be provided with the opportunity to buy “time”, either to bypass waiting enforced by the game mechanics or to skip repetitive “grinding” phases [15], or to unlock additional levels or areas of the game. Third, virtual items and benefits compared to the non-paying users can be bundled into a premium user account or a subscription [12]. Fourth, in-game advertising and potentially offering an ad-free upgraded version against a fee can be used to monetize the players. Fifth and finally, game operators can sell information about the players to third parties, typically marketers [16].

Freemium games are typically designed to engage the player immediately, because there is no initial cost causing a lock-in effect [17]. In addition, the game mechanics used in freemium games motivate the players to make in-game purchases [18]. As a result, sustained play and customer lifetime value are critical for the economic success of freemium games.

Game Analytics. Game analytics is a subset of analytics applied to the game development [19]. Analytics in turn refers to using business intelligence in the process of discovering and communication patterns from the data and using the recognized patterns in solving business problems [20].

The need for game analytics has increased as games have become more sophisticated and complex [21], and the rise of mobile gaming and the freemium business model has also had its effect. Gross et al. [22] maintain that studying online games is important from a managerial viewpoint “to understand ways that interactions while gaming can be improved, in order to make better games.”

Today’s video games, particularly online games and social media games, can collect data about almost all players’ in-game activities [4]. Game developers can use this data to obtain information about e.g. potential sources of player frustration [23]. For example, Replica Island, a game for Android devices, employed a player tracking system to identify instances where players were facing difficulties (e.g. player deaths). The whole metrics system was implemented by a one-man team at virtually no monetary cost, and has proven to be extremely valuable in identifying problems with the game design [23]. However, according to Drachen et al. [24], game analytics is far from the traditional value chain of the video game development and hence not very highly prioritized.

Prior game analytics research has employed the purchase funnel concept to illustrate the challenges the freemium model poses [3, 15, 25]. The AIDA-model (awareness, interest, desire, and action) from consumer behavior literature describes the process of new product adoption [26]. In application marketplaces where people typically download freemium games the customer can go through the process from awareness to action in seconds. However, out of the people who have downloaded a freemium game, only a fraction, e.g. 5 per cent will pay anything [15]. As a result, successful employment of the freemium model requires sustained player engagement and efficiently managing a large pool of non-paying players towards conversion during the course of the play. A freemium game can be a hit without ever being profitable, and thus monetization is essential [27].

3 Empirical Research

Data Collection. The empirical data was collected with five semi-structured interviews from experienced professionals from companies developing freemium games. The companies they worked in were based Finland. Obtaining the empirical data was challenging since finding the number of seasoned professionals with in-depth knowledge on freemium games as business in general and game analytics in particular is small. At the time of the interview, each of the five interviewees was in the process of developing a freemium game and using software to collect and analyze gameplay data. In addition, topics related to game analytics were frequently considered company confidential and hence many prospective informants declined the interview.

The theme of the interview was presented to the interviewees in advance. A semi-structured approach was selected due to the relative novelty of the research area. According to May [28] the semi-structured approach allows the interviews to be flexible and interactive.

The interviews were done face-to-face in sessions from fifty minutes to one hour twenty minutes. All interviews were recorded and transcribed by the first author. In addition, notes were taken during the interviews. Appendix 1 summarizes the data collection.

Analysis. The analysis process started already during the interviews as notes about potentially insightful themes were taken. In the first round of analysis, the interviews were transcribed and coded based on the research question [28]. Through iterations of the data, a constant-comparative method was applied to identify, elaborate, and clarify categories [29]. Emergent categories were examined within and across interviews to determine salience and recurrence. Interconnections and discrepancies between the interviews and previous literature and between different interviews were also coded. The recurrence of certain key themes and limited number of new ones emerging over coding process indicated that the data exhibited saturation.

Results. We identified two main themes that describe the different roles of game analytics for small and medium-sized freemium game developers. These are analytics as a communication tool and analytics as a decision support tool. Based on these two themes we derived an emergent theme, analytics as necessity.

Analytics as Commnunication Tool

“[Analytics] are a kind of tool for studios to justify their decisions, for example why a certain game is not ready for launch yet, because we need to improve this metric. The investors are more willing to give extra time, when they can see that in the long run the game will make more money if improved.” (P4)

As the quotation above illustrates, investors and publishers are very keen on analytics. The interviewees clearly stated that the key performance indicators have to be reported frequently to investors and publishers, and lack of improvement in these metrics will lead to questions. On the other hand, analytics help game designers make their case when they feel the launch schedule needs to be postponed as the game is promising but behind schedule.

“If I were a game publisher, I would ask teams to soft launch and provide retention and ARPDAU [average revenue per daily average users] numbers before I would invest anything.” (P3)

Freemium games are typically continuously improved through scheduled updates and developers want to see the impact of changes as soon as possible. Sizeable and potentially risky adjustments are tested out with a small subgroup of players so that regular service would not be disturbed. Similarly, before the game is launched worldwide, it can be released in smaller market (e.g. Canada or Finland). The purpose of these soft launches is to collect data to ensure that the game will be profitable.

“We used to have a thing where once a week every studio [under that publisher] would report to the headquarters in California in an hour long conference call and give a pre-formatted presentation in which the key metrics were analyzed, future plans to improve them laid out etc.” (P4)

The direct business benefits of game analytics include informed financial decisions, such as rationalizing marketing spending and budgeting for launch. The interviewees were very conscious that the video game industry has had several high-profile costly flops, i.e. projects that went overtime and over budget and made considerable losses. The interviews also indicated some of which could potentially have been avoided with the help of analytics. In addition, the interviewees recognized pressure from many stakeholders, competitors, players, and publishers on the gaming companies to adopt game analytics.

“Retention is the most important metric in the game industry” (P2)

The interviewees considered retention rate as the most important individual metric of business success in freemium game development. Consequently, all interviewees maintained that they are keen on tracking and monitoring retention figures. Retention rate refers to how players keep playing a game for subsequent sessions. A simple example of measuring retention is tracking the number of game overs per player. Typically retention was measured over a time period (e.g. 7 days, a month). The interviews also revealed that retention is a key metric for monetization as well as the basis of funnel analysis. In addition, retention is used to improve first impressions and the tutorials.

Interestingly, yet there was a consensus regarding importance and benefits from game analytics, the interviews also indicated that increasing volumes of data make it more challenging to extract relevant information from it. The interviewees wanted to keep the analytics process as simple as possible and focus on the key performance indicators (KPIs) and their trends over time and different versions.

“The most common wisdom in this free-to-play model is that if you don’t have retention you are never going to make money. Retention stems exactly from that the game itself has some interesting aspects and is in some way fun.”(P4)

In addition to retention, metrics that measure monetization and additional retention metrics, e.g. average revenue per user (ARPU), conversion rates, tutorial funnels and day 1, day 7, day 14 and day 30 retention rates, were mentioned as examples of metrics that most game developers monitor, or at least should monitor.

Additionally, customer lifetime value is used to measure to what extent the costs related to customer acquisition are covered. Customer lifetime value is an aggregate of other metrics, namely cost-per-install and ARPU, and is used to guide marketing spending. The following quotation depicts the role of customer lifetime value:

“It is mostly based on things like is it profitable to invest in marketing the game. LTV (Lifetime Value) will tell you that.” (P1)

Analytics as a Decision Support Tool

“You cannot make a good game with just analytics, it’s very challenging – [everything new] comes from the creative side” (P2)

The informants, particularly the ones who had been actively involved in actual game design, stressed quite strongly that analytics does not, and it also should not, drive game design. The interviewees maintained that analytics can support decision-making by e.g. occasionally disqualifying the intuition of the designers, but that analytics does not offer ready solutions. Further underlining that the design philosophy is not data-driven but data-supported, the interviewees stressed that game developers can and will outsource data collection, crunching and storage, but never game design. The analytics and metrics do not make the games, but they can help in making them better. The following quotation describes the role of analytics in relation to design:

“[Regarding Game Analytics] maybe the larger benefit, at least in our case, is that instead of driving development, they are used to spot errors in the code and clear design mistakes. - - It’s more about monitoring – first you design, then you code and then you monitor how well did it go.” (P5)

We observed an interesting controversy regarding the interviewees’ views on the role of game analytics. While all believed in the value of analytics and some emphasized tracking as much as possible to ensure having sufficient amount of data whenever needed, others preferred a more strict selection of metrics to avoid information overload and collecting what they referred as “vanity metrics”. These interviewees also more deliberately emphasized the importance of evaluating the benefits against the respective costs:

A recurring theme in the interviews was that the informants considered players unpredictable and that they often behave differently than the designers expected, often seemingly irrationally. Analytics can help designers to understand players’ perspective of the gaming experience. For example, one interviewee described a mobile game that had a design mechanic punishing players for erroneous behavior by reducing his/her points. An alternative solution was to end the game immediately after an error was made and force the players to start over. Somewhat counter-intuitively the latter version, which the designers felt was more “hardcore-oriented”, was more appealing to players and led to higher retention levels. As a result, game analytics proved that, against game conventions and designers’ expectations, the harder and more punishing version was more popular.

The analysis tools and methods in use were largely uniform among informants. Maybe for this reason they did not see analytics as a real source of competitive advantage. As all our informants worked for small or medium-sized companies with very limited human and financial resources to be spend on analytics, they had simply adopted ‘off-the-shelf’ analytics tools. Since most small and medium-sized game companies use a similar set of tools, the informants considered the expertise and proficiency in using tools and the ability to draw the right creative conclusions the best way to derive value from game analytics. The informants also stated that the use of analytics differs considerably between smaller and larger companies in the freemium games market.

There was a strong consensus among the informants that data allows real and accurate insights about player behavior. All of the interviewees had experience from primarily quantitative analysis and metrics. The informants’ view was that game companies seldom utilize qualitative data, as it was considered more taxing, less effective, and harder to implement with third-party solutions. The informants also stated that interviewing players about their gameplay habits can lead to misleading results since people can seldom tell what they really want and would use.

“We don’t do analytics because they are cheaper, but because they are better. – – I never trust people who say “If you would develop this, I would use it all the time”. Only when I can measure that they really use it, I will believe it.” (P3)

All informants stated that the analytics data is actively shared and communicated within the development team, but the interviews also indicated that game companies seldom share their sales numbers or metric data in public. Companies can however compare the KPIs of new launches with their prior games. This way game analytics also help guide the portfolio management of game publishers.

The data-supported design process is iterative. First, a visible problem is noticed in high-level metrics. Then the designers seek for possible causes by drilling down into the data. Specific changes are made to the game, and then the effects are measured. There is a constant loop for validating design decisions.

“[We tend to find a] – –high-level issue and then try to find one specific user experience issue that you think you change, and then you iterate the process.” (P4)

A typical way to tackle the issues that are found during the development process is to create two or more different versions of the game. These versions are then randomly distributed to players and their respective performance is measured. This is referred to as A/B testing. It cannot be used extensively for every decision since developing each alternative consumes resources and the inferior versions are a wasted effort. Acquiring relatively reliable results from the A/B tests also takes considerable time.

“When something new is added to the game, it is done in two different ways and half of the players get version A and the other half gets version B and then the metrics are compared. There may be a hypothesis behind the test, but it is more about trying to find what works and then developing the better version further, leave it as is or abandon both.” (P1)

Funnel analysis was another common analysis method used by the informants. Tutorials are a typical example of a funnel. Throughout the tutorial phase gradually fewer and fewer players reach each subsequent step. They have not necessarily paid anything for the game yet, so it is essential that they do not churn out this early. Measuring the return rate or retention of players in subsequent steps (or levels) in the game was the most common way of analyzing the quality of said funnels. When the designers note a notch in the retention, they know that there is a problem in that part of the funnel.

Analytics as a Necessity. The findings strongly suggest that game analytics plays an important role in communicating with investors and publishers as well as in supporting game development and manager’s decision-making. Hence, analytics was seen as a necessity in freemium game development. Every informant pointed out several benefits of game analytics. In fact, operating without utilizing analytics was considered “flying blind” and analytics was seen as a means to reduce the risk of failures. Interestingly, the interviewees also repeatedly stated that there had been no need to state an explicit business case to justify investments in game analytics – the benefits were that clear.

“It is clear that – especially in this free-to-play model – you cannot operate under the mentality that you just launch a game and hope for the best.” (P1)

However, utilization of game analytics is characterized by the lack of resources. This explains why game analytics was seen as something that is important but not as a differentiating factor or a potential source of competitive advantage. Table 1 summarizes the main findings.

Table 1. Summary of findings
Full size table

4 Discussion

Theoretical Implications. The findings indicate that game analytics play a pivotal role in freemium game development. The interviewees emphasized a combination of designer creativity and analytics is required to obtain the best results and repeatable results. Our results imply that retention is the most important metric to track. Game analytics was not viewed as source of competitive advantage, at least for smaller companies relying on standard third-party analytics tools, but considered more as a risk reduction tool. The possible competitive advantage of utilizing game analytics is derived from the experience and creativity of the analysts and designers.

Prior research has presented the vast possibilities and introduced sophisticated solutions for game analytics [1, 3]. However, our results imply that the analytics processes used by small and medium-sized freemium developers are rather simple. There are two opposing philosophies in data gathering. One stresses that data cannot be tracked retroactively, and tracking as much as possible ensures you have the data when you need it. The other warns about collecting “vanity metrics” that do not contribute to game design and only confuse the analysis process [see e.g. 30].

Very large volumes of data call for increasingly sophisticated and powerful analysis tools, which in turn increase the costs. Prior research has promoted utilizing a combination of data sources (e.g. analytics, interviews, biometric measuring) to improve results [10]. However, our results show that small and medium-sized freemium developers primarily focus on measuring retention and a few other key metrics. Furthermore, the data analysis methods were simple and many of the professionals cited that they do not have advanced database management skills.

Altogether, these observations align with prior research arguing that alongside being actionable, the results from analytics should be human-readable and easily interpretable [as in e.g. the model presented in 31].

Retention rates were further utilized in evaluating which alternative was better in A/B tests and improving funnels, especially tutorials. The methods used resembled the split testing used by Andersen et al. [32] to recognize the most engaging design choice from three alternatives. The small and medium-sized freemium developers also showed interest in adapting more complex analytical procedures, but did not have either the skills or resources to implement them. The interviewees did not share much about the gathering of qualitative data, which further emphasizes that the focus is on directly actionable, quantitative metrics.

Prior literature on game analytics has recognized the managerial importance of actionable results [33]. Our results add on this body of research by emphasizing the value of the predictability and decision support in analytics. In addition, our results demonstrate that for freemium game developers, game analytics provided important tools for investor communications. Third, game analytics can help in optimizing marketing spending by customer lifetime value calculations or campaigning for more time from the publisher citing promising retention trends.

Managerial Implications. Compared to game industry’s prior reluctance to adopt analytics [7], our results indicate a change in mindset. Our set of interviews among game industry professionals indicated that the benefits from game analytics are so self-evident that they are not even always explicitly stated. Since basic analytics are today available for even startup companies, the initial costs of implementing game analytics are relatively low. For example, applying basic telemetry data analysis is one the most cost-efficient way to do user research, and even one-man teams can afford it [34].

Our results imply that game analytics is viewed as a risk reduction tool. Using analytics during the development stage can decrease the risk of total failure in the launch stage. In addition, even the basic level of analytics with the standard key performance indicators can assist companies in e.g. terminating projects that are unlikely to generate sufficient revenue to become profitable.

Certain metrics have become close to industry standards. Key performance indicators that measure monetization and retention, such as ARPU, conversion rates, tutorial funnels and day 1, day 7, day 14 and day 30 retention and customer lifetime value were mentioned as examples of the key metrics. These metrics are also used to evaluate and predict the financial success of the game.

Many of the interviewees actually stated that investors require key ratios such as retention rate and ARPU when they discuss potential investments with the teams. Similarly, publishers expect results and want hard numbers to confirm the feasibility of the game concept. For example, certain leading publishers also already demand weekly reports that cite key metrics from their developers.

Limitations and Future Research. Like any other empirical research, the present study is subject to a number of limitations. First, the empirical data was collected solely from small and medium sized companies developing freemium games that were based in Finland. For example, large freemium developers such as Supercell are well known for their advanced use of game analytics. Thus, we suggest future research with a broader empirical coverage.

Second, the freemium model as well as analytics is employed in other fields such as online music as well as more traditional fields such as insurance. Future research could examine how analytics is utilized in other sectors to find best practices that could be applicable across industries.