Data-driven method development and evaluation for indie mobile game publishing

Abstract

With the emergence of mobile distribution channels, the traditional game value chain has produced new changes, leading to the emergence of the mobile value chain. Independent (Indie) game developers can upload their games directly through third-party app stores and publish them themselves. However, many indie game developers have issues with game publishing, especially updating the new version, promoting the market, and forecasting revenue for their games. This paper aims to provide a method to guide indie mobile game developers with mobile publishing. This new method mainly focuses on addressing the main challenges from the indie game developer’s side. The method includes a new concept of mobile game publishing logic and an online analysis tool along with the guidelines. It shows how to collect and analyze data and guide new version updates, marketing promotion, and revenue forecasts. In practice, the method was provided to six indie game companies and guided their mobile game publishing, and related data were collected and analyzed for evaluation. Based on the survey and interview results, the usefulness, usability, and confidence in the method were positive, and the method improved the indie game developers’ mobile game publishing and benefited their game business.

1 Introduction

With the development of mobile games, and especially the appearance of distribution channels such as the App Store and Google Play, the traditional retail game value chain has produced some changes, leading to the emergence of the mobile value chain [14], as shown in Fig. 1. It has become more flexible for game companies to publish mobile games, and the Newzoo shows mobile game revenue accounted for 52% of the global game revenue in 2021 [35]. There has been an emergence of smaller game studios more and more focused on mobile game development, and most of these are independent (indie) game developers. Indie game developers have limited game development resources compared to big game companies, and usually they are organized as smaller development studios [17]. Indie game developers have started to upload their mobile games and publish themselves through distribution channels without asking for help from publishers. As for the online value chain, indie game developers even can publish their games through websites. However, due to the lack of experience in game publishing, it is difficult for them to successfully put their games on the market.

Fig. 1

Comparison with different game value chains [14]

There is a disjunction of three different types of independence used to define indie games, namely financial, creative, and publishing independence. Financial independence means that the developer funds their own game with no external finances from any third party. Creative independence means that the creative process is flexible, and the developer can create whatever they want. Publishing independence means self-publishing, where the developer is also the publisher [16]. Compared with financial independence and creative independence, helping indie game developer publish their game is the meaningful part. However, most indie developers underestimate how much work is required for game publishing. Attilio [3] points out that most indie game developers are unaware of the many tasks behind getting a game to market that they have to handle themselves, and most indie game developers who are good at game development lack experience in game publishing [19]. They have particular issues with user acquisition, transferring users into loyal and paying players [31]. Therefore, there is a demand for a systematic method to facilitate indie game publishing.

In addition, with the development of the game industry, research in the field of games is also deepening. Many interdisciplinary knowledge areas and theories have been used to promote the development of the game industry. Business Intelligence (BI) has been applied in game development and game research which is referred to as game analytics [10]. Su et al. [44] classified the game analytics used in the game industry based on the game value chain and pointed out the potential research gaps. As for indie game publishing, related research shows that most indie game developers are willing to publish their games themselves, and they use third-party analysis tools such as GameAnalytics and Firebase to help them publish [42]. However, they still have challenges using these tools to solve practical issues, such as new version update evaluation, marketing promotion, and revenue forecast, because the available tools have limited functions that have not covered these aspects. Although third-party analysis tools provide much data with dashboards, they still have issues about what data to collect to address these specific problems, how to analyze the data, and how to use the analysis results to guide the actions [42].

In short, although mobile game publishing has become more flexible, indie mobile game developers still face challenges. They usually underestimate the workload of mobile game publishing and are short of the required knowledge and skills. Besides this, the available game analysis tools such as iTunes Connect and App analytics provide only basic functions, and these tools with limited functions make it hard to address specific issues such as the new version update, marketing promotion, and revenue forecast. This paper provides a data-driven method to guide indie mobile game publishing. The rest of this paper's structure is as follows. Section 2 introduces related research on game analytics and publishing. Section 3 discusses the research method and research process. Section 4 describes the method with the analysis tool architecture and applications. Section 5 presents the evaluation criteria and five evaluation steps along with the interview and survey data collection and analysis. Finally, in Section 6, the conclusions are made and future research is discussed.

2 Related work

The concept of BI was first introduced by Hans Peter Luhn in 1958 [2], and BI can be applied in many fields to improve efficiency and save costs through effective decision-making. Davenport and Harris [9] define BI as incorporating the collection, management, and reporting of decision-oriented data and the analytical technologies and computing approaches used on the data. The BI system combines data collection, storage, and management with analysis tools to provide decision-makers with complex internal and external information [34], and game analytics identifies and communicates meaningful patterns used for decision-making in the game industry [10]. Game analytics is a source of BI in game development [12]. As for the use of game analytics, it can identify in-game balancing issues [26], visualize the player's movement path on the map [33], and reduce game development costs [23]. As for mobile game analytics, Drachen et al. [11], [p. 1] point out that "researchers struggle to keep up with the work being done in a field where the industry has access to both more resources and more data. In essence, the field is in its infancy, and the available knowledge is heavily fragmented." This is also the main reason this research focuses on mobile game analytics and plans to provide the data-driven method with an online analysis tool for indie mobile game publishing.

Regarding the game analytics used for indie game developers, Koskenvoima and Mäntymäki [27] conducted in-depth interviews with a group of small and medium-sized game developers about why they use game analytics. Their results showed three main reasons for using game analytics. First, game analytics can be used to assist game development and game design. Second, the game developer can effectively reduce the risk of game development and publishing. Third, through game analytics the developers can negotiate with investors and publishers. However, this research only shows why indie game developers use game analytics instead of exploring the main challenges for the indie game developer and how to use game analytics to address these main challenges because they need to develop new skills and competencies for their game publishing. In addition, Mäntymäki et al. [29] point out that indie game developers tend to rely on basic analytics tools and a few key metrics. However, the lack of interest in the qualitative data that have been reported highlights that the resources allocated to the analysis processes are limited. In addition to this, indie game developers are interested in adopting more sophisticated analytical procedures, but they lack the resources to implement them [29]. In particular, skills in data management are a bottleneck for adopting advanced analytics. Limpach [28] pointed out that the publishing challenge for indie game developers is that further updates have to be developed after the initial game release, and this requires implementing analytics to monitor gameplay, retention, and monetization. As a result, the main challenges in terms of marketing promotion, new version update evaluation, and game revenue forecast were the focus of the work for this article.

As for game publishing, Broekhuizen et al. [6] point out that many researchers have argued that online technologies will revolutionize creative industries and allow entrepreneurial content producers to publish independently and to appropriate the entire value of their creativity, thus making incumbent publishers obsolete. In addition, as the costs of reproduction and online distribution of content become negligible, publishers' scale-based resource advantages that create entry barriers are largely eliminated. O'Donnell [36] points out that corporate consolidation of game studios under umbrella publishing companies has severely limited the ability of developers to gain access to the networks necessary for creating games or distributing them. Kerr [25] points out that small game companies often rely on cooperation with publishers to get their games to market. The past decade has witnessed an explosion of small indie game companies attempting to take advantage of new tools, platforms, and market channels to launch new types of games. Karlsen [24] found that publishers usually talk about the free-to-play (F2P) games, and even if game developers choose to handle game publishing themselves, they still need support in game publishing. Limpach [28] points out that it is challenging to deliver one universal explanation about publishing because individuals have their own subjective opinion of publishing, and game publishing sums everything up. Traditionally, publishing has been handled by separate actors on the market, and publishers have handled advertising, marketing, and distribution [38].

As for the game publishing model, Moreira et al. [32] used the ARM (acquisition, retention, and monetization) funnel model to do the basic analysis of game publishing, as shown in Fig. 2. The model describes the entire customer life cycle from awareness to purchase and the percentage of customers at each stage of the whole life cycle through the purchase channel [15]. However, this ARM funnel model was initially used for social games, which have social network integration compared with mobile games [1]. The model visualizes how gamers pass through a funnel in three stages – acquisition, retention, and monetization. The acquisition of social games mainly comes from viral channels, which are more valuable to game developers because they help reduce user acquisition costs. The K factor is a key indicator of game virality, and it can show the number of players who join the game through viral sources. Thus, the ARM funnel model mainly focuses on the viral K factor promotion effect, especially if K-LTV (Life Time Value) is greater than CAC (Customer Acquisition Cost). However, the ARM model only describes the player's life cycle from awareness to purchase. It cannot show how the game content affects acquisition, retention, and monetization, and thus the relationship between the new players from the channels, new content, and revenue is hard to capture by the ARM funnel model.

Fig. 2

ARM funnel model [1]

The main reasons for the ARM funnel model extension are as follows. First, the ARM funnel model was initially designed for social games, not for mobile games, so it mainly focuses on the viral acquisition by inviting friends from social platforms. Second, the ARM funnel model is derived from the AIDA (Attention, Interest, Desire, and Action) model [18], and it only reflects the three states related to the game players. Third, the ARM funnel model is relatively abstract, and it ignores the existence of more player behaviors with different scenarios beyond just acquisition, retention, and monetization. Third, the ARM funnel model ignores the game content itself, especially how the new content affects acquisition, retention, and monetization during game publishing. In practice, the ARM funnel model is extended from social game publishing to mobile game publishing by adding the game content into the existing ARM funnel model and introducing more metrics and analysis to guide indie mobile game publishing in a structured way. “Data-driven” is an adjective used to describe a process or activity, and a data-driven process is based on data instead of being driven by intuition or personal experience [39]. Thus the data-driven method presented here enables indie game developers to use data analysis to make the right decisions concerning the publishing of mobile games.

In short, game analytics can be recognized as a powerful tool to aid decision-making. However, based on the initial literature review, there are still potential research gaps, especially for game publishing analytics. In particular, how to improve player analytics for different game content, how to perform analytics on new version updates, how to make the most of marketing promotion using game analytics, and how to perform revenue forecasts still need further research.

3 Research methodology

Design Science Research (DSR) is a problem-solving paradigm that seeks to enhance human knowledge by creating innovative artifacts [50]. Hevner et al. [21] point out that design science creates artifacts that satisfy a given set of functional requirements using the knowledge expressed in the form of constructs, methods, models, and instantiations. They discuss how design science addresses research by building and evaluating artifacts designed to meet the identified business needs. The design science in information systems research describes the conceptual framework for understanding, executing, and assessing information system research [21]. This research is based on extending and optimizing the existing model and providing a new artifact that can address the main challenges of indie game developers and guide their mobile game publishing. Because the DSR can be used to create artifacts that satisfy a given set of functional requirements using the knowledge expressed in the form of methods, it is suitable for this research.

For the application of the DSR method in the research work, artifact creation was an iterative process that included four phases, as shown in Fig. 3. In the first phase, a tentatively designed model was proposed to optimize the ARM funnel model. This phase was enacted in this study by combining two indie game companies' projects based in Sweden to demonstrate and verify the performance of the new version update [43]. In the second phase, based on the feedback collected from the previous validation and demonstration results, the artifact was re-designed with guidelines to help more indie game developers publish games. This phase brought in another two indie game companies based in Sweden, and we worked with them and addressed their real problems, including marketing promotion evaluation [45] and revenue forecast [46]. In the third phase, based on the previous guideline application, more helpful feedback was collected through interviews with the four indie game companies based in Sweden. The artifact was iterated according to the suggestion from these indie game companies, especially the indie game companies' suggestion that it would be nice to develop a platform that provides the conclusions of the analysis just by clicking a button. Based on these suggestions, the free-to-play analysis platform (F2PAP) was designed and developed [41] to provide solutions for new version update evaluation, marketing promotion, and revenue forecast. In the fourth phase, the F2PAP analysis platform was integrated as a new data-driven method, including mobile game publishing logic, related metrics and analyses, and guidelines. Six new indie game companies based in Asia were introduced to using the method for guiding their indie mobile game publishing. In addition to this, qualitative and quantitative research methods were involved in the evaluation, and interviews and surveys were used to collect data and then form the final research conclusions based on the analysis results.

Fig. 3

The DSR method application in this research

In short, the ARM funnel model was initially designed for social games, not for mobile games, and it ignores the game content itself, especially how the new content affects acquisition, retention, and monetization during game publishing. To help indie game developers understand the mobile game publishing logic and to guide them in mobile game publishing using game analytics, this research followed the DSR research process, extended the ARM funnel model for mobile game publishing, and developed and iterated the method. Six new indie game companies based in Asia were introduced to the method to guide their indie mobile game publishing. The method was evaluated by their indie mobile game projects with different case studies, especially how well it can guide indie mobile game publishing and address the main challenges with publishing.

4 Data-driven method development and application

In order to solve the main challenges faced by indie game developers during their mobile game publishing, we propose a new method based on the previous artifact iteration that can guide indie developers in using game analytics to publish their mobile games. This method is composed of the following components.

• The new concept of mobile game publishing logic introduces the game content as input to the ARM funnel model, and it recognizes mobile game publishing as two inputs – the new content and the channel – and one output, the revenue.

• Based on the new concept of mobile game publishing logic and different player behaviors, such as the new player, active player, inactive player, paying player, and churn player, more metrics are introduced to measure the changes.

• Two different kinds of data analysis are provided, including comparative analysis and correlation analysis with Pearson’s correlation coefficient for the new version update and marketing promotion evaluation. One time-series prediction model, the Holt-Winters model [8], is involved in the revenue forecast.

• A data analysis platform called the F2PAP was developed based on this new concept of mobile game publishing logic, especially the two inputs of the channel and new content and the output of revenue, which guide indie game publishing.

The F2PAP was developed using R Shinyapp and deployed as an online game data analysis tool. The architecture of the F2PAP is shown in Fig. 4. It includes the hierarchy, components, and applications with analysis and prediction methods. Indie game developers can access the F2PAP at any time and from any place, and they can quickly get the analysis results by uploading the corresponding data to the analysis platform using different data templates. This platform is used to address the main challenges during indie mobile game publishing. It can be used for different F2P mobile game publishing modes, including the free download and in-game payment mode. As shown in Fig. 5, the publishing theory list in the F2PAP includes the main task of mobile game publishing, mobile game publishing logics, and player behavior changes with different scenarios, such as new player, active player, inactive player, paying player, and churn player [10]. The mobile game publishing theory shows what mobile game publishing is and what is related to it, while the mobile game publishing logic shows how mobile game publishing works. Based on the mobile game publishing logic, three functions have been developed, including the evaluation of the new version update, the evaluation of marketing promotion, the revenue forecast, and data templates are provided for the three functions. The indie game developers only need to collect game data based on the different data templates, upload it to the F2PAP, get the analysis result, and then take related action based on suggestions.

Fig. 4

F2PAP Architecture

Fig. 5

The F2PAP with the new concept of mobile game publishing logic

In practice, the F2PAP is deployed online for indie game developers to use. It can be accessed from this link https://sutor-yan.shinyapps.io/F2PAP/.

4.1 Function 1: New version update evaluation

The new version update evaluation function was developed and integrated because of the requirements of indie game developers. For example, an indie game company that has just launched its mobile game needs to keep releasing new versions during the mobile game publishing process to attract and maintain game players and to extend the mobile game lifetime. However, it is hard for them to know if players favor their new content.

Combined with the new concept of mobile game publishing logic, the new version update can be recognized as new content input. The relationship between the new content input, in-game players, and revenue output should be determined. Some new metrics are used to measure the new version update performance, including the install change rate, DAU (Daily Active User) change rate, inactive player change rate, paying player change rate, churn player change rate, and revenue change rate. In the F2PAP, indie game developers can evaluate the results using the new version update function. They only need to collect and upload game data according to the new version update template, and they can directly get the analysis results and suggestions about the new version performance.

As shown in Fig. 6, the new version update evaluation presents the analysis results from one of the indie game companies after using this function in practice. The changes in key metrics can be obtained from the comparative analysis before and after the new version update, and the relationship between different variables can be determined by correlation analysis. In fact, the new version update evaluation mainly focuses on explaining the differences before and after the new version release and comparing key metrics. Besides this, correlation analysis is also involved, which can help indie game developers deeply analyze the game version update metrics using Pearson's correlation coefficient. By comparative analysis and correlation analysis, the indie game developers can comprehensively evaluate the new version update and identify potential problems behind the data related to the new version update, such as if the new version update has issues with the new player acquisition, retention, or monetization. Suggestions are then provided based on the analysis results in order to guide the indie game developers in improving their performance. These suggestions include encouraging the in-game player to invite their friends, sending push notifications to inactive players during the new version update, attracting them to play, providing the first payment rewards for new players, and encouraging them to pay.

Fig. 6

New version update evaluation

4.2 Function 2: Marketing promotion evaluation

The marketing promotion function needed to be developed and integrated because this is the main challenge for indie game publishing that needs to be addressed. This can be implemented based on the method because marketing is the channel input in the mobile game publishing logic. The main role of the channel is to input high-quality new players through marketing promotion. In this method, the channel quality is related to marketing promotion and new players' performance in the game. For example, the higher the proportion of new players converted to paying players, the better the quality of the channel. Through the method, the indie game developers can evaluate the quality of the different channels not only by the channel metrics such as CPC (Cost Per Click), CPI (Cost Per Install), eCPM (Effective Cost Per Mille), CVR (Conversion Rate), CTR ( Click Through Rate), and Budget, but also by the in-game metrics such as Install, DAU, Paying Players, and Revenue. In the F2PAP, indie game developers can get the evaluation result by using the marketing promotion function. They only need to collect and upload game data according to the marketing template and they can get immediate analysis results and suggestions about marketing performance.

As shown in Fig. 7, the marketing evaluation presents the analysis results from one of the indie game companies after using this function in practice. The changes in key metrics can be obtained from the trends and comparative analysis during the marketing promotion, and the relationship between different variables can be determined by correlation analysis. As for the marketing promotion, the comparative analysis compares the different channel performances during the marketing promotion and finds the best channel for user acquisition and monetization. Correlation analysis is used to identify all of the metrics' relationships with the marketing promotion by Pearson's correlation coefficient and to identify potential problems with marketing promotion. Optimization suggestions are provided based on the analysis results and guide indie game developers to improve their marketing promotion, such as spending a small budget to try different marketing channels first, choosing and locking in the high-quality channels, replacing marketing assets, and doing additional testing for new player acquisition.

Fig. 7

Marketing promotion evaluation

4.3 Function 3: Revenue forecast

The revenue forecast function was developed and integrated because of the requirements of indie game developers. For example, one of the indie game companies launched its mobile game in more than two years. However, during the game publishing process, they faced issues forecasting their game revenue, significantly how to predict the revenue for the following year and to set it as a benchmark to drive their game publishing. Based on the revenue forecast, they can plan how much of the marketing budget needs to be used for user acquisition. It was also valuable to set revenue forecast as a benchmark to evaluate game publishing performance. Thus, the method can help indie game developers get an internal revenue benchmark by making a reliable prediction based on the Holt-Winters prediction model [8], which uses historical revenue data as the input, and the predicted revenue can be obtained by using the revenue forecast function. In the F2PAP, the indie game developer can get the revenue forecast directly by uploading their historical revenue data according to the revenue template and clicking the revenue forecast tag.

As shown in Fig. 8, the revenue forecast shows the prediction results from one of the indie game companies after using this function in practice. It used the last two years’ monthly revenue data to predict the following year’s monthly revenue. Then it reminded them to set the revenue forecast as a benchmark and to compare it with the real revenue. If the real revenue could not meet the benchmark, it also proposed suggestions based on data analysis to drive indie mobile game publishing. These actions included releasing the new content to incentivize active players to pay, recommending hardcore players with their favorite items to generate more revenue, and providing rewards and discounts to paying players to avoid churn.

Fig. 8

Game revenue forecast

In short, most indie game developers involved in this research have used third-party analysis tools such as GameAnalytics and Firebase to help them with game publishing. However, they still have issues using these tools to solve practical problems – such as new version update evaluation, marketing promotion, and revenue forecast – because the available tools have limited functions that have not covered these aspects. Although third-party analysis tools provide dashboards, they still have issues with what data to collect to address these specific problems, how to analyze the data, and how to use the analysis results to guide further actions.

5 Evaluation of the data-driven method

According to Venable et al. [52] and Hevner et al. [21], evaluation is one of the critical aspects of DSR. During this stage, the designed artifacts need to be evaluated to determine if they fulfill the purpose for which they were created. In order to evaluate the proposed method, six different indie game companies based in Asia were invited to use the method for their indie mobile game publishing. The selection of sample companies for evaluation had the following criteria. First, the game companies were small in scale and relied on limited resources to develop and publish games that meet the standards of indie game companies. Second, the selected companies were all developing and publishing F2P mobile games. Third, the selected game companies were interested in this research and were willing to use this method to guide their mobile game publishing. More than a dozen indie game companies were approached, but only six of them met the criteria and accepted the invitation. The data in the evaluation process were collected through interviews and surveys with indie game developers from these companies. These data were further analyzed after collection and were used to measure the usefulness and usability of the method. The evaluation of this method sought to test the method with actual mobile game publishing cases and to investigate the changes in confidence in handling the mobile game publishing before and after using the method. As shown in Fig. 9, the evaluation process was divided into five steps.

Fig. 9

The five steps of the evaluation process

The first step was to send a pre-survey to the six indie game companies and collect basic information such as the indie game developers’ opinions about game publishing, their main challenges in game publishing, and their confidence in handling mobile game publishing. The second step was to send the method, including the F2PAP game analysis tool and the corresponding guidelines, to the indie game companies and let them know how it might solve their issues. The method included the publishing theory, the main task of mobile game publishing, the mobile game publishing logic, and descriptions of player behavior changes. Based on publishing theory, three functions were provided to address mobile game publishing challenges, namely the new version update evaluation, the marketing promotion evaluation, and the revenue forecast. The third step was to conduct an ex-ante survey, Survey 1, which was mainly to understand the first impressions and feelings of indie game developers on the method before using it. The fourth step was to conduct an ex-post survey, Survey 2, after indie game developers used the method, and this survey mainly focused on the feedback collection after using the method in their mobile game publishing. The fifth step was to conduct the interviews with six indie game companies and make the conclusion based on the data analysis. As shown in Table 1, some employees from six indie game companies participated in this research, and 58 participants finished the pre-survey, 56 participants finished Survey 1, and 50 participants finished Survey 2.

Table 1 The selected game companies for three surveys

Regarding the evaluation criteria, Tsakonas et al. [49] proposed an evaluation model using usability and usefulness, as shown in Fig. 10.

Fig. 10

The evaluation model [49]

Usability defines the quality of the interaction between the user and the system, including effectiveness, efficiency, and satisfaction, and it helps the user manipulate a system efficiently and exploit all the available functionalities. Usefulness is about users and content components and relevance to user tasks and requirements. Usefulness is how a specific information item will serve the user's information needs. Tsakonas and Papatheodorou [48] point out that usefulness is an extension of the concept of relevance, a research focus of user and information behavior studies, while usability studies investigate the quality of interaction between the user and the system. Hevner et al. [21] identified different attributes that should be evaluated in an artifact, including functionality, completeness, consistency, accuracy, performance, reliability, usability, and other relevant quality attributes. Mazzarol and Reboud [30] defined five elements that form the cycle of management activity for a small business related to the five Es of Efficiency, Effectiveness, Efficacy, Ethicality, and Elegance. Rubin and Chisnell [40] point out that different authors have different views about usability, and usability is a quality that many products possess, and the goals are typically defined in terms of measurable attributes. Bevan et al. [4] demonstrate the standards related to usability and provide examples of measures that can be used in usability evaluation, and they also define what should be included in usability evaluation reports for usability tests. Brooke [7] points out that if an artifact's usability is defined by the context in which that artifact is used, the measures of usability must be defined by that context. In short, usability can be recognized as the extent to how specified users can use a system, product, or service to achieve specified goals with effectiveness, efficiency, and satisfaction.

Based on the evaluation model and related definitions, some criteria are set to analyze the feedback, including usefulness, effectiveness, efficiency, satisfaction, and confidence in handling mobile game publishing, and these can be compared before and after using this method. In order to understand more details about the indie game developers using the method, the core members from each indie game company were selected, such as their CEOs or publishing managers, to conduct one-on-one in-depth interviews. The interviews with more than twenty questions were all recorded, transcribed, and coded by MAXQDA. The conclusions were drawn by combining the results of the surveys and the interview data.

In the test cases, the specific implementation of this method mainly focused on testing it in mobile game publishing with different indie game projects. This method is integrated into the F2PAP with the new concept of mobile game publishing logic, the three functions of new version update evaluation, marketing promotion, and revenue forecast, together with the related data collection templates, data analysis method, and guidelines. In order to address the main challenges from the indie game developer side, they only need to collect the related data according to the data template and upload it to the F2PAP and get the analysis result automatically. Then, based on the analysis result, they can choose from the associated suggestions.

From the perspective of research ethics, all collected data complied with the GDPR, which expanded the scope of data protection [47]. In practice, all of the in-game data were collected by third-party analysis tools that followed the GDPR and only collected the research data without any personal information. Due to these third-party analysis tools having limited functions that did not cover the aspects of new version update evaluation, marketing promotion evaluation, and revenue forecast, the collected data needed to be uploaded to F2PAP for analysis. Each participant was confirmed to understand the purpose of involvement and agreed to the conditions for participation. All the participants had the right to stop the interview or withdraw from the research at any time. Before the interview, the author also checked if they were agreeable to recording the interview and got permission to transcribe and use the related materials and data anonymously. Heimo et al. [20] define the various business logics related to video game development, especially F2P games that need to balance profit and ethics, and this research satisfies the ethics of the video game business logic. First, this research is more about helping indie game developers evaluate and analyze the game publishing performance, such as new version updates performance, marketing performance, and revenue forecasts, rather than directly teaching them how to make money by choosing different business logics. Second, the suggestions provided are not all related to making a profit but focus on the player experience in order to optimize the game experience and provide better services to players. For example, the new beginner's guidance advises facilitating players in playing the game better. Third, player feedback is encouraged so as to collect data to guide the new version update, such as the players' ratings and comments. Finally, the method can guide indie game developers to make decisions on marketing budgets. If the evaluation result was not good, which means the players may dislike the game, they can consider stopping the marketing promotion and focusing on optimizing their games. In fact, the metrics involved in the method are mainly focused on channel, new content, revenue and retention, new player, active player, and churn player. The method can guide indie game developers to optimize their games and provide better game services to potential players.

This research targets indie game developers. Therefore, those who want to use this method should be indie game developers from indie game companies doing mobile game publishing themselves. Besides this, the method mainly focuses on guiding the publishing of F2P mobile games, which means that indie game companies should have F2P mobile games. In the specific evaluation, six indie game companies were involved in validating the method, as shown in Table 2. These indie game companies were all based in Asia, and they all handled the F2P mobile game business themselves and required guidance for game publishing, so they were interested in this research.

Table 2 The selected game companies for the evaluation

In short, in order to conduct the artifact evaluation with more indie game developers, a selection was made of the potential indie game companies. The game studios were recruited through purposeful sampling, and the sampling targeted those interested in using this method and who were planning to use it for their mobile game publishing. The method was sent to them through the purposeful sampling of such subjects, and the related data were collected for the evaluation.

5.1 Interview data analysis

For the interview evaluation, more than twenty interview questions were designed that included indie game developers' game publishing information and the method of using feedback. These questions targeted the new concept of mobile game publishing logic, usefulness, efficiency, effectiveness, and satisfaction. This research sought to establish how indie game developers use the method in practice and how they think about the new concept of mobile game publishing logic. Thus, the interview questions had four subsections. First, some basic information about the indie game companies and indie game developers was required, such as names, roles, and numbers of employees. Second, game publishing information was collected, such as how they publish games and what they think needs support during their game publishing. Third, the new concept of mobile game publishing logic was evaluated, such as how they think about it compared with the ARM funnel model. Finally, after using the method, their thoughts and feelings were collected. Most questions were related to usefulness, efficiency, effectiveness, and satisfaction.

The empirical data were collected through six interviews with indie game companies based in Asia. Regarding the professional backgrounds of the interviewees, four of them were CEOs of game companies and two of them were publishing managers. The semi-structured interview method was adopted in the specific interviews, making the interviews more flexible [13]. Each interview was recorded and transcribed. The qualitative analysis method was adopted for the interview analysis, which involves interpreting interviews, observations, and documents to find substantively meaningful patterns and themes [37]. The analysis process included four parts. First, all the interviews with the six indie game companies were recorded and transcribed into corresponding documents. Second, based on being familiar with the interview data, MAXQDA was used to assign preliminary codes to the interview data and to look for patterns and themes among the codes, and then all interviews were coded until saturated. Third, the themes and codes were reviewed again and categorized and similar codes were grouped. Finally, the final analysis report was released, and some quotations are cited below to validate the data.

5.1.1 The new concept evaluation

The first category, new concept evaluation, referred to the mobile game publishing logic based on the ARM funnel model. The indie game developers were asked related questions, for example, on how they viewed the new concept of mobile game publishing compared to the ARM funnel model.

First, indie game developers mentioned that the ARM funnel model was primarily designed for social games. However, social games and mobile games are different in user acquisition and distribution channels, so the new concept of mobile publishing logic is necessary to help the indie game developer sort out the key points for mobile game publishing.

“It can help us sort out the key points in the whole mobile game publishing business. It is the important step to do the follow-up work. The publishing logic of this platform is probably more aligned with the focus of mobile game publishing than the ARM funnel model.” (CEO of Company 3)

Second, the new concept of mobile game publishing logic recognizes mobile game publishing as having two inputs and one output. It provides a clear map for them and lets them realize the key points of mobile game publishing and what they need to do during mobile game publishing and helps them better understand mobile game publishing.

“The mobile game publishing logic is simple. It is easier to learn. There is an input for market and an input for new version update, and then you can look at the change in player behavior, active users, inactive users, and paying users. The output is revenue, and the logic of the platform is simple and clear.” (CEO of Company 1)

Third, among the other third-party analysis software such as GameAnalytics and Firebase, the participants did not see a similar solution and analytics platform that could provide the same analysis functions and give guidance for F2P mobile game publishing.

“I know that the ARM model is mainly for social games, but this platform is quite cool. I cannot see another platform like this one that can provide a very good analysis for our free-to-play games as well as guidelines for us to use. The knowledge in the platform gives us a clear picture of what we need to do, and this is also super cool, and we can simplify the publishing logic.” (CEO of Company 2)

The above quotations and discussions show, compared with the previous ARM funnel model, that the new concept of mobile game publishing logic helps indie game developers understand mobile game publishing. The interviews clearly show that the new concept offers a good summary of the mobile game publishing logic and contains the key points in F2P mobile games. In addition to this, it provides a clear picture of mobile game publishing and guides indie game developers in improving their game publishing.

5.1.2 The usefulness evaluation

The second aggregate theme, usefulness, refers to using the method for mobile game publishing. The six indie game companies were asked questions about what they think about the usefulness of the method.

First, in terms of usefulness evaluation, the indie game developers reported that the method is useful because it provides solutions to address the specific issues during their game publishing, such as marketing promotion and new version update evaluation, and it provides useful guidance for their game marketing promotion and new version update.

“It’s very useful. It can solve the specific problems in our game publishing. For example, it can give us very clear insight in marketing promotion.” (Publishing Manager of Company 5)

“Because before using this we didn’t know what features were good and what features would help us to make more money. When we use this tool, we can determine if the feature is good.” (CEO of Company 6)

Second, some feedback from indie game developers concerned about ease of use. They only needed to collect game data based on the templates provided by the method, upload them to the analysis platform, and directly get the conclusions and suggestions for optimization.

“In the past, it was just to imagine the problem and guess the player’s behavior. But now, we don’t have to think about which part of the content is wrong, and as long as we upload the data to the platform for analysis, we can get conclusions as to which part is wrong. We can directly take actions according to the suggestion and optimize the game publishing.” (CEO of Company 1)

Third, for the specific use process, indie game developers who do not have enough analytical experience and programming skills can still use the method to analyze their data, so they find it useful.

“The usefulness, I could say that it is simple to use. As I said, I do not have any programming skills, I just want to know how to do the marketing analysis, but using this platform, the functions are quite easy to use.” (CEO of Company 2)

In the interviews, the indie game companies agreed that this method is useful because it can solve the specific problems in their game publishing. In practice, they only need to collect the related data according to the templates and upload the data to the platform for analysis, and then they can conclude which part is wrong. Then, they can choose to take action according to the suggestion and optimize their mobile game publishing.

5.1.3 The efficiency evaluation

The third aggregate theme, efficiency, involves evaluating the effectiveness of indie game developers after using the method. The six indie game companies were asked questions related to efficiency in the interviews, and their feedback was analyzed.

First, the specific efficiency comes from the speed in getting the analysis results. Indie game developers only need to submit data according to different data templates and upload them to the analysis platform to get the results directly. The method can save time.

“That will save a lot of time for us too, like when we need to compare the data by ourselves it will choose all the data that we uploaded, and it will save a lot of time, and then it will give some advice, and we can follow those suggestions and compare it with our experience. So that is a good way to save time and work efficiently.” (Publishing Manager of Company 4)

Second, the specific efficiency comes from the fact that even people without analytical experience can still use it as long as they fill in their game data according to the data templates and upload them to the analysis platform. The whole analysis process can even be handled by only one person.

“We think it’s quite easy to use, especially for an inexperienced person who can download the data according to the template. Filling the data into the template and uploading it to the platform, you can actually complete the analysis in just a few minutes.” (CEO of Company 1)

“Because it’s very easy to use, we can just fill in the sample data, and one person can collect the data and import it into the tool and get the result very quickly.” (CEO of Company 6)

Third, the method improves the efficiency of indie game developers, especially when it comes to solving specific game publishing issues, such as what needs to be done after a new version update, how to choose high-quality channels for user acquisition during the marketing promotion, and how to set revenue targets efficiently and get the revenue forecast quickly.

“This tool dramatically improves our work efficiency. The specific effect evaluation after each new version update can give us more comprehensive analysis feedback and some essential suggestions, which are particularly needed after the new version release. Then, the third function is the market evaluation. It can help us to more precisely position the channels that are suitable for us to do the promotion, including what channels are not appropriate for us. In this new user’s acquisition part, it is vital to provide decision-making. The fourth, revenue forecast, gives us a more comprehensive goal that can guide our game publishing and expectations.” (CEO of Company 3)

In short, according to the above quotations and discussions, the method is efficient for indie game developers. Because they only need to fill the data in the templates by one person and upload them to the F2PAP, they can get the results quickly.

5.1.4 The effectiveness evaluation

The fourth aggregate theme, effectiveness, shows how the method improves indie mobile game publishing. During the interviews, the questions related to effectiveness were asked about the effectiveness evaluation.

The effectiveness was evaluated and performed by addressing the practical issues indie game developers face during their game publishing. It is clear from indie game developers’ feedback that the method helped them acquire valuable and cost-effective users, especially in terms of marketing promotion.

“This can help us solve several issues. The main problem is the customer acquisition problem. We can get the most valuable and cost-effective players. Besides this, how to promote the conversion of existing active payers to paying players, the pay conversion, and lead the churn players to return can be effectively solved by the suggestions. Revenue forecasts also can help us understand the goal and the target of the publishing business, and it’s not as confusing as before. This is something that I feel is going to solve our problems throughout the publishing business.” (CEO of Company 3)

In addition, effectiveness is measured and demonstrated in terms of the positive changes after using the method. These changes include changing the traditional way of indie game publishing, especially the indie game developer's mindset about mobile game publishing. It includes what metrics are important, how to analyze them, and how to optimize game features and monetization based on the analysis result.

“We publish our games according to traditional ways. This tool changes our situations. It can tell us what metric is important and it can give us a comprehensive view for mobile game publishing.” (Publishing Manager of Company 5)

“I think the tool will figure out what we need to do in the next feature in the next version. We need to improve the feature related to the new version and related to monetization.” (CEO of Company 6)

Based on the above quotations and discussions, it can be concluded that most indie game developers realized that it was easy to use the method and benefited from knowing the game performances. In practice, this method can help them with the data analysis, discover potential problems with new version updates, marketing promotion, and revenue forecast during their mobile game publishing, and then provide guidance, especially on what actions should be taken to optimize their publishing efforts.

5.1.5 The satisfaction evaluation

The fifth aggregate theme, satisfaction, shows how the indie game developers were satisfied with the method. Questions about satisfaction were asked during the interviews, and their feedback was analyzed and discussed.

First, the satisfaction from indie game developers after using the method mainly focused on the functions and guidelines provided by the method to solve practical problems, such as marketing promotion, new version update evaluation, and revenue forecast, and then provided suggestions and actions based on the analysis result.

“We can plan the data we need to collect in advance during the game development and then analyze the market promotion, new version update, and revenue forecast by collected data according to the template. We have no previous experience in this part, but now we can achieve it through this platform, so it also provides a good idea for new game development and publishing.” (CEO of Company 1)

Second, the indie game developers were satisfied with the method because it saves time and money. For example, it only needs one person, takes very little time, and provides guidance for marketing promotion, new version update evaluations, and revenue forecasts.

“I think it’s a great tool to help us analyze, both in terms of cost, time, and overall usability, and it helps our team get into the whole process of publishing based on data analysis more quickly. So I think it’s a good thing that we will continue to use in future game publishing.” (CEO of Company 3)

Third, indie game developers are satisfied with this method because it has been integrated into their daily work, and they are willing to continue to use it to guide their mobile game publishing based on data analysis in the future.

“I would like to do some further work on this platform because as far as I used this, I think it is very friendly because there are no advertisements. I uploaded my data, and it took like one or two seconds to calculate and give me the result. That is a very good part. So I will keep using this, and it also gives me the publishing theory, so I will continue to check those theories to support my daily work.” (Publishing Manager of Company 4)

According to the findings, the indie game companies involved in this research gave positive feedback about this method. They were satisfied with the method that added value to their mobile game publishing. They believed that it could tell them what metric is important and give them a comprehensive view of mobile game publishing. It helped them make the right decision, such as what they need to do in the following new version update, in choosing the high-quality channels for user acquisition, in setting the revenue benchmark, and in driving their mobile game publishing. It also provided good insights for new game development and publishing based on data analysis.

However, some feedback about weaknesses was also received, especially that it is hard for some indie game developers to understand the correlation analysis, which needs to be improved. Besides this, the optimization suggestions were also collected. For example, the graph and the font in the F2PAP are small and need to be improved. If they could click the chart and show some numerical details, this would be better.

5.2 Survey data analysis

According to previous evaluation criteria, the survey evaluation of the method also focused on usefulness and usability. The primary purpose of this evaluation was to apply the method to the different indie game companies with various game publishing projects. Surveys were conducted to analyze the usefulness and usability of the method after applying it in real mobile game publishing projects. The evaluation process of the method was to use it in the different mobile game companies with different F2P mobile game projects in order to address specific problems during the mobile game publishing process. The usefulness and usability need to be measured in real-world cases where indie game developers have implemented and tested the artifact. In this case, the evaluation was applied in six indie game companies’ mobile game publishing projects where the method was tested. The positive evaluation results were confirmed when usefulness and usability were measured via surveys with indie game developers.

5.2.1 Questionnaire validation

Measurement reliability

Measurement reliability means that the multiple items measure the same construct, and it means that the scale measures what it is intended to measure [51]. Precision reflects the number of response options associated with the survey questions, and it refers to the degree of consistency or reliability of the questionnaire test results. Reliability is used to measure whether the results of the sample responses in the questionnaire are reliable and whether they are genuine responses, and the higher the test reliability, the more reliable the result [51]. A questionnaire must be analyzed for its reliability before being used in order to ensure that the questionnaire results are valid, stable, and accurate.

KMO and Bartlett’s test

Validity analysis was performed to check whether the sample size was appropriate for the number of variables used for the study, and the survey was validated by the Kaiser-Meyer-Olkin (KMO) and Bartlett’s Test [22]. KMO measures the sampling adequacy for successful factor analysis. Bartlett’s test is another indicator to measure the strength of the relationship among variables. If the value is between 0.6 and 0.7, the validity is acceptable. As shown in Fig. 11, for the KMO and Bartlett’s test analysis, the calculated value was 0.711, which means the survey had acceptable validity. Based on the validity analysis, it is clear that the survey results were valid, stable, and accurate. Boone and Boone [5] provide information for the correct analysis of Likert data, and the mean, median, standard deviation, and minimum and maximum values can be calculated for individual Likert-type questions. For Likert scale attributes, the mean and standard deviation are most recommended. These recommendations were taken into account to analyze the results. The survey questions were related to changes in usefulness, satisfaction, effectiveness, efficiency, and confidence before and after using the method.

Fig. 11

Questionnaire validation

In practice, the evaluation was composed of more than 20 questions, using a 7-point Likert scale (from strongly agree to strongly disagree). Because the participants had different roles in their indie game companies. In order to get valuable analysis results, the surveys completed by participants who directly handled the game publishing from the indie game companies were chosen to analyze first. As shown in Table 3, for the in-depth investigation, 22 participants were selected from 58 pre-survey respondents, 22 from 56 Survey 1 respondents, and 21 from 50 Survey 2 respondents based on their roles in the companies.

Table 3 The selected participants from six indie game companies for survey evaluation

The analysis results obtained from the survey 2 were used for measuring different characteristics related to effectiveness (Q1, Q3, Q4), efficiency (Q2, Q9, Q18), satisfaction (Q5, Q10, Q11, Q20, Q21), usefulness (Q6, Q7, Q8, Q12, Q13, Q14, Q19), and confidence (Q15, Q16, Q17, Q22, Q23, Q24, Q25, Q26, Q27) in Survey 2.

5.2.2 Detailed survey responses

The method was evaluated positively according to the surveys conducted with the indie game developers from six indie game companies. After using the method, the usefulness, effectiveness, efficiency, and satisfaction were measured. All answers were rated high and had mean and median values greater than 5 out of 7 possible points. The mean, median, standard deviation, and minimum and maximum values are shown in Table 4.

Table 4 Questionnaire statistical analysis result

In short, the main results were obtained after analyzing the responses from indie game developers about the usefulness and usability of the method. In order to avoid any bias in the survey results, only the indie game developers who handled the mobile game publishing were chosen for the in-depth analysis. In addition, the general aggregated analysis of the usefulness and usability answers was performed with the mean, standard deviation, and minimum and maximum values. In order to further understand what specific questions are related to each evaluation criteria, the individual analysis of the answers was also performed related to different criteria.

The survey results on individual analysis of the answers to each question about effectiveness are shown in Table 5. Each question was chosen carefully to represent the feedback about using the method related to effectiveness.

Table 5 Descriptive statistics related to effectiveness

The results for the three survey 2 questions about efficiency are shown in Table 6. Each question was chosen to represent the feedback regarding efficiency when using the method. An overall positive result was observed.

Table 6 Descriptive statistics related to efficiency

The survey results related to satisfaction are shown in Table 7. Each question was chosen to get feedback regarding satisfaction with using the method.

Table 7 Descriptive statistics related to satisfaction

Survey results related to usefulness are shown in Table 8. Each question was chosen to represent the feedback regarding the usefulness of the method.

Table 8 Descriptive statistics related to usefulness

In short, the evaluation process applied to indie mobile game publishing focused on indie game companies based in different areas with diverse mobile game publishing projects. All of the users from the six indie game companies who handled the mobile game publishing and participated in the mobile game publishing reported satisfaction with the method. The interviews with six indie game companies confirmed this by providing positive feedback about using the method. The survey results also show positive answers regarding the effectiveness, efficiency, satisfaction, and usefulness after using the method because the related answers were rated high and were above five in their mean values, as shown in Fig. 12.

Fig. 12

Questionnaire histogram analysis

5.2.3 Evaluation of confidence changes

A comparison was made in the survey data showing the confidence changes before and after using the method for mobile game publishing. The first was the pre-survey before using the method, and the other was Survey 2 after using the method. The mean confidence score changed from 3.59 to 6.03, thus showing a significant improvement (P < 0.05) after using the method (Fig. 13).

Fig. 13

Confidence comparison

The results from surveys and interviews showed positive results regarding the usefulness and usability of the method, and additional characteristics related to confidence changes in handling their mobile game publishing were also analyzed. The results showed positive feedback about using the method to support decision-making in mobile game publishing, especially for the new version update, marketing promotion, and revenue forecast. Apart from the conclusions that can be drawn from the responses, the feedback provided in the open questions during the surveys revealed similar findings, such as “I do think it is useful for mobile game development and publishing, especially in the view of indie mobile game development. It might be able to help us to extend our business in a good direction with a lower cost.” This was reinforced in the previous interviews conducted after using the method with different participants. All of them showed interest in the method and its usefulness and usability. The method was useful and valuable to different indie game companies.

6 Conclusion and future work

This paper focuses on solving the mobile game publishing challenges faced by indie game developers. Because most indie game companies are good at game development but lack experience concerning game publishing [19], they have issues with user acquisition and transferring users into loyal and paying players [31]. This research starts from indie game publishing's main challenges [42], extends the existing ARM funnel model from social games to mobile games, and then provides a new concept of mobile game publishing logic, as shown in Fig. 5. It summarizes the mobile game publishing as two inputs (channel and new content) and one output (revenue). It helps indie game developers to understand game publishing and evaluate and analyze game publishing performance, such as new version updates performance, marketing performance, and revenue forecast. In order to facilitate their use of game analytics to guide mobile game publishing, the F2PAP has been designed and developed, and it is deployed as an online game data analysis platform based on the new game publishing logic. The platform focuses on the inputs and outputs of the mobile game publishing logic and provides different solutions. Indie game developers only need to provide the corresponding data according to different templates, upload the data to the platform, and automatically get the analysis results. In practice, the related data analysis for mobile game publishing includes the new version update evaluation, marketing promotion evaluation, and revenue forecast.

Previous research [32] showed that the ARM funnel model had been applied to game publishing, and third-party analysis tools have been implanted with user acquisition, retention, and monetization functions, such as GameAnalytics and Firebase. These basic analytics are available today even for startup companies, and the initial costs of implementing game analytics are relatively low [27]. However, related research indicates that indie game developers still face the challenge of applying BI in mobile game publishing, and even though they have used these third-party analysis tools, they still have issues with data collection, data analysis, and marketing [42]. They cannot use these third-party analysis tools to evaluate their game's new version update, marketing promotion performance, and revenue forecast because the available tools have limited functions that have not covered these aspects. Mäntymäki et al. [29] point out that indie game developers have a broad interest in adopting more sophisticated analytical procedures but lack the resources to implement them. In particular, skills in data management are a bottleneck for adopting advanced analytics. Besides this, Limpach [28] pointed out that the publishing challenge for indie game developers is that further updates must be developed after the initial game release and after there is analytics to monitor. The main contribution of this research is to provide a data-driven method to address these issues with an online analysis tool, the F2PAP, which can guide indie game developers with their mobile game publishing. In order to determine the usefulness and the value of the proposed method for mobile game publishing, the final evaluation was made to apply this method for mobile game publishing. Interviews and surveys were used to measure the usefulness and usability of the method, and the final results showed that this method improved the indie game developers' confidence in handling mobile game publishing and benefited their game publishing business.

However, some weaknesses and optimization advice from the indie game developers were also received. For example, the correlation analysis is a little hard to understand, especially for the Pearson's correlation coefficient. Also, the graph and font in the F2PAP are a little small and need improvement, and if they could show some numerical details, the metrics chart would be better. In addition, the method is also related to the game business model because it is only used to evaluate and guide F2P mobile game publishing and is not suitable for pay-to-play (P2P) mobile games.

The P2P business model and many indie game developers only focus on downloading data without caring about the data in the game, and this needs further research. In fact, although the method provides generalized solutions for F2P mobile game publishing, there are still specific requirements for different game genres during mobile game publishing, such as the Role-Playing Game (RPG), First Person Shooter (FPS), and Massively Multiplayer Online Role-Playing Games (MMORPG). It is significant to continue to apply this method to more games with different genres and to study the differences between game genres and their needs for game publishing to further optimize the existing method and provide a more customized solution. This is because, besides the generalized metrics, different genres need to use specific metrics, such as FPS games that need to track the heatmap of deaths and kills.

From a market competition perspective, game companies and platforms often resort to questionable monetization practices due to the intense competition in the gaming industry. Game development needs to consider whether the monetization design is ethical, and therefore we need to address ethical issues in the method and tool design. Considering the research ethics, the metrics involved in the method are not only related to monetization, such as revenue, but also focus on the channel, new content, retention, new player, active player, and churn player and measure the marketing and new content performance as well as players’ behavior changes. This tool is more about helping indie game developers evaluate and analyze the game publishing performance rather than teaching them how to monetize. Thus the related data collection not only focuses on the revenue but also on the changes in player behaviors to optimize the game experience, provide better services to players, and avoid churn.

Appendix

Survey descriptions and instruments

Thank you for taking part in this survey and contributing to this research! Below is the introduction to our research, please read it first and then make your final decision. Thanks!

Aim:

The aim of this survey is to gather your opinion about some quality characteristics of this F2P analysis platform after using our data-driven method. Please answer these questions after using the platform and its data-driven method.

Anonymity:

The information obtained from this questionnaire will be employed just for research purposes and the anonymity of the participants is ensured. No individual information will be disclosed.

Data usage regulations:

The data collected in this survey will only be used for our research. It could be included in academic papers, policy papers, or news articles. The relevant research data will be kept as an archive for ten years.

Your rights:

During the whole survey process, you have the right to terminate or withdraw from the research. If you complete the survey, it means that you allow us to use the relevant research data for our research.

Research Ethics:

From the perspective of research ethics, our research strictly follows the requirements of GDPR and only collects necessary data related to the research. All the research processes follow the research ethics and avoid the collection of private data.

Duration:

The questionnaire will take about 15 minutes to complete. If you don't know the answer to one of the following questions, you can skip that question. Thank you for your cooperation!

Consent

I have read the relevant instructions, agree to participate in the research, and agree to apply my survey feedback results to the corresponding research. Please select “√” for the below choice.

In practice, the Tencent Survey Tool is used to do the online surveys, and attached are the links:

Pre-survey address: https://wj.qq.com/s2/8567901/21bd

Survey 1 address:  https://wj.qq.com/s2/8568055/1aa6

Survey 2 address: https://wj.qq.com/s2/8568107/8bb7