A Pilot Study of Mining the Differences in Patterns of Customer Review Text Between US and China AppStore
With the fast growing of AppStore market and the developing of techniques in opinion mining, this study was aimed to investigate the sentiment and opinions of customer reviews in both China AppStore and US AppStore, and identify the difference of key term and patterns of apps reviews among different genres and between China AppStore and US AppStore. Results showed that there were small differences in using adjective words used or expressing key opinions. The result of this study could help publisher to extract useful customer feedback from customers reviews when publishing apps in foreign countries.
KeywordsCross-cultural product and service design Cultural differences Review mining
In recent years, mobile services and platforms have achieved critical mass in the information and communications technology industry. The key to their success has been mobile app services, including naive softwares and platforms that offer internet-based services with good user experiences . With iOS being one of the major mobile phone operation systems, its app service platform, Apple’s App Store (henceforth, AppStore) also prosper in app service market with a growing number of publishers and users. Since AppStore launched with only 500 apps and a dozen developers in July 2008, the market increased to over 2,281,240 apps and 529,078 active app publishers in April 2016 . By April 2016, there were 155 AppStore territories that are available for apps to be sold in the corresponding countries or regions . And up to September 2016, a total of 140 billion of apps were downloaded by users from all over the world . In this rapidly growing market, for publishers that have their apps published in multiple AppStore territories, it is very important to adapt to the local market, and adjust contents of apps accordingly.
AppStore provide a rich source of information about apps, including one app’s price, description, technical information, and customer ratings and reviews, which could provide both qualitative and quantitative data about the customer perception of the apps, and is very important for both customers and apps publishers. On the one hand, customers’ ratings and reviews of apps would affect other customers’ purchase decisions, this effect is equivalent to the persuasive effect studied in the advertising literature . Meanwhile, online customer review system is one of the most powerful channels to generate online word-of-mouth , and earlier studies have found that word-of-mouth may affect others’ decisions in different social contexts . According to previous studies, online reviews have significant impact on sales [4, 6]. On the other hand, for publishers, customer review is a major source for the feedback. Other feedback source for apps including e-mail feedback and blogs. Feedback could reveal bugs or features of the current version that need to be fixed or improved.
Customer review in AppStore is spontaneous customer feedback, which has rich sources of information. However, these sources are much less structured than traditional surveys for customer satisfaction studies. The information is contained in free-style text, not in a set of answers elicited for a specific set of questions. With the advent of automatic techniques for text mining such as clustering and key term extraction, free-form customer opinions can be processed efficiently and distilled down to essential topics and recurring patterns of content. Researchers have begun to focus on the analysis of opinion typically using supervised machine learning techniques . For example, by analyzing online reviews of computer game, the characteristics of computer games and user experience in game play could be identified . By using linguistic techniques, researchers have extracted and analyzed the most important factor a moviegoer considers when rating a movie online, and found that reviewers mainly discuss their personal evaluation rather than discouraging or encouraging readers to see the movie . By clustering rare textual opinions based on point-wise mutual information and using externally imposed review semantics on a data set from Amazon containing sales data and consumer review data for digital cameras and camcorders over a 15-month period, researchers have analyzed the consumers’ relative preferences for different product features and use the textual data to predict future changes in sales .
With the fast growing of AppStore market and the developing of techniques in opinion mining, this study was aimed to investigate the sentiment and opinions of customer reviews in both China AppStore and US AppStore, and identify the difference of key term and patterns of content in apps reviews among different genres and between China AppStore and US AppStore. The result of this study could help publisher to extract useful customer feedback from customers reviews when publishing apps in foreign countries, and provide a insight of cultural differences in writing apps reviews between Chinese and American. To be specified, the research questions of this study were:
RQ1. Between America AppStore and China AppStore, and among different genres, is there any difference of patterns in customer review for top-selling apps?
RQ2. What is the portion of app-review relevant words in the review text for top selling apps, is there any difference among different genres, and between China AppStore and America AppStore?
Review text and relative data were collected from the following four genres in both US AppStore and China AppStore: Social Networking, Photo & Video, Games, and Entertainment. The reason of choosing these four genres was that in the top chart for each genre there were enough common apps in both US AppStore and China AppStore, and there were enough reviews for apps in the top chart. For each genre, apps in the top 200 free apps chart and top 200 paid apps chart were collected, therefore the total number of apps that were included in this study were 3200. A web crawler was developed to collect the app information, review text, and other relative data. For each app, the collected app information including region (US or China), app name, genre, release date, overall average rating, overall number of ratings, and app price for paid apps. Fifty most recent reviews were collected for each app, or all of the reviews if total number of reviews was less than 50. Review title was collected together with review text for each review. Apart from the most recent reviews, all reviews of some selected apps were collected as well.
Natural Language Processing
For reviews wrote in English, processing raw review text including the following steps: removing irregular characters, converting to lower case, word tokenization and part-of-speech tagging, stemming and lemmatizing, and removing stop words, calculating word frequencies, and generating term frequency-inverse document frequency (TF-IDF) matrix. For reviews in Chinese Store, steps of processing raw text were similar to processing English text, with the lack of converting to lower case, and stemming and lemmatizing. We used Natural Language Toolkit (NLTK)  for English word tokenization and part-of-speech tagging, and Jieba  for Chinese word tokenization and part-of-speech tagging. When generating TF-IDF matrix, each review was treated as a document.
We selected k-means clustering algorithm to cluster the reviews. This algorithm is widely used in document clustering and text-mining for it’s simplicity. Considering the fact that online reviews have a wide variance in lengths, we chose cosine distance for k-means algorithm so that the cluster results would be independent to the lengths of reviews. The cosine distance were calculated from TF-IDF matrix, which were calculated during the natural language process
Noise Point Detection
The density-based spatial clustering of applications with noise (DBSCAN) algorithm views clusters as areas of high density separated by areas of low density. Clusters found by DBSCAN can be any shape, as opposed to k-means which assumes that clusters are convex shaped. This algorithm can be used to detect noise point, but the result is heavily related to the input parameters. In this study, we used DBSCAN to find noise reviews which were less relevant other reviews.
Number of reviews collected
Number of reviews collected
Photo & video
3.1 Frequencies of Adjectives
Figures 1 and 2 showed the top 20 frequent adjective words of US reviews and Chinese reviews in the whole review collection. Words like “good”, “great”, “fun”, and “easy” were most frequent in both US and Chinese reviews. For US reviews there was no adjectives with negative sentiment in the top frequent adjective words. Similarly, for Chinese reviews, only one adjective with negative sentiment, which was “boring”, occurred in the 20 most frequent adjective words. For Chinese reviews, the term of “not bad” was the most frequent adjective words and had much higher frequency than the rest of adjectives. In US reviews, “great” and “good” were top two frequent words, and compared with Chinese reviews, the gap between the frequence of the most frequent adjective and the frequencies of the rest of the adjectives was smaller. The results suggested that customers in both AppStores were more likely to express positive sentiments. And the high frequent of the term “not bad” in China AppSore may caused by the habit of using “not bad” as a common pet phrase among Chinese people.
Top 20 frequent adjective words of four genres of US reviews, words that were not common among the four genres were presented in bold
Top 20 frequent adjective words
Great, good, other, new, free, easy, able, many, much, nice, same, awesome, few, cool, only, update, bad, different, old, first
Great, good, easy, other, awesome, free, new, many, much, able, nice, cool, only, different, simple, first, same, amazing, few, perfect
Great, good, other, new, fun, awesome, much, many, free, first, same, hard, little, only, able, few, different, easy, cool, bad
Great, good, other, new, free, many, awesome, much, easy, able, cool, old, few, same, only, first, nice, bad, little, different
Top 20 frequent adjective words of four genres of Chinese reviews, words that were not common among the four genres were presented in bold
Top 20 frequent adjective words
Not bad, convenient, very good, at all, fantastic, best, simple, fun, rich, fluent, easy to use, powerful, clear, boring, again, perfect, concise in visual, pretty fun, beautiful, successful
Not bad, convenient, simple, powerful, fantastic, at all, best, very good, perfect, easy to use, again, easy, special effects, clear, concise in visual, fun, just average, rich, blurred, important
Not bad, at all, pretty fun, conscientious, simple, boring, again, fantastic, very good, simple, rich, perfect, fun, just average, fluent, best, exquisite, important, delicate, severe
Not bad, convenient, at all, fluent, very good, rich, best, fantastic, simple, clear, again, perfect, powerful, boring, fun, just average, pretty fun, conscientious, easy to use, concise in visual
3.2 Cluster Results
K-means clustering algorithm was performed on both US reviews collection and Chinese reviews collection. We ran multiple k-means with k various from 2 to 16 for both US reviews collection and Chinese reviews collection, and extracted top 50 terms in each cluster for each ran. Then we inspected the results for each ran manually to see if the reviews were clustered by topics or features. The final k values were both 12 for US reviews collection and Chinese reviews collection. The extracted top terms and clustered reviews focus were showed in Tables 4 and 5.
K-means result for US reviews collection
Number of reviews
Top 10 terms
Upgrade loved app, app recently compatible, app past really, upgraded app recently, unusable, loved app past, upgraded app, made much, spoiled, app past
Way many ad, every, ad pop, play, fun, ad every, good, time, get, great
Advertisements in app
Game, try, app crash, crashing, work, even, play, keep, time try, crash every
Pretty, cool game, app cool, really, game, game cool, seems cool, pretty cool app, cool good, really cool app
App love, much, fun, use, awesome, really, love love, like, like app, app much
love this app
Fun play, much, game fun, great fun, really fun, play, super fun, fun fun, nice game, fun use
Great app use, great app work, app use, app work great, great great, work great, app work, great app easy, use, app easy
Awesome game, ever played, game much, played, game addicting, game best, addicting, love game addicting, much
Really, good, like, great original, new feature way, adding new feature, feature way playing, way playing, feature way, playing make
Time, amazing, really, easy, one, would, make, best, please, play
Need fix or update
Actually work, love work, work perfectly, perfectly, love work great, actually, really, get work, work like, even work
App works good
Time, work, refund, buy, work waste, get, even, get money, app waste, want refund
Waste of money, want refund
K-means result for Chinese reviews collection (translated)
Number of reviews
Top 10 terms (translated)
Live video, software, like, phone, good, fun, photo, game, effect, easy
Version, endless, update, case, phone, photo, bug, album, reason, log in
First time, download, great, display, functions, good, recommend, find, really, work
Good use experience
Update, good, can’t open, things, display, uninstall, write reviews, delete, like
Uninstall, free, video player, disgusting, really, annoying, app, wish, right away, bad
First try, work good, friends, fun, trustworthy, app, really, display, like, simple
Easy use, interesting
Feel, support, indeed, game, childhood, can’t stop playing, recommend, originality, real, cute
Fun, filter, friends, app, wish, stickers, display, work great, support, recommend
Usability of photo editing
Wish, support, friends, great, functions, utility, powerful, really, phone, reviews
Trash, effect, supper, friend, fun, support, useful, app, download, live video
Try this app out
Wait for, fix it, support, every time, bug, can’t log in, trash, system, can’t open, video
Crash, really, good, classic, interesting, time-killer, can’t use, player, support, great
Noise Review Detection
Number of noise review detected
Number of noise reviews
Percentage of noise reviews (%)
Photo & video
This study investigated the difference of key term and patterns of content in apps review text among different genres and between China AppStore and US AppStore. We presented a preliminary method for mining customer opinions from free-style review text. This review text mining technique could be used in customer opinion mining and customer satisfaction survey for mobile app publishers and other interested producers with further modification and improvement. The results showed that in general the key term used and opinion expressed in reviews of China AppStore and US AppStore were similar, only minor difference was found. One of the differences was that the reviews wrote by customers in China AppStore were more specifically related to the genres of the reviewed apps. The other differences was that customers in US AppStore were more used to ask customer services for refunds. This differences may caused by the fact that the internet-based services were relatively new to Chinese customers than US customers, and the return policy was more mature in US. As Chinese customers were less used to ask or complain to customer services, they may complain more in the reviews, hence the result of their reviews were more specifically related to the genres. This study was a pilot study, there is still more to explore with review text in this manner, and the comparison between two review collections needed to be more quantified, and works related to culture differences were still needed to investigate further.
This research was supported by the National Natural Science Foundation of China (NSFC, Grant Number 71471095). This study was also supported by Tsinghua University Initiative Scientific Research Program under Grant Number: 20131089234.
- 1.Apple: Apple - choose your country or region. https://www.apple.com/choose-your-country
- 4.Chen, Y., Fay, S., Wang, Q.: Marketing implications of online consumer product reviews. Bus. Week 7150, 1–36 (2003)Google Scholar
- 7.fxsjy: Jie ba - Chinese text segmentation. https://github.com/fxsjy/jieba
- 11.PGbiz: app store metrics. http://www.pocketgamer.biz/metrics/app-store/
- 12.Simmons, L.L., Mukhopadhyay, S., Conlon, S., Yang, J.: A computer aided content analysis of online reviews. J. Comput. Inf. Syst. 52(1), 43–55 (2011)Google Scholar
- 13.Statista: cumulative number of apps downloaded from the apple app store from July 2008 to September 2016 (in billions). http://www.statista.com/statistics/263794/number-of-downloads-from-the-apple-app-store