1 Introduction

Technological improvements and novelties have not only provided us with new hardware options including mobile devices, bigger screens and diverse peripherals such as game controllers, but also inevitably led to the adoption of a more user-oriented approach to design more user-friendly platforms. From the first conventional keyboards to the advanced gesture-based interactive technologies, the need for a high-quality user experience has emerged and increased. In the last decade, in order to cope with this transition adopting a user-centered perspective, numerous studies have been conducted making the term user experience (UX) one of the pinnacles of technological evaluation and progression.

The ever-changing and rapid developments in technology can also be observed in the context of video games, which are considered one of the leading indicators of technological advancement. Video games offer new perspectives for discussions and studies on UX; however, conventional UX and usability methodologies are not sufficient to provide a full understanding of the video game experience, since they mostly focus on productivity in digital platforms rather than entertainment which underpins the gameplay experience. This major difference even leads us to change the terms we use in the context of gaming; replacing ‘usability’ with ‘playability’ and ‘user experience’ (UX) with ‘player experience’ (PX) [1, 2].

PX can be observed in different gaming platforms from consoles to mobile devices and from peripheral-based to gesture-based games requiring novel interaction mechanisms including different screens and peripherals, thus revealing a complex structure at various levels [2]. It is critical to understand the complex nature of PX through user research. However, there are only a limited number of studies investigating PX in detail and therefore, this study aims to create a framework for the analysis of this phenomenon based on previous work on usability and UX in the relevant literature.

The available studies on gaming experience only present a set of PX and playability heuristics on a theoretical basis; therefore, there is still a need to conduct empirical research collecting data from various gaming platforms. This study focuses on the analysis of multi-platform player experience based on a PX and playability heuristics framework derived from the literature on usability and UX. The proposed framework is also tested in a multi-platform game setting to seek ways to contribute to the improvement and enrichment of the framework.

2 Playability and Player Experience

Sánchez et al. (2009) proposed a new approach to PX with their playability model (PM) revealing the differences of evaluative goals in several specifications [4]. The term playability is used in the analysis of a video game or its design aspects. Sánchez et al. identified playability as, “a set of properties that describe the Player Experience using a specific game system whose main objective is to provide enjoyment and entertainment, by being credible and satisfying, when the player plays alone or in company” [6].

Several authors [e.g. 1, 3, 7, 8, 10] have offered numerous heuristic models to evaluate playability as well as genre-specific models such as heuristics for mobile games [3] or advanced table top games [9]. However, none of these models were validated through tests and/or comparison with other methodologies. We believe that these models need to be validated with user tests in multiple platforms. Usability focuses on utilitarian aspects such as task completion, elimination of errors, external rewards and reducing workload whereas playability is concerned with hedonic aspects such as entertainment, fun to beat obstacles, intrinsic rewards and new things to learn. [5]. Therefore, conventional usability approaches and heuristics cannot be used alone to describe PX.

The first set of heuristics specific to the structure and model of video games were created by Federoff [8]. To identify playability heuristics, Federoff gathered traditional usability heuristics from literature review, observed the game development procedures of a company and conducted interviews with the game team of the company. Focusing mostly on game engagement and storyline, Federoff suggested that video games have the following three features; ‘game interface’, ‘game mechanics’ and ‘game playability’.

Another set of heuristics, Heuristics of Playability (PLAY), were proposed by Desurvire (2009) based on the evaluation of usability. The PLAY heuristics were developed with the help of game industry professionals and grouped into three categories: The first category, Gameplay, contained heuristics related to players’ feel of control, challenges, goals, consistencies, balance and the notion of fun; the second was concerned with coolness, entertainment, humor and emotional immersion; and the final category, Usability and Game Mechanics, consisted of the heuristics on documentation, status and score, feedback, terminology, burden on player (as in not putting unnecessary burden on player), screen layout, navigation, error prevention and game story immersion [10].

Korhonen and Koivisto [3] developed another approach to PX, called ‘Playability Heuristics for Mobile Games’, which is closely related to other methodologies regarding playability. The authors proposed a model focusing on mobile technologies and their use in entertainment applications with an emphasis on mobility. They suggested, “The mobile context has some unique characteristics, which require special attention during the evaluation” and discussed these characteristics in detail. Korhonen and Koivisto supported the previous studies in the literature in that traditional usability heuristics lack comprehension and cannot be directly applied.

Even though Korhonen and Koivisto [3] indicated that usability heuristics cannot be applied to video games, similar to Sánchez et al. [1], they based their heuristics mostly on user interface. In addition, they pointed out that the former playability approaches were not feasible for mobile platforms and did not offer a novel perspective. Therefore, in their proposed heuristics model, in addition to gameplay and game usability, they included the mobility module. In mobile interaction, users often have environmental distractions such as lighting, weather conditions or noise. There can also be other people in the vicinity, affecting the gaming experience of users. Therefore, Korhonen and Koivisto identified three main mobility heuristics as ‘the game and play sessions can be started quickly’, ‘the game accommodates with the surroundings’ and ‘interruptions are handled reasonably’ [3].

All the above-mentioned methodologies and approaches are viable; however, one [11] has recently integrated the various definitions and categories of playability heuristics into the framework of ‘playability’. The PM framework developed by Sánchez et al. provides an easy-to-manage and well standardized set of heuristics to understand PX and its relation with UX.

The PM framework is inspired from the three heuristics of usability; effectiveness, efficiency and satisfaction as well as their correlation with playability. As a result, seven different heuristics are proposed as follows: effectiveness, learnability, satisfaction, immersion, motivation, emotion and socialization.

3 Methodology

In this study, the framework of analysis is based on the PM framework of Sánchez et al. (2009, 2012) and the parameters of effectiveness, learnability, satisfaction, immersion, motivation, emotions and socialization. In the tests, we also utilized the mobility heuristics developed by Korhonen and Koivisto [3].

Since playability analysis is a complex process due to various perspectives, Sánchez et al. suggested “Playability Facets” into consideration, in which they used those facets to categorize different elements of video game architecture. Each facet of playability identify different attributes of playability. The notion of playability facets is to function as a tool to study playability across different video game elements. Along with other methodologies and categorizations, playability facets cover categories of interpersonal and intrapersonal playability.

Mobility heuristics devised by Korhonen and Koivisto [3], and Interpersonal and Intrapersonal Playability categories proposed by Sánchez et al. are relatively new to the literature (Fig. 1). We acknowledge that they have brought a novel perspective to the playability analysis. In our study, for a better analysis of playability, we introduce a comprehensive approach not only in terms of game features but also concerning the separate evaluation of various complex game elements assessing the parameters in two categories; ‘On-Screen’ and ‘Off-Screen’.

Fig. 1.
figure 1

Relations between video game elements for the analysis of Playability/PX

Full size image

The notion of On-Screen represents the game elements that have been extensively examined in the literature, such as game interface, mechanics, gameplay and storyline. Off-Screen represents the aspects that have only been partially explored to date. These are the environmental and social factors affecting the video game experience of individuals. With the development of cross-platform and multi-platform games, there is a need to consider these social and environmental conditions for a better assessment of PX. The framework we propose allows us to holistically scrutinize all internal and external environmental elements of PX by incorporating the most recent yet incomplete approach by Sánchez et al. and the mobility heuristics developed by Korhonen and Koivisto. In addition, the division between the Off-Screen and On-Screen elements contributes to the comprehensive and complementary structure of this framework. In order to provide an in-depth analysis of multi-platform PX, two different gaming platforms were evaluated; a mobile device (HTC smartphone) and a video game console offering full body gesture-based interaction (Microsoft Xbox 360). The latter platform was selected for testing, not only because there are limited number of studies regarding the novel interaction possibilities that it provides, but the experience provided might offer physical, mental and social benefits and offer a transition from success oriented play to playing only for fun [13].

The qualitative differences between these two platforms were examined using the Fruit Ninja game which is available for both Xbox 360 and mobile platforms. The Fruit Ninja game was originally designed for mobile gaming platforms specifically targeting the mobile phone users. Therefore, the gesture-based version on the Xbox 360 console is only an adapted version of the original game. The game focuses on ‘gameplay’ rather than other game elements such as narrative. This property has been anticipated to facilitate the comparative data gathering and analysis during the study. Both versions for the platforms had the same game mechanics, in our case using an imaginary sword (finger in mobile version, gestures for the Kinect version) to cut objects. The similarity in gameplay and mechanics enabled us to observe and focus on the experiential differences caused by the gaming platforms.

For the tests, a 50 in. LED TV and Xbox 360 including the Kinect peripheral and a HTC 820 smartphone with a 5.5 in. screen were used. For a more realistic experience, a naturalistic test environment was prepared. Representing one of the major user segments in mobile and console game ecosystem in Turkey, the participants were eight university students from Turkey, dimidiated in terms of gender, with prior experience in using smartphone technologies and ages differing between 19 and 23. Majority of the subjects have played the Fruit Ninja game on mobile platform before and only one of them had prior, but limited experience with Kinect peripheral. The test procedure was based on three steps: In the first step, a semi-structured pre-test interview was conducted to collect demographic information about the participants, their gaming background and their experience in relation to using technology and particularly gaming platforms/controllers. The second step was the task observation phase, in which each participant was asked to play the Fruit Ninja game on an Android phone and Xbox 360 game console sequentially. We adopted usability testing for understanding and analyzing playability in detail which provided the possibility of witnessing specific quotations from players, which validate real experiences [12]. With the taskflow which reflects a default gaming experience and incorporate essential game attributes so that the players could have realistic experience of the game. Tasks were to enter the game selection screen (task 1), starting the game in classic mode (task 2), going back to the main menu via stopping/pausing the game (task 3), re-entering game selection screen (task 4) and starting and playing the game in arcade mode (task 5). Behavioral data was collected via video recording and mobile eye-tracking equipment. The final step was a post-test interview. This last phase consists of an in-depth interview to understand and analyze player experience in detail. This phase also enabled an attitudinal analysis. The findings of the study were expected to provide an in-depth understanding of PX in two different platforms with a solid framework containing the parameters of usability and UX.

4 Results and Discussion

Effectiveness:

The effectiveness of the platforms was evaluated in terms of how easy the game was to play and how much effort it required. All the players stated that the game was easier to control and the tasks were easier to achieve in the mobile platform. Figure 2 presents a comparison of the two platforms in terms of the average completion time of tasks for each player (Task 4 was removed from the dataset in the table since it was not included in the Android platform as a new action). The task completion times being less on the mobile device indicates that this platform better facilitated the achievement of specific goals, in our case, the tasks explained in the previous section.

Fig. 2.
figure 2

Average completion time of tasks for each player in the two platforms (Color figure online)

Full size image

The third task was about stopping or pausing the game and was not completed by any players in the Xbox 360 platform. This was expected to be found via the trial and error method but none of the actions of the players stopped the game from running. Some players were frustrated and gave responses such as “Do you really believe that we can find a way to exit this game?” (P1), “I think there is no option for it” (P1 & P2), “Would it stop if I turn my back to it?” (P3), and “Stop Kinect stop!” (P4). Participants also gave non-verbal reactions such as turning their back against the screen (P3), waving hands (P3) or even ignoring the task and keeping on playing (P6, P7, P8). Post-tests also showed that the players found the Kinect peripheral much more confusing.

Learnability:

This refers to the player’s ability to comprehend and master the game system and mechanics. The learnability heuristics are characterized by game knowledge, player skill, difficulty, frustration, speed of learning and discovery.

All the participants were able to understand the game mechanics and had no problems or questions about it. The action of slashing fruit came naturally to all players in both platforms (using arms gestures on Xbox and swiping fingers across the screen in the mobile platform). In terms of the player skills, in the Xbox 360 platform, some players experienced peripheral-related problems caused by system feedback. Even when the players were able to react quickly, the system received the input with a delay, which frustrated the players. However, the players tried to adapt to this defect during play and master the game.

The difficulty aspect of the game can be considered in direct relation to the game completion performance since the Fruit Ninja game is mostly based on game mechanics and dynamics. The mobile platform was found to be less difficult to interact with compared to Xbox. The players expressed their frustrations with the difficulty of playing the game as follows: “I needed to cut the fruit with my left hand!” (P1) and “I tried hard, but maybe next time!” (P7). Furthermore, they responded to negative situations by blaming themselves rather than the system although the main reason was the problems with the peripherals.

Regarding the interaction mechanics of the game in the mobile platform, the players mostly acted intuitively and considered it familiar due to prior experience with similar applications. As a result, the participants could quickly relate to the interaction methods. However, in the Xbox 360 platform, the players found the interaction methods to be new, which made space for discovery. During gameplay, some participants tried to improve their skills using different interaction methods such as making a door knocking gesture (P3).

Immersion:

The immersion heuristics are concerned with conscious awareness, absorption in the game, game realism, control dexterity and socio-cultural proximity with the game. Players acted like ‘ninjas’ mimicking generic martial art movements (such as using the side of their hand to make a cutting gesture to score points). This demonstrated the realism aspect of the system. In other words, the Kinect peripheral provided the players with an opportunity to mimic gestures similar to the observed socio-cultural actions. Furthermore, environmental distractions such as other people talking in the room and noise from the street allowed us to observe some players losing their concentration while some others taking no notice of what was going on in their surroundings.

In the study, the players seemed to be under stress when playing the game on Xbox 360. Unlike the mobile platform, many players tended to react anxiously to the loading screens of the Xbox 360 system, biting their nails and/or loosing focus. However, few players responded in the exact opposite way, enjoying the company of the other participants and even dancing during the loading of the game. Therefore, it can be concluded that in the Xbox 360 platform, there is a certain degree of correlation between social capabilities and immersion in the game.

Satisfaction:

The satisfaction heuristics refer to the gratification or pleasure derived from playing the game. It is characterized by game fun, disappointment with content and attractiveness. During post-tests, the majority of the players stated that they would like to play the game again. Players who preferred Xbox 360 considered this platform to be more fun to interact with due to the novel interaction methods and the bigger screen offered by the system. It was also observed that during gameplay, not only the player but also the other participants had fun since they enjoyed watching or commenting on the active player’s effort. This finding may be evaluated in the context of sociability. In terms of aesthetics and content, both platforms offered very similar aspects, rendering it impossible to make a comparison between the two.

Motivation:

The results regarding motivation give an idea about the game characteristics prompting the player to undertake and continue to perform specific actions until they are completed. The motivation heuristics are characterized by encouragement, curiosity, players’ self-improvement and the diversity of game resources. In the current study, six players were eager to play the game again after the tests. The only reason for the loss of players’ motivation and encouragement was the errors and unresponsiveness of the system which mostly occurred in the Xbox 360 platform. On the other hand, playing in front of other participants increased the motivation to play the game again.

The players were mostly responsive to the different types of fruit appearing on the screen when playing the game on Xbox 360. The players were curious about how the Kinect peripheral worked, experimenting with several different gestures to achieve the goals and asking other participants about their opinions. However, to play the mobile version of the game, the players only had to swipe their fingers across the screen.

Emotion:

The emotion heuristics indicate the players’ involuntary impulses in response to the stimulus of a video game. The characteristics of emotion are player reaction, game conduct and sensory appeal for game elements. The emotion heuristics are analyzed via observation. The players who played the Fruit Ninja game before tended to get more excited when they realized that they could play the game using the Kinect peripheral but after experiencing certain problems with the platform, there was a significant loss of excitement among these players.

In the classic mode of the game, when a big fruit appeared on the screen (as an object to be slashed), the players reacted with enthusiasm, even yelled. One player got so excited that she accidentally pressed the volume button of the mobile phone just after seeing a bigger fruit appearing on the screen. Such actions are examples of impulsive reactions to the given stimulus. In both platforms, the players had fun and control over the game in general and clearly understood the objectives. In the Xbox platform, the participants were more responsive to the actions of the active player and their reactions were also louder when the active player made a mistake.

Socialization:

The socialization heuristics refer to the set of attributes, elements and resources that promote the social dimensions of the game experience in a group scenario. They have the characteristics of social perception, group awareness, personal implication, sharing of social resources, communication techniques and interaction rules. In this study, the players were in constant social interaction during the tests since they were allowed to watch each other play and talk during gameplay. This was to observe the social interactions and provide a natural environment to conduct the tests. In addition, we had the chance to explore whether there were any differences between the two platforms in terms of the socialization aspect.

The observations and interviews showed significant differences between the two platforms with Kinect providing a significantly more sociable environment. In the Xbox 360 platform, the players were able to comfortably watch and comment on each other when playing whereas the mobile platform rather isolated the active player. The Xbox 360 platform was observed to encourage group awareness more than the mobile platform. In Xbox 360, at first, the players were competitive but then they quickly shifted to a more cooperative communication resulting in increased personal implication, group awareness and sharing of social resources. Some players shared their ideas about their interaction with Kinect, asking for other participants’ help to discover more features of the system. For instance, one player asked the spectators, not the test supervisors, whether he had made a mistake saying, “did I press the wrong button?” (P1). Even though the active and passive players could watch and interact with each other as a group also in the mobile platform, they continued to talk about the Kinect peripheral during the mobile test and how they would try to interact with it next time. In addition, the bigger screen offered by the Xbox 360 platform made it more comfortable for the other players to watch the active player during gameplay.

Mobility:

In several occasions, in the Xbox 360 platform, the Kinect peripheral needed re-calibration, significantly delaying the start of the game. These calibration problems also caused errors in the system response such as the game starting itself and/or receiving wrong input during gameplay. The length of time the loading screen was displayed was similar in both platforms. Regarding the game accommodation with the surroundings; circumferential elements such as lighting or outside noise affected the game experience in the Xbox 360 platform more than in the mobile platform.

The Kinect peripheral needed to be precisely setup to work flawlessly and even then, caused several problems during the tests since the natural light in the room changed from time to time. Having other players in the vicinity also had an adverse effect on some players, causing them to lose concentration as explained in the immersion heuristics section. However, during the mobile platform tests, the players were also distracted by other players in the vicinity when they heard them talk to each other. This was not particularly a problem for the Xbox 360 platform since all the players watched the game being played and provided feedback to the active player. In terms of interruptions being handled reasonably; the mobile platform was easy to pause and allowed the players to easily go in and out of the game as opposed to the Xbox 360 platform in which none of the players were able to stop or pause the game when asked.

5 Conclusion and Future Work

This study analyzed multi-platform PX based on a playability heuristics framework derived from the literature on usability and UX. The proposed framework was tested on the same game in two different platforms to seek ways to contribute to the enrichment and improvement of the existing PX frameworks. Based on the results, it can be concluded that the players were glad that they experienced playing the game on Xbox 360; however, they lost interest in the game due to the playability problems they encountered. At the end of the tests, most players stated that they preferred to play the game on the mobile platform rather than on Xbox 360 with the Kinect peripheral.

Despite both platforms offering identical game mechanics, the players had problems reaching certain areas of the screen using the Xbox 360 system due to the bigger screen size. Furthermore, the big screen size negatively affected the players’ reaction time to game stimuli. In terms of playability, the mobile platform was more preferable in general but the novelty of the peripheral included in the Xbox 360 platform enhanced the experience. The motivation heuristics were improved since the peripheral encouraged the players to experiment with other possible interaction methods. The results on the immersion heuristics were better for the Kinect peripheral due to its intrinsic quality of rendering the player a direct input and increasing realism going beyond the limitations of the game.

The results on the socialization heuristics indicated that the Xbox 360 platform with a bigger screen size and the Kinect peripheral better enhanced the social experience. However, in terms of effectiveness, the mobile platform players achieved the goals faster and scored better. The Xbox 360 version of the game lacked the necessary tutorials and/or indicators for players regarding some aspects such as pausing the game, thus resulting in frustration. The players stopped interacting with the game from time to time and therefore lost interest in the game. The Kinect peripheral and the bigger screen size in the Xbox platform allowed more social interactions during and even after the tests. In addition, the interaction problems and system errors did not affect the socialization aspect. With its quality of being new in terms of the interaction method used, the Xbox 360 platform caused more excitement at the beginning. The full body gesture-based interaction also enhanced the immersiveness of the game, allowing the players to role-play during the game. However, in terms of mobility, the Kinect peripherals were susceptible to interruptions and the environmental elements were less forgiving. Yet, none of the players were observed to leave the game atmosphere. We consider that games designed specifically for the Kinect peripheral would provide better PX compared to those adapted from the mobile versions.

In the development of the proposed framework, several methodologies were taken into consideration to form a feasible and comprehensive framework to better analyze and understand PX. In this study, we introduced a new framework in which we grouped the game elements into two as Off-Screen and On-Screen elements for a comprehensive and complementary analysis of PX. We found that the mobile heuristics were specifically identified for mobile games and platforms, yet there were limited number of heuristics regarding peripherals such as Kinect and console games that are significantly different in terms of the screen sizes and/or the environment they offer.

Future research is needed to identify interchangeable and specific modules of heuristics for specific platforms. There is also the need for comprehensive comparative studies to increase the generalization and validity of our results. In future work, we aim to conduct similar tests with a bigger sample size and in comparable environments to further analyze the proposed division of On-Screen and Off-Screen elements, placing a greater emphasis on the Off-Screen aspects of the experience.