Game Learning Analytics: Learning Analytics for Serious Games
Video games have become one of the largest entertainment industries, and their power to capture the attention of players worldwide soon prompted the idea of using games to improve education. However, these educational games, commonly referred to as serious games, face different challenges when brought into the classroom, ranging from pragmatic issues (e.g., a high development cost) to deeper educational issues, including a lack of understanding of how the students interact with the games and how the learning process actually occurs. This chapter explores the potential of data-driven approaches to improve the practical applicability of serious games. Existing work done by the entertainment and learning industries helps to build a conceptual model of the tasks required to analyze player interactions in serious games (game learning analytics or GLA). The chapter also describes the main ongoing initiatives to create reference GLA infrastructures and their connection to new emerging specifications from the educational technology field. Finally, it explores how this data-driven GLA will help in the development of a new generation of more effective educational games and new business models that will support their expansion. This results in additional ethical implications, which are discussed at the end of the chapter.
KeywordsSerious games Game learning analytics Learning analytics Game analytics Educational standards
Nowadays, video games are one of the most popular entertainment activities and one of the largest entertainment industries, with the total number of gamers increasing every year. In the United States, more than 55 % of the population plays video games, and 42 % of them play for at least 3 h per week, with an average player age of 35 years (Entertainment Software Association, 2015). Some reports describe that over 70 % of children and teenagers across the European Union, and over 90 % in the United States, play videogames (Granic, Lobel, & Engels, 2014; ISFE, 2014). In addition, these figures keep growing rapidly with the generalization of smartphones and other mobile devices as potential game platforms.
All these players devote time to games because they provide engaging and motivational content that connects with the players more deeply than linear forms of media and that presents scenarios where players are challenged (rather than forced) to perform better. This has prompted a rapidly increasing interest in using games for educational purposes, not simply because “it is what kids are paying attention to,” but because the design of good games is very closely aligned with the design of good educational experiences, always trying to push the players/students to reach just beyond their current competence levels (Koster, 2004).
However, several challenges are slowing the acceptance of games as a trusted and powerful educational resource. These challenges include the significant development cost of digital games, a lack of understanding of how the students interact with the games, and a general lack of tools to improve our understanding of the educational impact that the games actually have on students. These challenges must be overcome if we wish to fully realize the potential of games as learning tools.
This chapter explores the nature of these issues and focuses on the impact of combining existing knowledge about learning analytics (an emerging research and assessment tool in educational settings) and game analytics (an emerging research and assessment tool in game development) to better assess and understand how games affect education and training.
Predict student performance
Provide students with personalized and scaffolded game experiences
Increase student retention rates (i.e., fewer dropouts)
Improve the design of future serious games
Improve the cost efficiency of using games in education
With these goals ahead, this chapter is structured as follows: section “Serious Games” describes in greater depth how games can (and should) be used in education, turning them into what is commonly referred to as serious games, along with the main challenges that could be tackled by introducing better research and assessment techniques. Section “Data-Driven Analysis of User Interaction” is focused on the two previously existing fields (learning analytics and game analytics), describing their current goals, tools, and expected results. Then, section “Game Learning Analytics” describes the combination of those areas, describing a general analysis and visualization approach along with a series of scenarios contemplated to improve the educational impact of games. This section also contemplates the specific challenges of bringing these analytics techniques into serious games. After this general introduction, section “Standards and Supporting Projects” focuses on how to address some of the technical challenges using current e-learning standards. Section “Business Models” goes further ahead and explores business and application models that facilitate the acceptance of serious games in different settings. This exploration of the future ahead is continued in section Ethics and Risks, focused on the ethical implications of collecting massive amounts of educational data, a problem that requires further attention. Finally, section “Concluding Remarks” summarizes the key points from the chapter and outlines the main conclusions.
The term serious games typically refers to any use of digital games with purposes other than entertainment (Michael & Chen, 2006), including training simulations, social criticism games, and advergames (games created to promote a product, a service, or a company). Nowadays, this term is used even for gaming approaches applied to solving complex problems in a collaborative way. However, most academics typically use the term serious games to refer to the use of digital games for educational purposes (some authors consider that the term was coined by Clark Abt in his book “Serious Games” in 1970 (Djaouti, Alvarez, Jessel, & Rampnoux, 2011)).
This idea of using games to improve education has been around since the publication of the first commercial digital games (Malone, 1981), but it has been in the past few years that it has really gained traction within the academic community. In fact, in their literature review, which included more than 7000 papers concerning entertainment games, educational games and serious games, Connolly, Boyle, MacArthur, Hainey, and Boyle (2012) highlighted the diversity of research on positive impacts and outcomes associated with playing digital games. Even if some detractors consider that playing games is associated to additional problems, such as violence and addiction, most recent studies point to the benefits clearly outnumbering the negative effects (Granic et al., 2014). In fact, although game adoption has not yet been generalized in the educational arena, the academic debate has moved beyond discussing whether games have educational potential to refocus on how to create better and more effective educational games.
While, in their origins, serious games were used mainly for training in specialized domains such as medicine or military, it is now easy to find examples from language learning (Baltra, 1990; del Blanco, Marchiori, & Fernández-Manjón, 2010) and engineering (Ebner & Holzinger, 2007; Mayo, 2007), while in the workplace, examples range from basic skills for the inclusion of disabled persons in the workforce (Torrente et al., 2014) to advanced skills such as leadership (Aldrich, 2004).
However, to encourage the use of video games as learning tools, there is a need to develop a deeper understanding of how video games actually affect the learning process, of the skills and techniques that games can provide, and of the way that they can be matched with student preferences. This deeper understanding forcibly starts with better assessment – which can be approached from two complementary sides: assessing the educational effectiveness of games (e.g., via traditional pre-post experiments) and using games themselves as assessment tools.
The next subsections focus on the potential of assessment (both within and with games) and analyze some of the barriers and limitations that prevent a wider acceptance of serious games, together with new opportunities enabled by technical and societal changes.
Assessment and Game Evaluation
In educational technology, there is an increasing interest in the concept of evidence-based education, where educational models intended to improve teaching and learning can be validated against actual data obtained from of their application. This trend relies on developments in computer data processing capability and is creating new assessment opportunities (Bienkowski, Feng, & Means, 2012). Any interaction with a computer can be captured and, either in real-time or at a later date, analyzed.
Basic information collection has always been done with learning management systems (LMS), providing a degree of insight into student actions. Games are intrinsically more interactive than static learning materials such as text documents, slides, or quizzes and therefore have the potential for revealing much more information. In contrast to static learning materials, games already take into account user actions during game play, for example, to adjusting game play to the current user or to provide a final score. However, from the educational perspective, even if the final score is useful as a summary of user proficiency in the game, the detailed gameplay may hold far more valuable information about how the players interacted with the game, such as where the user became stuck, or where the user encountered problems understanding a key concept.
In terms of using the games for assessment purposes, detailed analysis of game interactions can yield detailed information about how each player interacted with the game, providing instructors with insights into the gameplay session. However, the information yielded is often hard to transfer to grades. If these game-play behaviors could be distilled and presented in a meaningful way to teachers, they could be a very powerful source of information regarding student misconceptions and progress in the mastery of targeted concepts.
In turn, these in-depth analyses of player interactions can also provide valuable lessons on the design of the game itself. Sometimes the lack of learning effectiveness may be derived from wrong design decisions, such as overly long sequences of text that the players skip, or specific levels that players are unable (or excessively challenged) to escape, and should therefore be redesigned. Gaining a better understanding of how players interact is a pathway toward creating better and more effective serious games.
Limiting Factors and New Opportunities
Regardless of the increasing acceptance of serious games, it is true that there are limiting factors that are hindering serious games adoption within educational institutions and enterprises. At the same time, technology, devices, and game businesses are rapidly evolving, creating new opportunities that can increase their impact and effectiveness.
Among these limiting factors, some should be highlighted: (1) the high development and maintenance cost of serious games, (2) concerns about how to effectively deploy serious games in educational settings, and (3) for teachers, and especially those that grew without videogames, a steep learning curve to make effective use of games.
Game Development Cost
Regarding the game development cost, serious games usually have a much smaller budget compared to high-quality commercial games (usually called AAA games). Currently, most serious games are funded by governmental agencies, research projects, or nonprofit organizations. As the SG market matures, the entertainment game industry’s economic model, where high costs can be compensated by massive sales, is not yet available to would-be SG developers. This is beginning to change, with increasing industry interest and funding directed at learning technologies in general and serious games in particular. This will create new economies of scale, enabling larger deployments to larger cohorts, thus reducing their cost per student.
New commercial authoring tools (such as Unity3D or Unreal) provide easy and almost free initial access to their platforms. Both allow the creation of high-quality games without deep technical and programming knowledge, vastly lowering the barriers to game creation. These tools also reduce development costs by supporting cross-platform development: games can be designed to be playable on multiple platforms (e.g., PC, Android, IOS, and Web), without the developer having to rewrite substantial parts for each targeted platform. Cross-platform development also simplifies after-launch game maintenance greatly and isolates games from constant changes in technology. This is highly positive for educational games, since once that games become key elements of learning, their contents will need to be open to adaptation, and the games themselves will have to remain playable on newer devices.
Despite the use of sophisticated authoring tools, the complexity of creating quality educational games is still beyond the reach of most users. While the technical and programming requirements may have decreased, most of the cost of developing games is incurred in other tasks, such as the creation of artistic assets (including graphics, animation, music, etc.) or, for serious games, adequate instructional designs.
Game deployment refers to the process of making the game available for its players. Traditional PC games must be installed in each machine prior to play, may conflict with existing software and/or the underlying operating systems, and must be configured to report correctly to the learning analytics platform in use. With shared facilities, and institutional red tape to deal with, setting up a game in a classroom can be a daunting task for any but the most determined teachers.
The above scenario is changing fast with improved Internet connectivity and technological changes such as HTML5, which allow games to be playable on modern browsers, regardless of underlying operating systems and without requiring any additional software installation. Multi-platform games can even eliminate the need for labs, since students can play them on their own game devices, such as laptops, smartphones, or tablets. Mobile devices are also contributing to change the total number and demographics of the gamers, with casual gamers (people that play games in their smartphones or tablets during short periods of time) on the rise (Entertainment Software Association, 2015).
Teachers play a key role in SG adoption, as they can be the main drivers in the change but often report that they lack adequate training (e.g., pedagogical approaches) and are therefore not sure of how to integrate games into their teaching (Takeuchi & Vaala, 2014). Increasing teacher adoption requires further insights into how students actually play and learn, the availability of games better adapted to the corresponding curricula, and supporting tools that help teachers know what is happening when the games are deployed into the classroom.
Gaining additional insight into how students play, learn, and improve their skills is a key enabling factor for a wider adoption of serious games: it would facilitate their acceptance by instructors, by allowing them to “peek” into the learning process and facilitating their intervention when required.
Regardless of these potential limitations, there are also new opportunities that can boost the adoption of serious games. Increased penetration of technology, and in particular mobile devices, has had an important impact in the demographics of gaming, with the rise of casual games and gamers.
From the point of view of learning technology, one of the better opportunities is due to the generalization of the massive online open courses (MOOCs). MOOCs offer a new opportunity for serious games, as creators are interested in providing new interactive content that engages the user and could help to mitigate the high drop-off experienced in courses. However, there is no standard way to include games as content in MOOCs, and MOOC APIs (application programming interfaces) do not provide support for game integration beyond considering it a type of custom exercise. This means that use of a serious game in MOOCs currently requires the full programming of the ad hoc integration not only to launch the game but also to connect the game outcome with the MOOC management and visualization facilities (Freire, del Blanco, & Fernández-Manjón, 2014).
Finally, wearable technologies constitute an important qualitative leap in digital gaming and simulation. These devices can provide gamers with new ways to access information and interact with it, anywhere and anytime (Barfield & Caudell, 2001). In recent years, the explosion on these kinds of technologies and the corresponding reduction in cost has allowed educators to research and apply it in some specific fields (such as medicine). This technology could become an effective tool to enhance learning; but game authors and educators must first find out how to take advantage of the huge amount of data that wearable devices can provide in serious games.
Data-Driven Analysis of User Interaction
The analysis of large sets of user interaction data is another trend that has experienced a very fast growth over the past few years. The web analytics techniques used by different service providers (e.g., Google, Facebook) are an example of the insight that can be gained from exploring big datasets.
This trend has also extended to the two domains that serious games typically connect: education and digital games. Each area has developed its own set of techniques to analyze how their users (students and players) interact with digital contents: learning analytics in the case of education and game analytics in the case of digital games. Next subsections describe them in more detail.
Many authors have attempted to define learning analytics in the last years. According to Baker and Inventado (2014), it can be defined as the exploitation of data to benefit education and the science of learning – although emphasizing human interpretation of data and visualization, to distinguish it from Educational Data Mining, which relies more heavily on automation. Tanya Elias (2011), in Learning Analytics: Definitions, Processes and Potential, provides a more functional definition, where learning analytics are what allows data collected in learning management systems (LMS) or content management systems (CMS) to be used to improve teaching and learning. In a broader definition, other authors prefer to define learning analytics as the exploitation of educational datasets – collected by interactive learning environments, learning management systems, intelligent tutoring systems, e-portfolio systems, and personal learning environments (PLEs) – for the evaluation of learning theories, learner feedback and support, early warning systems, learning technology, and the development of future learning applications (Greller & Drachsler, 2012).
Nevertheless, one of the most accurate definitions was stated by Long and Siemmens and supported by other authors: “Learning Analytics is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Long & Siemens, 2011; Siemens, Dawson, & Lynch, 2013). The key element here is that this is a data-driven process. Sometimes, as noted by Baker and Inventado (2014), collection and analysis have been considered to overlap with Educational Data Mining (EDM), although other authors prefer to use the term EDM to refer to the tools of analysis, rather than their application (Bienkowski et al., 2012).
One of the simplest incarnations of learning analytics focuses on tracking how large numbers of students interact with an online learning management system (LMS) simply by consulting the access logs from the web server (Arnold & Pistilli, 2012). An in-depth analysis of such logs can potentially identify behavior patterns that correlate with academic failure (e.g., students that seldom access support materials or with long periods between each login) so that early remediation actions can be sought with students that exhibit such behavior. And while the example above may be too obvious, when the reference dataset is large enough (e.g., thousands of students), this becomes a big-data problem where intelligent data-mining techniques may be able to detect less obvious correlations (Ferguson, 2012). This vision of learning analytics as a big-data problem has become more relevant along with the growth of the interest in massive open online courses (MOOC), in which the very large student cohorts can generate huge datasets (Clow, 2013) and the ratio of instructors to students is much lower than in traditional settings, making any and all automation especially attractive.
However, the potential power for analytics increases as the learning materials become more complex. In this sense, the vision of the learning object (a reusable, standardized package of “learning” to be combined and deployed into lessons on any standard-supporting system) contemplates that an individual piece of learning could be tracked within an LMS (Polsani, 2003). When this piece of content is, for example, a PDF file, the information available to the system is focused exclusively on the moment in which the student downloaded the file and the moment in which the student requested another file. However, learning objects have grown into more complex assets that may run in the student’s computer and connect back to the server to communicate tracking information (Torrente, Moreno-Ger, Martínez-Ortiz, & Fernández-Manjón, 2009). Standards such as IMS Common Cartridge and SCORM (Sharable Content Object Reference Model) standardize this type of progress reporting (del Blanco, Marchiori, Torrente, Martínez-Ortiz, & Fernández-Manjón, 2013), focused on scores and completion rates, hence enabling a rich interaction model that can serve as a starting point to collect additional analytics-oriented interactions, with a higher levels of abstraction than those available in LMSs’ comparatively raw event logs.
Once data are ready, there are two prongs in “traditional” learning analytics – the teacher side and the student side. From the student side, the main aim is to let students know how they are doing in the course. Self-assessment and motivation are important elements in online courses, and learning management systems may use these data analysis techniques to let the students foresee their potential outcomes by comparing their performance with other students.
The other side of the equation is that of teachers/instructors, who need to track progress both to adjust the speed and contents of their courses and to inform grading. These analyses may help instructors identify struggling students and maybe even offer specific remediation actions for those students. Furthermore, the availability of assessment, tracking, or classroom management characteristics are some of the key elements that teachers consider when selecting applications to be used into their classrooms (Takeuchi & Vaala, 2014). Even so, some teachers are reluctant to use what they call “machine evaluation,” mainly because they do not fully understand the underlying technology. While automated evaluation may provide more student data, it is the teachers that choose to use it who have the final say and who will bear the responsibility for errors. Thus, making evaluation more transparent for the educational system is important to avoid this source of distrust.
Additionally, once a learning analytics system is able to predict a negative outcome and propose remediation actions, real-time intelligent adaptation becomes a real possibility. The analyses can be used to build “learner models” that can be applied to propose customized lesson plans for the students (this may add a new twist to the field of adaptive hypermedia (Brusilovsky, 1996)). And while the field of dynamically adaptive learning environments has been somewhat stagnant in the last few years, part of the reason for this is the challenge of creating and maintaining user models, a task that may be facilitated by using learning analytics techniques.
Game analytics is the term used by the video game industry for the application of analytics to game development and game research to better understand how users play their games, find errors, and improve the gameplay experience (Seif El-Nasr, Drachen, & Canossa, 2013). While the purpose of learning analytics is to support the online learning industries, that of game analytics is to support the growth of digital (entertainment) games (Loh, Sheng, & Ifenthaler, 2015).
Entertainment game developers face significant challenges in terms of creating games with a good user experience. By making developers aware of how their customers actually play games, the user experience can be significantly improved, driving game sales and, for certain game types, in-game purchases.
Game analytics systems can collect many types of data. Depending on what aspect of the game cycle they are related to, those data could be seen from two different perspectives: one more technical about the game and the game infrastructure, and another more focused on the user data and experience.
From the technical perspective, some systems track metrics of the game development process itself, such as the number of bugs in the code, the time to fix them, or how this number varies with each new version of the game. This helps programmers to keep the development process under control, a major concern in complex pieces of software such as games. During testing and deployment, other systems record performance metrics such as frame rate or memory usage in the machines where the games are installed. These metrics can reveal hardware and software bottlenecks that prevent the game from running smoothly.
Game metrics can be leveraged by serious games. A game (educational or not) can generate vast amounts of interaction data, indeed, far more than almost any other form of content, since game interactions are typically built around a very short feedback cycle of interaction-reaction (Van Eck, 2006). This means that even a short gameplay session can generate very large amounts of data. Taken to an extreme, it is possible to reconstruct a complete sequence of the player’s actions, instant by instant, by replaying gathered interaction data. This is frequently used for nonanalytics purposes, such as allowing real-time spectating of gameplay from the point of view, possibly with added commentary, or viewing such gameplay at a later date.
The application of data-mining and visualization techniques to player interaction logs can provide very valuable insights to game developers of how their players are interacting with the game. Such analyses can be performed in small groups (e.g., a small round of beta-testing) or, as is the case with many mobile games, after their deployment, gathering data from players playing the game worldwide (this remote collection of data is usually called telemetry).
One of the most typical aspirations of such analyses is the identification of typical stumbling points. Game developers may be interested in knowing whether some levels or game situations are excessively challenging (or excessively easily), so that the game design can be improved. The analysis of how a large cohort of players proceed through the game can be very helpful in determining these specific stumbling points.
Other analyses focus on identifying potentially unreachable areas (that are included in level design but are never visited by players), on conflicting game states (e.g., unexpected situations that generate software errors) or even detecting popular game areas where players spend more time interacting significantly. This last example is especially relevant, since it opens the gates to new monetization opportunities, either through targeted advertising or, more frequently in the past few years, by allowing game developers to deduce when is the most appropriate moment to suggest a micro-transaction within the game.
All these game analytics techniques have evolved separately from learning analytics, with a very different vocabulary and typically with a different agenda (e.g., improving player retention or revenue versus improving learning). However, both disciplines look for games with a better user experience and offer very exciting opportunities when combined in an educational game. There are already some research systems that try to capture in-game data on play and learning such as ADAGE (Assessment Data Aggregator for Game Environments), a clickstream (telemetry) data framework that looks inside the data stream of educational games (Owen, Ramirez, Salmon, & Halverson, 2014).
In the same vein, some efforts of game designers focus on “stealth assessment” (Shute, 2011) which refers to the nonintrusive background (or deferred) analysis of player interactions with assessment purposes. This allows the analysis of player interactions that would not be traditionally considered gradable outcomes. Instead of conducting evaluation only at the end of the game and usually based on the score or final game state, any interaction can be “mined” for assessment purposes. This opens up two related scenarios in terms of games and assessment: (1) the use of games as assessment artifacts and (2) the assessment of serious games in terms of their educational effectiveness, the adequacy of their design, or the identification of potential stumbling points (Bellotti, Kapralos, Lee, Moreno-Ger, & Berta, 2013).
At first sight, these two aspects may seem interchangeable (and indeed, they are often mixed). A very common approach when validating a game is evaluating them exclusively in terms of their learning outcomes – known as the “if the players learn, the game is valid” approach. This is an oversimplification that neglects many of the advantages of using serious games: increased motivation and engagement, a deeper understanding of the underlying principles being taught, and the experience gained through exploratory interactions in a safe environment. Additionally, some authors consider that games provide a more authentic learning that is often not adequately reflected in traditional evaluations (e.g., written tests) – questioning the method used to evaluate learning itself (Shute, Ventura, Bauer, & Zapata-Rivera, 2009; Torrance, 2007) .
Game Learning Analytics
When creating a serious game, the educational goals of learning analytics and the tools and technologies from game analytics should be combined, in what could be called game learning analytics (GLA). This combination can contribute to a generalization and a better use of the serious games. Having data of what is happening while the user is playing is key to relating gameplay with actual learning and to move from only theory-based approaches to more data-driven or evidence-based approaches. This in turn can help to contrast the educational approaches and eventually to better understand how the learning process happens.
Instrumentation – The game-side components required for the game to periodically store information on player interaction. Traces could then be sent to the collection and storage server in batches, to accommodate for offline play and to reduce the number of small remote data transmissions that may put too much pressure on the collection server.
Collection and storage – A server-side component to receive, classify, and store all interactions sent by the instrumentation, allowing future querying and aggregated analyses.
Real-time analytics – It is highly desirable to access key analytics in real time (or with minimal delay). As the next section will show, this would allow an instructor to make targeted interventions during a gameplay session to maximize the learning effectiveness. These reports should be based on lightweight analysis and may display the last interactions of each player or the number of current players. Typically, they operate with “time windows,” such as “in the last 5 min.”
Aggregated (batched) analysis – Much higher aggregation, based on more complex analysis, makes sense when stakeholders need a broader view of different gameplay sessions. These analyses need to run overall interaction data collected and aggregate results from each individual gameplay.
Key performance indicators (KPIs) – Educators can identify quantifiable outcomes as KPIs, a term borrowed from business intelligence. Grades, completion, or educational effectiveness are key measures in educational contexts, and analytics systems should attempt to find links between other data features and these KPIs.
Analytics Dashboard – The sets of analyses and related visualizations are frequently encased in “analytics dashboards,” to be queried by stakeholders. They should provide a general overview of key indicators and ideally be configurable to suit the user’s needs, such as allowing new analyses to be set up and launched. A good interface allows overview, zoom, and filter and to look for details on demand (Shneiderman, 1996).
Through these elements, there are many possible scenarios and potential applications to facilitate the assessment of the effect of serious games, as well as the design of the games themselves.
GLA benefits, however, come for a prize: those educational games need to be designed to support learning analytics and must gather and send additional aspects of player interactions and progress back to the LA framework, even though only some of those aspects will actually be used to improve gameplay. This allows observable game behaviors to be tied to a competency-based model that can be used to infer learning outcomes. During game design, its developers should identify educational situations that are relevant to the competency-based model and determine how they will be assessed from within the game and later through analytics. Streamlining this process for developers in a game-independent way is challenging, since it requires certain aspects of the game state (usually is represented in the game at a much lower level, such as variables, flags, or identifiers) to be communicated back to the LA server (Hauge et al., 2014).
There are very different usage scenarios where game learning analytics could be used depending on the kind of information collected and how and when this information is analyzed and accessed.
One scenario is based on a real-time gathering, analysis, and presentation of the GLA information. For instance, it is difficult for a teacher to know what is happening in a classroom when games are used. The idea is to simplify teachers’ task when using games by providing real-time information of the actual students’ interaction with the games while in the classroom. This approach delivers assessment data to teachers in real-time, making information available in a mobile device so that they gauge the classroom’s general level, help students that are struggling with the game, or identify those that outperform and could benefit from additional activities (Freire et al., 2014).
A different scenario is that of off-line analysis. This implies the collection of all the interaction data for later analysis once the game-playing session is finished. It represents an invaluable source of information on how the targeted population has actually played the game. Once collected, datasets can be aggregated in different ways and subjected to a large variety of analyses. Results obtained at this stage can shed light into the real effectiveness of the game as a learning tool. For instance, low scores reached by some students in the game’s final assessment could be related to specific causes such as misunderstanding of game rules or faulty pedagogical designs (i.e., due to a certain learning goal that has not been adequately conveyed in the game). Therefore, while real-time analysis addresses users’ behavior, off-line analysis can reveal patterns in how the student interacts with the game throughout the experience.
When designing a game, information on how a specific population plays a game, and their learning strategies, could mark a milestone in the way the game is developed to better fit the preferences of its target audience. Many researchers pointed out how gender differences do affect the effectiveness in game learning (Chou & Tsai, 2007; Lowrie & Jorgensen, 2011; Papastergiou & Solomonidou, 2005). However, recent studies put the spotlight in a different place: the broad play style (such as “casual” or “hardcore”) of players can explain those differences better than their gender does (Manero, Torrente, Fernández-Vara, & Fernández-Manjón, n.d.). Thus, collecting and analyzing data from previous games or versions of a game can become an integral step when improving game design that targets a given demographic.
Information extracted from game analysis could provide a priceless insight into the reliability of the game itself. Those results could add a new variable to our equation: we must measure not only how a concrete tool improves students’ knowledge but also the effectiveness of the tool itself. Data provides information about what students finished the game, the amount of time spent on completing each stage (or mini-game), or whether the gamers read the provided instructions for a specific challenge or made quick click to reach the following screen.
Therefore, GLA should be a useful tool to compare and contrast the educational results obtained by serious games when their users are grouped according to different demographic characteristics (such as gender or age). As an example, as mentioned above, studies have demonstrated the influence of gender in the effectiveness of serious games. However, those studies based their results either on in-game assessments or in paper-based questionnaires. GLA may effectively enrich the knowledge on why a particular game is working better for males or females, shedding light on questions such as: What part of my game is most effective for each gender? Or why is a game targeted at females not working as expected? Or even, what should I change in my game to make it more gender oriented? Those and other questions may find their answers in in-game collected data and further analysis. Moreover, gender differences are only the tip of the iceberg in demographically tailored serious games. Different demographic factors such as socioeconomical status (Sánchez & Olivares, 2011), cultural differences (Guillén-Nieto & Aleson-Carbonell, 2012; Hainey et al., 2013), or gaming profiles (including gamers’ preferences and habits toward videogames) (Manero et al., n.d.), among others, could be at stake whenever GLA starts yielding tangible results.
Even if companies and educational organizations recognize the potential of learning analytics, its use in games is still very limited in the market. A survey with 21 European game studios reflects that there is a high interest in the application of learning analytics within games, but its actual use is scarce and companies are afraid of complex and cumbersome implementations (Saveski et al., 2015).
Infrastructure and Cost
Collect traces from games that are being played. In Fig. 3, the tracker component embedded in the game sends these traces to the collector. The collector de-identifies traces, so that only class instructors can re-identify them later on.
Analyze the data to feed analytics queries, either as the data is being received, at a later date, or only when requested to analyze it. In Fig. 3, the traces received in the collector follow two paths: one to a real-time analysis module, where some results are calculated instantly, and another to storage, where they will be available for batched analysis.
Report and accept analytics queries from suitably authorized stakeholders (instructors, managers, game developer, students) with varying degrees of access and anonymization. In Fig. 3, the dashboards access the results from the both real-time and batched analysis and display them to different stakeholders.
Collecting game traces in a robust and scalable fashion requires a high-availability, high-bandwidth service that can guarantee that all incoming traces will be processed (possibly throttling sources to preserve bandwidth) and additional infrastructure to adequately classify and anonymize these traces.
Analyzing large volumes of data requires scalable, fault-tolerant analytics setups that guarantee that all traces are adequately processed, even in cases where particular processing nodes may fail from time to time, as well as additional support for setting up what analysis should be performed when on what data.
Process and store incoming traces.
Manage exact analysis types to be performed.
Perform real-time analyses on incoming traces.
Perform batched analyses on stored traces, either before switching to real-time (when a new real-time analysis is being set up) or during interactive data exploration.
Manage authorization and authentication.
Allow authorized users to configure, access, and alter/interact with analytics dashboards.
Feed the dashboards of authorized users with (possibly real-time) data.
A maintainable and scalable analytics solution should be built from existing software that can already address individual concerns or sets of concerns. For example, analysis may rely on Storm or Spark – both mature projects that focus on analyzing large volumes of information in a scalable and resilient fashion. In this sense, off-the-shelf software can cover many aspects ranging from the Learning Record Store (which stores and provides access to traces), authentication and authorization, or the generation of visualizations.
Tasks that do not need the same degree of fault tolerance and scalability can be implemented with standard web interfaces. For example, configuring the analyses themselves is much more infrequent than actually running them and therefore does not require as much attention to fault tolerance or scalability. However, designing an intuitive user interface for analytics that is both powerful for experts and welcoming for novices is a significant undertaking in and of itself.
To streamline integration with third-party modules, it is highly recommendable to follow a service-oriented architecture (exposed, for instance, via a REST API) that does not require human intervention to perform as many tasks as possible. For example, new modules that can automatically authenticate themselves and register to receive updates on certain types of events are much more useful than those that must be painstakingly configured and refreshed manually.
Contrary to what the high number of tasks and services may seem to imply, this chapter have only described a minimalistic setup; additional integration with other educational institutions’ systems would add more requirements and modules to the system.
Standards and Supporting Projects
GLA opens a large number of opportunities to improve both serious game development and student assessment. However, in addition to the specific technical issues described in previous section, the GLA platform must be integrated with the current educational platforms and SG development platforms in order to achieve a successful ecosystem. This problem is not new for serious games, because their adoption supposed a great challenge at different levels: organizational, educational, but also from a technical point of view. Serious games have to collaborate in a wider and already established educational ecosystem that is dominated by learning management systems (LMSs). Game integration in these contexts has been usually shallow, with a very limited communication with LMS, which is developed ad hoc for each game and specific LMS combination (Shute & Spector, 2008).
The interoperability problem has been already studied by del Blanco et al. (2013) where different integration and deployment models for serious games in LMS depending on the capabilities and standards’ support of the LMS are proposed (such as using SCORM or IMS). Although with limitations, the proposed models show how to integrate a serious game with the myriad of existing LMSs (commercial, open source, and proprietary) just by using existing e-learning standards. Still, the most frequent model is the “black box” where the LMS is used only to distribute the game, with minimal information being sent back to the LMS. The lack of detailed interaction data prevents conducting further analysis. The use of e-learning standards has alleviated the integration complexity, reducing the N-game-LMS integration problem to a compliance problem with a reduced set of standards. Thus, this very same approach can (even more) alleviate the integration scenarios between LA platforms for serious games and general LA platforms.
Subject of the event, That is the person, tool, or, in general, the actor that carried out the action that generated the event.
Action of the event. The action, operation, or, in general, interaction that has been performed as part of the activity that generated the event.
Object of the event. The target, subject, or, in general, the domain element involved in the interaction.
This data model is therefore centered on tracking activities such as “John played Tetris,” where John is the subject, played is the action, and Tetris is the object.
Some additional information can be added to this data model. For instance, a next step could be including the activity’s context, the result of the activity, or the authority that asserted the validation of the activity. Moreover, both xAPI and IMS Caliper offer a common vocabulary for actions and object description that it is extensible (allowing for the creation of specific application profiles that can suited to specific domains or applications).
SGs generate activity events of different granularities. Some of them can be low-level events, such as raw interactions (mouse clicks, keyboard strokes, screen touches, etc.), or other, higher-level interaction events (a player enters or exits game areas or grabs/uses in-game objects); and yet others can be coarse-grained events that aggregate low-level events in a meaningful way (a player completed a level or scored 1000 points). Using their extensibility, both xAPI and IMS Caliper can describe multiple types of events.
Both initiatives decouple the collector API from the actual storage component, allowing the replacement of components and enabling an ecosystem where tools from multiple vendors can coexist. For instance, this would be compatible with third-party services that analyze low-level events and aggregate them into coarse-grained events that are also collected and stored. As of this writing (October 2015), the xAPI is more evolved in this regard, offering not only an API to track events but also an API to search and query events for a particular subject (actor). Using third-party analytic tools, the query API can also be used by the SG platform itself in order to adapt its behavior and adjust game difficulty for each player, querying the results from the analysis calculated in the server.
Another key aspect to make the most advantage of LA is to provide instructors with tools that facilitate reporting and evaluation. The reporting aspect can be useful for both the instructor and the student. Particularly for the latter, it would be easier if the reporting tools are integrated directly inside the LMS, hiding the technical details required to access the LA platform. In the past, this type of integration required the development of specific extensions (e.g., Moodle blocks/plugins, LAMS/SAKAI tools, etc.) for each supported LMS. To address this issue, IMS proposed the IMS learning tools interoperability (LTI) specification (IMS Global Consortium, 2015b), providing a generic means to launch third-party external tools directly from within the LMS.
Although the IMS LTI specification was intended to be used to integrate meaningful learning activities (e.g., to integrate the SG platform inside the LMS), it is possible to abuse the specification as a single-sign on facility in order to allow the instructor to embed visualization widgets (e.g., leaderboards, flame graphs, etc.) that can be useful for students in assessment reports. In addition to the integration of visualization tools, IMS LTI allows sending data from the LA platform to the LMS assessment record (score), which will be stored in the LMS’s gradebook.
In this same line of work, the SCORM-to-TLA Roadmap (Advanced Distributed Learning, 2015) ADL describes four phases for transitioning from standard packaged content and an LMS-centric approach to a more distributed one based on collected information (i.e., learning record store) or to full service-based learning platform where an LMS is not even required (TLA stands for Training and Learning Architecture). In all those cases, the role of the corresponding specification (SCORM, xAPI) is described.
The application of learning analytics to serious games opens up a new range of possibilities of new business models and even new services. The availability of actual user interaction data from game deployments can have a substantial effect in the market situation, since a plethora of data can exploited with new purposes. Here, the issue of data ownership achieves new relevance. In many cases, when the game is deployed in app marketplaces (Apple App Store, Google Play), the marketplaces provide certain useful services (e.g., number of installations) – but most of the interaction data are neither collected nor kept in the marketplace. When either the developer or the client can collect all the interaction data, new opportunities based on the actual use of the games arise.
The next section presents some ideas of the new business models enabled by pairing serious games with learning analytics.
Serious Games as a Service
Instead of selling a specific game to a specific client, the service of using the game can be sold. This greatly simplifies deployment for the client (say, an educational institution), which would not have to provision any of the supporting infrastructure and could rely on the provider for maintenance and support.
This is already beginning to happen; examples can be found at https://www.glasslabgames.org/, where individuals or schools can buy different kinds of subscriptions to educational games.
However, this approach requires a clear model of accountability and new structures for organizing the access to the game (e.g., class or group organization). Additionally, when two or more parties share a product such as a serious game as a service, the authorship and use of the gameplay data generated by game could become a point of friction. In these scenarios, stakeholders should clarify who owns the game-collected data, as well as their exploitation rights, including all the considerations about data protection and sharing.
Serious Games Versus ROI
The availability of detailed user interaction data allows for a better understanding of how the game is used in a company and is critical to understand the return of investment (ROI) obtained in observable terms, such as number of workers playing the game or concepts acquired. The indicators can be directly integrated into the reporting dashboards provided by the learning analytics system. This profitability analysis could also help to create a new atmosphere in corporate training to support the development of new games and the increase of the budget allocated for those games.
Integration of Games in Bigger Systems
Until now, games are mostly used as independent pieces of content that, in the best of the cases, are integrated or connected with e-learning systems (e.g., LMSs such as Moodle). With a learning analytics module, the information from the game can be collected and integrated in a richer way with new systems. For instance, this information could be integrated with the system proposed in the LRNG project (https://www.lrng.org/), a “21st century ecosystem of learning that combines in-school, out-of-school, work-based, and online learning opportunities that are visible and accessible to all.” Moreover the generalization of new standards such as xAPI or IMS Caliper could greatly contribute to this effort (Advanced Distributed Learning, 2015).
Learning How People Actually Learn with Games
The use of game learning analytics can lead to the availability of huge amounts of interaction data that can be mapped to increases in knowledge. Analysis of this data can help us to better understand how people learn or, at least, to identify what behaviors consistently make people learn or to gain new knowledge or skills. As more learning analytics data is collected, anonymized, and made accessible through standards (e.g., xAPI), the door will be open for third parties to search for such insights. Others will then be able to build on their results, avoiding costly mistakes and improving both game designs and analytics themselves.
Ethics and Risks
When collecting learning analytics data, compliance with laws and regulations on personal data privacy, data access and storage, data retraction, and data aggregation requires great care and attention to detail. These aspects can become even more complicated when deployments span more than one country with different laws or when using cloud storage where there is no clear information regarding the physical location of stored data. All those considerations become even more critical when dealing with certain domains where data contains truly sensitive information, such as the medical domain (in this case usually the LA application needs to be reviewed and approved in advance by an ethics committee) or underage students.
A number of risks should be addressed when planning a data protection scheme, including security, privacy, and anonymization. The first step is to adopt a clear ethics policy that covers all relevant aspects, including the use of informed-consent forms prior to collecting data from users, and which clearly specifies the purposes for which the data will be used. There can be no doubt on the ownership of collected data and who can use the data and under which circumstances. According to Prinsloo, Slade, Hall, and Hill (2013), the ideal scenario would be for all concerned stakeholders in the educational process (i.e., student, teachers, manager) to benefit from learning analytics data.
The generalization of learning analytics requires, at least, the enforcement of data access control and protection. Besides, in order to minimize risks, data should not be personally identifiable, either because the information is not collected or because it is anonymized immediately after collection, using strong cryptography to provide a zero-knowledge system (where even the operators cannot decipher protected data). For example, in Fig. 3 section “(Technical Challenges”), the small key in instructor roles indicates that only they would be able to access protected information and only for those classes that they are responsible for.
New game learning analytics infrastructure should include all these features by default, both because this avoids the significant costs of attempting to secure the system after the fact and because such post hoc efforts are very prone to oversights that render them unsuccessful at actual security: security must be present in all design decisions in order to be effective.
Ethics in learning analytics is a much broader and complex topic, with no widely accepted guidelines or codes of practice. An essential benchmark on the topic is Sclater (2014), where a very comprehensive literature review of the ethical and legal issues in related fields is presented together with some of the answers that other fields have come up with.
There is a whole new range of opportunities of the application of learning analytics to the serious games. Game learning analytics (GLA) will help to gain insight about how players are actually playing and will inform a new generation of increasingly effective educational games.
The serious games business is beginning to grow and is expected to mature as a market in the next years, at least in specific domains (e.g., medicine, military). Mobile device adoption (i.e., smartphones, tablets) and emergent devices (such as wearables) will help this growth, by boosting the spread of serious games in both formal and vocational education. It is well known that the game industry has been collecting user data for many years, trying to increase player retention and, hence, profit. However, since the game industry, unlike academia, is not committed to sharing their tools and platforms, and due to other reasons (e.g., limited budget), serious games have not yet taken full advantage of analytics.
Multiple projects and initiatives to simplify the application of GLA are currently underway. On one side, there are projects that promote the technology and use of GLA (e.g., the RAGE project), increasing the availability of information and employing open-source code to ease reusability. On the other side, the educational technology field is developing new standards for the description, storage, and query of game analytics. These standards will simplify not only data collection but also interoperability and the inclusion of games in other preexisting learning infrastructures.
Finally, GLA is not only driving the effective creation of better educational games, but it is also helping to generate a new range of business opportunities. Until now, educational games were considered almost works of art, where predicting its real effectiveness or educational dimension was a question of educated guesswork. Game learning analytics techniques will bring data-driven approach closer and, therefore, greater accountability and support for new and unforeseen business opportunities.
Conclusions and Future Work
This chapter has described the main concepts, opportunities, and challenges for the nascent field of game learning analytics (GLA), a vital ingredient to bring serious games into mainstream educational practice.
As serious games researchers, educators, occasional serious game developers, and authors of a GLA platform, the chapter’s authors are deeply committed to this goal. As members of the EU H2020 RAGE Project, started in early 2015, they are currently developing a free, open-source, fully fledged GLA infrastructure that follows the general architecture of Figs. 2 and 3 and that can be deployed by serious game developers and institutions to analyze and learn from their games and players.
This work has been partially funded by Regional Government of Madrid (eMadrid S2013/ICE-2715), by the Complutense University of Madrid (GR3/14-921340), by the Ministry of Education (TIN2013-46149-C2-1-R and FPU12/04310), by the RIURE Network (CYTED 513RT0471), and by the European Commission (RAGE H2020-ICT-2014-1-644187, BEACONING H2020-ICT-2015-687676).
- Advanced Distributed Learning, U. S. D. O. D. (2014). Training and learning architecture (TLA): Experience API (xAPI) https://github.com/adlnet/xAPI-Spec/blob/master/xAPI.md. Accessed 10 May 2016.
- Advanced Distributed Learning, U. S. D. O. D. (2015). SCORM to TLA roadmap http://adlnet.github.io/SCORM-to-TLA-Roadmap/. Accessed 10 May 2016.
- Aldrich, C. (2004). Simulations and the future of learning: An innovative (and perhaps revolutionary) approach to e-learning. San Francisco, CA: Pfeiffer.Google Scholar
- Arnold, K. E., & Pistilli, M. D. (2012). Course signals at purdue: Using learning analytics to increase student success. In Proceedings of the 2nd international conference on learning analytics & knowledge. New York, NY: ACM.Google Scholar
- Baker, R. S., & Inventado, P. S. (2014). Educational data mining and learning analytics. In Learning analytics (pp. 61–75). New York, NY: Springer.Google Scholar
- Barfield, W., & Caudell, T. (2001). Fundamentals of wearable computers and augmented reality. Mahwah, NJ: CRC Press.Google Scholar
- Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of serious games: An overview. Advances in Human-Computer Interaction, 2013, 1–11. doi:10.1155/2013/136864 (Article ID 136864).Google Scholar
- Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Draft paper submitted to the Office of Educational Technology, U.S. Department of Education.Google Scholar
- del Blanco, Á., Marchiori, E. J., & Fernández-Manjón, B. (2010). Adventure games and language learning. In First international workshop on technological innovation for specialized linguistic domains: Theoretical and methodological perspectives (TISLID 10) (pp. 1–9).Google Scholar
- Elias, T. (2011). Learning analytics: Definitions processes and potential. http://learninganalytics.net/syllabus.html.
- Entertainment Software Association, (ESA). (2015). 2015 essential facts about the computer and video game industry http://www.theesa.com/article/2015-essential-facts-report-finds-nearly-half-u-s-plays-video-games/.
- Freire, M., del Blanco, Á., & Fernández-Manjón, B. (2014). Serious games as edX MOOC activities. In Proceedings of the 2014 I.E. Global Engineering Education Conference (EDUCON) (pp. 867–871).Google Scholar
- Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Educational Technology & Society, 15(3), 42–57.Google Scholar
- Hainey, T., Westera, W., Baxter, G., Connolly, T. M., Beeby, R. B., & Soflano, M. (2013). Students’ attitudes toward playing games and using games in education: Comparing Scotland and the Netherlands. Computers & Education, 69, 474–484.Google Scholar
- Hauge, J. B., Berta, R., Fiucci, G., Manjon, B. F., Padron-Napoles, C., Westra, W., & Nadolski, R. (2014). Implications of learning analytics for serious game design. In 2014 I.E. 14th International Conference on Advanced Learning Technologies (pp. 230–232). IEEE. doi:10.1109/ICALT.2014.73Google Scholar
- IMS Global Consortium. (2015a). IMS calliper analytics.Google Scholar
- IMS Global Consortium. (2015b). IMS global learning tools interoperability implementation guide v1.2 final. https://www.imsglobal.org/activity/caliperram.
- ISFE. (2014). GameTrack European digest: quarter 3 2014. http://www.isfe.eu/sites/isfe.eu/files/attachments/gametrack_european_digest_q3-14.pdf. Accessed 10 May 2016.
- Koster, R. (2004). Theory of fun for game design. Scottsdale, AZ: Paraglyph.Google Scholar
- Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In Serious games analytics (pp. 3–29). Cham, Germany: Springer.Google Scholar
- Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, 46(5), 31–40.Google Scholar
- Manero, B., Torrente, J., Freire, M., & Fernández-Manjón, B. (2016). An instrument to build a gamer clustering framework according to gaming preferences and habits. Computers in Human Behavior (accepted). doi:10.1016/j.chb.2016.03.085.Google Scholar
- Michael, D., & Chen, S. (2006). Serious games: Games that educate, train, and inform. Boston, MA: Thomson.Google Scholar
- Owen, V. E., Ramirez, D., Salmon, A., & Halverson, R. (2014). Capturing learner trajectories in educational games through ADAGE (assessment data aggregator for game environments): A click-stream data framework for assessment of learning in play. American Educational Research Association Annual Meeting, pp. 1–7.Google Scholar
- Papastergiou, M., & Solomonidou, C. (2005). Gender issues in Internet access and favourite Internet activities among Greek high school pupils inside and outside school. Computers & Education, 44(4), 377–393. doi:10.1016/j.compedu.2004.04.002. (http://adageapi.org/publications.html).
- Polsani, P. (2003). Use and abuse of reusable learning objects. Journal of Digital Information, 3(4).Google Scholar
- Prinsloo, P., Slade, S., Hall, F., & Hill, B. (2013). An evaluation of policy frameworks for addressing ethical considerations in learning analytics. In International conference on learning analytics and knowledge (pp. 240–244). doi:10.1145/2460296.2460344.Google Scholar
- Saveski, G. L., Westera, W., Yuan, L., Hollins, P., Fernández-Manjón, B., Moreno-Ger, P., & Stefanov, K. (2015). What serious game studios want from ICT research: Identifying developers’ needs. In GaLA Conference 2015. Rome.Google Scholar
- Sclater, N. (2014). Code of practice for learning analytics. JISC. Available online at https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics.
- Seif El-Nasr, M., Drachen, A., & Canossa, A. (Eds.). (2013). Game analytics – maximizing the value of player data. London, UK: Springer.Google Scholar
- Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. In Proceedings 1996 I.E. Symposium on Visual Languages (pp. 336–343). IEEE Computer Society Press. doi:10.1109/VL.1996.545307. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=545307
- Shute, V. J. (2011). Stealth assessment in computer-based games to support learning. Computer Games and Instruction, 55(2), 503–524.Google Scholar
- Shute, V. J., & Spector, J. M. (2008). SCORM 2.0 white paper: Stealth assessment in virtual worlds. Available online at http://scorm.com/tincanoverview/the-letsi-scorm-2-0-white-papers/.
- Shute, V. J., Ventura, M., Bauer, M., & Zapata-Rivera, D. (2009). Melding the power of serious games and embedded assessment to monitor and foster learning. In Serious games: Mechanisms and effects (pp. 295–321). New York, NY: Routledge.Google Scholar
- Siemens, G., Dawson, S., & Lynch, G. (2013). Improving the quality and productivity of the higher education sector. Policy and strategy for systems-level deployment of learning analytics. Canberra, Australia: Office of Learning and Teaching, Australian Government.Google Scholar
- Takeuchi, L. M., & Vaala, S. (2014). Level up learning: A national survey on teaching with digital games. New York, NY: The Joan Ganz Cooney Center at Sesame Workshop.Google Scholar
- Torrance, H. (2007). Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post-secondary education and training can come to dominate learning. Assessment in Education: Principles, Policy & Practice, 14(3), 281–294. doi:10.1080/09695940701591867.CrossRefGoogle Scholar
- Torrente, J., del Blanco, Á., Serrano-Laguna, Á., Vallejo-Pinto, J., Moreno-Ger, P., & Fernández-Manjón, B. (2014). Towards a low cost adaptation of educational games for people with disabilities. Computer Science and Information Systems, 11(1), 369–391. doi:10.2298/CSIS121209013T.CrossRefGoogle Scholar
- Torrente, J., Moreno-Ger, P., Martínez-Ortiz, I., & Fernández-Manjón, B. (2009). Integration and deployment of educational games in e-learning environments: The learning object model meets educational gaming. Educational Technology & Society, 12(4), 359–371.Google Scholar
- Van Eck, R. (2006). Digital game-based learning: It’s not just the digital natives who are restless. EDUCAUSE Review, 41(2), 16–30.Google Scholar