Advertisement

Multimedia Tools and Applications

, Volume 77, Issue 16, pp 21245–21263 | Cite as

Multisensory games-based learning - lessons learnt from olfactory enhancement of a digital board game

  • Alexandra Covaci
  • Gheorghita Ghinea
  • Chang-Hsin Lin
  • Shu-Hsien Huang
  • Ju-Ling Shih
Open Access
Article

Abstract

Serious games are becoming an alternative educational method in a variety of fields because of their potential to improve the quality of learning experiences and to facilitate knowledge acquisition and content understanding. Moreover, entertainment-driven learners are more easily motivated to benefit from the learning process through meaningful activities defined in a game context. Interfacing educational computer games with multisensorial interfaces allows for a seamless integration between virtual and physical environments. Multisensorial cues can improve memory and attention and increase the cognitive and sensory-motor performance. Despite of the increasing knowledge in sensory processes, multisensory experiences and interactions in computer based instruction remain insufficiently explored and understood. In this paper, we present a multisensory educational game - Fragrance Channel - and we investigate how enabling olfaction can contribute to users' learning performance, engagement and quality of experience. We compare results obtained after experiencing Fragrance Channel in the presence and absence of olfactory feedback on both a mobile and a PC. A knowledge test administered before and immediately after showed that our proposed educational game led to an improvement of performance in all the explored conditions. Subjective measurements carried out after the olfactory experience showed that students enjoyed the scenario and appreciated it as being relevant.

Keywords

Games Olfaction Learning Multisensory interaction History 

1 Introduction

Video games are a mainstream form of entertainment with a continuously increasing amount of time spent by children and adolescents playing them. As games become more complex and open-ended, they seem to offer a formative experience that allows players to express themselves while developing their own identity and knowledge. Digital technologies have a significant impact on users, challenging them into new perceptual and cognitive actions. Given their audience and popularity, it is natural to consider them an interesting alternative in which educational content can be delivered. Undoubtedly, the recent rapid developments in virtual and augmented reality will influence a pedagogical shift. Platforms like AltspaceVR1 bring new possibilities for transformative experiences through a wide spectrum of interaction resources. This will also influence the uptake of serious games that are already used as a tool for teachers to supplement traditional training methods in various domains, such as science and healthcare training [35]. However, in the actual context, serious games are able to address professional development with promising benefits for instructors, therapists or people with disabilities. The purpose of serious games is twofold: (i) to entertain, and (ii) to have an educational impact [5]. One approach to address this purpose is to make use of the whole spectrum of sensory experiences in creating meaningful interactions. According to [36], users’ interactions with technology are multisensorial. By adding a multisensory dimension to games, we could thus enrich the user experience and perception [20] while making them more accessible for people with disabilities. Moreover, employing additional sensory cues can also benefit the learning process. Multisensory learning argues that learning is reinforced by stimulating multiple sensory channels. This theory was initiated by Gardner in [21] and reinforced in [48], where the authors discuss the advantages of training and education that engages multiple senses over unisensory paradigms. The main argument is that the human brain is used to processing stimuli in different modalities at the same time and this multimodal information facilitates acting in the world. Multisensory information representation in a computer game facilitates schema construction by offering the learner an explicit representation of a concept [17].

Although multisensory technology has started to be embedded into low cost peripheral devices and wearables, the senses we call upon when interacting with computer based instruction (games, simulations and virtual worlds) are still restricted. Current implementations are focused on audition and vision, increasingly harnessing touch, whilst chemical senses (gustation and olfaction) remain underexploited [24]. Out of the insufficiently explored senses, stimulating the olfactory channel can be a powerful learning tool because it is linked to the limbic system, the part of the brain that processes feelings and emotions. Indeed, studies have shown how multisensory learning experiences that involve the sense of smell can lead to high levels of Quality of Experience (QoE) for the user [1, 2, 23]. Thus, olfactory feedback could be a particularly promising aspect in designing educational games. To achieve this, it is crucial to understand the properties of this sensory modality and to determine a meaningful design space for multisensory interaction. For this, we need to explore design methods that provide both qualitative and quantitative parameters.

In this paper, we propose an olfaction enhanced multi-player game for educating students about the seventeenth century Age of Discovery. This study is an exploration on ways to design multisensory experiences in educational games, having as aim the evaluation of our proposed interaction scenario. We compare (on both PC and mobile) the traditional unisensory learning approach with one that presents multisensory cues through the stimulation of the olfactory channel though congruent smells. Our focus is on immediate information acquisition and recall and on users’ QoE and satisfaction. To measure the knowledge acquisition, the participants in our study answered a knowledge test administered before and immediately after our proposed game. Engagement and enjoyment of the proposed concept were evaluated through subjective measurements - a satisfaction questionnaire. The paper is organised as follows. Section 2 presents the use of multisensory systems in building educational games and discusses issues related to design, information recall and users’ engagement. Section 3 introduces our proposed game. The user evaluation is carried out in Section 4, while Section 5 reports and discusses the results. Finally, future work is outlined in Section 6.

2 Related work and motivation

2.1 Educational games

Technology imposes an emerging reorganisation of workplace practices. The learning process is an important candidate to benefit from adopting these changes dictated by technological innovation. Education needs to become more modular and flexible and ready to move out of the classroom into informal settings [46]. The use of computer based instruction is starting to afford new options for both human creativity and social interaction in a variety of domains and it has a significant potential for transforming education. Effective learning is described by modern theories as active, experiential, situated, problem-based and providing immediate feedback [6]. Serious games can be considered the consequence of balance between learning and gaming elements [43]. Activities offered by these games are not only motivational, but also consistent with the effective learning theories. In serious games learning becomes an active process where real world situations can be posed allowing novices to develop content knowledge and problem-solving skills [6].

Serious games are designed to have an impact on the audience that goes beyond the pure entertainment aspect. In [7], the authors present a comprehensive review focusing on 142 papers (published between 2009 and 2014) that provided high quality evidence about the positive impacts and outcomes of the games, using the framework proposed in [13]. This framework for organising and understanding games considered aspects like: the main purpose of the game, the game genre, subject discipline, learning and behavioural outcomes. Knowledge acquisition was the most frequent outcome reported by serious games studies, while entertainment games considered also perceptual and psychological outcomes. Science, Technology, Engineering and Maths (STEM) subjects were the most popular across the topics, followed by health related games. The games used in STEM [8], business and economics [27], were aimed at knowledge transfer, while the ones used in health [44] or history, were focused on behavioural change [34] (e.g., Wi fit). Games were also used in extra-curricular learning contexts such as substance abuse [54] or aviation safety education [11].

In the STEM area, most of the serious games target science. OperationARIES! was developed as an intelligent tutoring system meant to help students acquire 21 core concepts of research methodology (e.g., correlation vs. causation) [19]. The storyline presents an invasion of the world where learners need to fight against bad scientists across three modules: Training, Case Study and Interrogation. Results showed that different core concepts might be assigned to different modules or training times. In natural science, a concept map-embedded game was proposed in [29] addressing the growth of butterflies topic. Players have to collect data, to fight and to solve puzzles to complete the gaming missions. The game showed significant results in improving student’s achievements and decreasing their cognitive load. In chemistry, educational games can fit abstract concepts like compound structure or chemical reactions. In [4], the authors propose a board like serious game for a general chemistry course and state that it might have contributed to better performance. Mechanical engineering classes have also been re-designed around a video game where learners had to race a simulated car around a track [12]. Students invested more time in learning the material and viewed the content as more valuable. Moreover, the game environment allowed them to deal with big, interdependent and authentic problems. Mathematics content was also integrated in a game based environment developed on e-books with activities designed for understanding equivalence or symmetry [28]. Results showed also in this case better learning achievements and increased attention in the digital game setup than in conventional approaches. In the discipline of History, students gained knowledge after experiencing commercial games (e.g., Age of empires) [47]. These games determined also a rise in enthusiasm for visiting museums or other historical places. The history of Czechoslovakia was the subject of one of the few serious games in this discipline [49, 50] that generated a positive acceptance in both students and teachers. 1910 is a serious game created for mobile devices to motivate and engage 6th grade students in learning Portuguese history [14] However, there is still no user study to assess the impact of this game on learning or entertainment. These studies showed that serious games are a convenient place with efficient outcomes that could be used in a variety of domains.

2.2 Unisensory versus multisensory exposure in the learning process

Studies on learning are often focused on unisensory experiences, although our interaction with our surrounding environment is multisensorial. Providing an individual with multiple sensory cues should facilitate a representation, thus numerous educational programmes have advocated the benefits of mutually supportive multisensory information. Indeed, psychologists and neurologists have advanced the field of multisensory perception over recent decades. Research has shown that the facilitative effect in information processing is significantly greater for multisensory stimulus combinations than within-modal combinations [25]. Multisensory processes facilitate memory [42], perceptual and implicit learning and training outcomes [3]. The advantage of the multisensory information extends also to paired associative learning in adults [18]. In [26], the authors have shown that multisensory audio/tactile cueing improves the performance (speed and accuracy) of visual search and reduces the amount of mental workload. Promising results were obtained also for children in incidental category learning [9] or numerical learning tasks [31]. Multisensory intervention has been successfully used also as a therapy for correcting and improving literacy skills [45] or for increasing sustained and selective attention of people with autism [16]. Moreover, in this case, the multisensory experience was even more effective by increasing students’ engagement with people and objects. Technology advances now provide the opportunity to combine and connect neuroscience knowledge with game design thinking, creating toolkits for learning, training, therapy. Thus, new design methods and frameworks need to be explored, that allow researchers and designers to harness all the available senses.

2.3 Olfaction based multisensory educational games

Currently, most interactions with digital information are characterised by a poor usage of sensorimotor capabilities [37]. Although the transition from desktop to mobile interfaces improved some aspects of user experience, it still relies mostly on vision, hearing and basic tactile feedback whilst other senses are not explored sufficiently. Current rapid prototyping platforms like Arduino2 that enable the rapid connection of low cost devices (e.g., a fan or a smell interface) stimulate multisensory research in a HCI context. However, identifying the appropriate multisensory cues for specific situations is still a trial and error process, many issues related to the design and measure of these cues remaining uncovered [33].

Multisensory cues have the potential to increase performance, user engagement, to intensify the role of presence and to trigger specific emotional responses in educational games. In [10], authors highlighted the advantages of olfactory feedback in serious gaming: reinforcement, real-life authentic learning, retention of information, improvement of concentration levels. Indeed, the fact that odours enrich the user experience was reinforced in several studies. Performing a task did not influence the user-perceived experience or enjoyment in an olfaction enhanced multimedia application [2]. The presence of a congruent smell increased the pleasantness of music in experiments presented in [56], this finding being important in establishing design guidelines for emotional feedback in serious games. The connection between smell and memory has also been investigated with results that benefit educational games. In [53], scent presentation had a positive effect on users’ ability to recall the details of a virtual environment. However, information assimilation was not influenced by odours in a different setup, presented in [2]. This indicates that finding an appropriate scenario that can benefit from the addition of multisensory cues is still a challenge that needs to be explored.

To our knowledge, there is no serious game using olfactory cues in addition to the audiovisual ones. Although, as we shown in this section, there are studies that address the impact of olfaction on key elements of both entertainment and learning performance, educational games were not paired with odours with the exception of a chemistry educational environment [52]. However in this study, there is no information about testing the proposed system with users. As a summary of this section, the effectiveness of a history themed serious game that engages olfaction is a topic that needs to be studied. Moreover, its study should include measures of knowledge acquisition and enjoyment. The current paper deals with these aspects by proposing a history game that engages the olfactory sense and by evaluating it in terms of knowledge acquisition and users’ satisfaction.

3 Fragrance Channel

The educational game we created allows players to have an alternative experience of a lecture on Age of Discovery, the historical period when Europeans started to explore the world by sea in search of trading partners, trading routes and goods. Our hypothesis was that our interactive multisensorial solution will sustain students’ interest and attention, while allowing them to broaden their knowledge in history, trading conditions and origins of different spices used for commerce in the Eastern World.

In particular, the scenario we chose for this experiment is centred around the trade between civilisations in Asia, Northeast Africa and Europe of spices like cinnamon, cassia, cardamom, etc., one of the world’s biggest industry. In Fragrance Channel, players impersonate European explorers, who navigate, conquer and trade spices while understanding the importance of this period in the discovery of new continents, tipping the balance of the world power and sparking the modern age. The learning activities were chosen to relate back to the educational objective - gain knowledge about the Age of Discovery. Additionally, we wanted to achieve other group of objectives: higher-cognitive (through the user’s involvement), affective or interpersonal.

In designing the interface, we have been inspired by board games because of their enduring appeal that transcends cultural and language barriers [15]. Moreover, board games are appropriate to illustrate the dynamics of historical events since they make players: (i) think about situations and scenarios that are not in the normal course of events, (ii) think about strategies about competition, (iii) think about obstacles [32]. Fragrance Chanel combines chance and skill (choosing the right strategy) to provide a competitive, but not intimidating environment.

3.1 Method

This section describes the two stages of Fragrance Channel and the events in the narrative of the game. Our proposed game starts with a configuration and customisation stage, where players choose the countries they want to represent and build the navy system for trading spices. Each country has specific characteristics matching the historical context: number of colonies, army strength, naval technology. The profile of a country is given by values assigned to these characteristics (e.g., Portugal, one of the primary players in the Age of Discovery has 13 colonies; Spain, who rode the second wave of expansion focused more on conquest than on trading has 11 colonies and 10 additional points for army strength, etc.). Each boat has also some key components (hull, oar, sails, and weapon - noted between 0 and 100 points) used to define the navy profile based on a set of computations that consider factors like: propulsion power, cargo capacity, deceleration, firing distance, damage, sailing force (see Fig. 1). Based on their country and navy profile, users can formulate a strategy that is coherent with their configurations. In this stage, players randomly choose task cards that contain information about three spices they need to retrieve. These spices have different characteristics related to the country that can retrieve them or to the colonies they could be collected from: one can be retrieved by any country, one can be collected only from the player’s colony, and one can be collected from the colonies of the other players. This leads to a different behaviour in the collection strategy: players who have the targeted spice in their colony, can simply sail to any port and trade for it, players who do not have the spices in their colonies have to fight with other ships and obtain them.
Fig. 1

System architecture

In the second stage of the game, the players travel the spice trade routes, buying and selling goods from port to port. The game map is divided in 34 zones where players could sail with 13 ports, where they can refuel and exchange items. However, out of the 13 refuel zones, players could anchor their boats just in their colonies - ports that are tagged with their flag. Six of the zones on the map represented key regions for sourcing some of the most profitable goods, spices such as: cocoa bean, coffee bean, ginger, pepper, cinnamon, or clove. When entering this stage, players are informed through a riddle about the spice with the highest value during the round. Students need to solve the riddle by looking for the spice information and its zone of production. Trading the highest scored spice becomes then the goal of the game. Players take turns in advancing on the map by selecting one of the four possible actions: Sail, Attack, Go to port, End of turn. To decide the distance sailed in each round, users throw dices: the points on the dices are combined with matching boat features (propulsion power and sailing force). Once one of the players requests a battle, the attack phase starts and its winner gets the spice. If one player is attacked, the quantity of the carried spice is reduced, thus he/she might need to restart the trading in the collection area of the spice.

The difficulty of the task is given by the values of all the involved parameters. These values were set and calibrated by continuous game testing and monitoring meant to adjust them to provide the best levels of engagement and performance. For this calibration phase, we used postgraduate students that tested the game under different game settings. For each of these tests, we stored the participants’ score and progression through the tasks. After the completion of the game, users were asked to rate any difference in difficulty using a Likert scale: 1) a lot easier; 2) easier; 3) equal; 4) harder; 5) a lot harder. Based on these answers we adjusted the settings of the game so that they provide optimal difficulty.

The game ends after 30 min and the winner is the user who traded spices most efficiently. To make the game engaging and emotionally arousing we introduced different actions and we depicted the consequences of players’ errors. Actions like Sail, Attack and Go to port come with specific visual effects and textual information. Communication between players is also encouraged through an in-game chat room that makes exchanging ideas, strategies and knowledge easy. Figure 1 presents the main components of the system.

3.2 Implementation

Fragrance Channel was developed in Unity3D3 as a multiplayer game that connects four people in a single session over the Internet. For this purpose we have used the Photon Network4 that allowed us to create multiple game groups to keep all the players connected to the same room upon game initialisation. Subsequently, after she/he joins the room, the player’s object will be launched over the network. All the clients are connected to the database of the SQL Server where we store data needed for the game to run: Player information, Boat information and Game behaviour. Players have also the possibility to communicate through text messages via the Client server using a chat room. These text communications will be also documented on the SQL Server.

4 Experiments and results

4.1 Motivation

Our goal is to investigate the existence of possible differences in students’ performance while experiencing the game on mobile devices and desktop environments, in the presence or absence of olfactory stimulation. Thus, we designed four test conditions: MobileWithoutOlfaction (Fig. 2b), PCWithoutOlfaction, MobileWithOlfaction, PCWithOlfaction (Fig. 2a). When olfaction was present, players had to smell the corresponding target spice at the start of the round. In doing so, we wanted to observe if the olfactory conditions lead to an increase of performance or if the game experience will be more entertaining and enjoyable.
Fig. 2

Students playing Fragrance Channel

4.2 Materials

Following our motivation, we developed Fragrance Channel with both mobile and desktop configurations. Subjects practiced the game on desktop PCs (Intel Core2Duo@2.8 GHz) and on Android mobile devices. The olfactory feedback consisted of real spices stored in jars opened by the players who were experiencing the impact of odours. This choice prevented the lingering of scents between rounds and allowed students to control their time of interaction with the smells.

4.3 Participants

For the evaluation of Fragrance Channel, we considered a sample of 117 participants, junior high school students in the southern part of Taiwan. They were divided in four groups in order to test the proposed four conditions (n1 = 32, n2 = 29, n3 = 28, n4 = 28).

4.4 Measures

4.4.1 Knowledge

To measure participants’ knowledge, we prepared a test with 16 questions meant to assess the acquisition of information related to the historical, geographical and spice-related context. This test consisted of eight true-false questions (T1-T8), awarded with 5 points each, and eight multiple choice questions(T9-T16), awarded with 7.5 points. More specifically, the 16 questions were referring to the identification and localisation of the countries that played a significant role during the Age of Discovery, their colonies, the origin of the spices and the invention of gunpowder. As a general criterion, only answers that were correct and complete were considered in evaluating the performance of the players. Knowledge was measured as the number of correct answers multiplied by the score of each category of questions.

4.4.2 Self-reported engagement

As we pointed out earlier, one of the purposes of a serious game is to be entertaining, attractive and motivating to a broad target audience. In addition to the targeted assessment performed to obtain the objective measures of learning, other factors can also indirectly lead to the educational process. Therefore, in addition to the objective tests analysed in the Results Section, we also administered a questionnaire to provide a measure of engagement as a subjective perception of students towards the proposed multisensory learning methodologies. At the end of the game, players that experienced Fragrance Channel in the presence of olfactory feedback were asked to think about their experience and rate their level of agreement for a set of eight questions that were tailored to our experiment and focused on different aspects of our learning approach.
  1. (1)

    The multi-sensorial experience helped me to better understand the related concepts taught in the class.

     
  2. (2)

    The multi-sensorial experience helped me to better assimilate the concepts taught in the class.

     
  3. (3)

    The multi-sensorial experience did not improve my learning experience.

     
  4. (4)

    The multi-sensorial experience helped me to better understand the practical experience of the learning process.

     
  5. (5)

    I enjoyed the multi-sensorial experience during the class.

     
  6. (6)

    The multi-sensorial effects were disturbing for me during the class.

     
  7. (7)

    I would like to have more classes/labs/courses that include multi-sensorial experience.

     
  8. (8)

    Please award the game either a gold, silver or bronze medal.

     
  9. (9)

    Comments/Suggestions (optional)

     

Questions Q1 to Q7 were rated on a 5-point Likert scale: 1 (strongly disagree), 2 (disagree), 3 (neutral), 4 (agree), 5 (strongly agree). In order to check for possible inconsistencies in the answers, there are two control questions (Q3 and Q6) that are negative forms of the previous questions. Q8 answers are ranked as 1 (gold), 2 (silver), and 3 (bronze). As the last part of the questionnaire, respondents were asked (in an open question, Q9) to provide additional comments to their experience.

4.5 Procedure

Participants were told that the goal of the experiment was to evaluate a game. Consent for participation was asked. After users gave their consent to participate at the experiment, they answered the knowledge questions for the pre-test. Participants were randomly assigned to the four experimental conditions (Fig. 3 depicts two of the experimental testing conditions) After they finished the game, users answered again the knowledge questions for the post-test and filled in the satisfaction questionnaire.
Fig. 3

Self-reported engagement

4.6 Results

4.6.1 Knowledge

We administered the knowledge test two times: before trying Fragrance Channel (pre-test) and immediately after trying it (post-test). Mean pre-test scores showed that before interacting with our game, participants were able to score only around 8% (MD = 8.08, SD = 13.6). An independent samples t-test showed no significant differences in pre-test knowledge in the Age of Discovery topic between the four groups. Mean post-test scores showed that after playing Fragrance Channel the knowledge of the students improved: MD = 54.76, SD = 19.19 in the MobileWithoutOlfaction condition; MD = 57.84, SD = 16.37 in the PCWithoutOlfaction condition; MD = 53.13, SD = 19.7 in the MobileWithOlfaction condition; MD = 53.83, SD = 12.05 in the PCWithOlfaction condition. A paired sample t-test showed that the differences between post-test and pre-test results for all conditions were significant: t = 13.396 (p < .001) for the MobileWithoutOlfaction condition; t = 14.478 (p < .001) for the PCWithoutOlfaction condition; t = 10.332 (p < .001) for the MobileWithOlfaction condition; t = 14.241 (p < .001) for the PCWithOlfaction condition.

The improvement in performance was analysed in a two-way mixed factorial ANOVA with the device variable (PC or mobile) manipulated within different feedback conditions (with olfaction or without olfaction) as a between-subjects variable.

The output of the ANOVA table (see Table 1) shows also that there is no evidence of a significant interaction effect between type of device and feedback, F(1113) = 0.24, p = 0.63. The main effect for the device used to play Fragrance Channel yielded a F ratio of F (1113) =2.23, p = 0.14 indicating that it was not significant for the performance improvement. The test for the effect of olfactory feedback shows us that there is not enough evidence to conclude that there is a significant olfaction effect F(1113) = 1.38, p = 0.24.
Table 1

Effects of device and feedback on knowledge acquisition

 

Df

Sum Sq

Mean Sq

F value

Pr(>F)

Device

1

851.40

851.40

2.23

0.14

Feedback

1

529.34

529.34

1.38

0.24

Device: Feedback

1

91.10

91.10

0.24

0.63

Residuals

113

43,208.07

382.37

  

4.6.2 Self-reported engagement

Table 2 and Fig. 3 summarise the average scores granted by the students to each questions, as well as the confidence interval for a 95% confidence level (computed by means of the t-student test) and the sampling quasi-variance. There is a general agreement that multisensory experience enhances the learning (Q1, Q2, Q4, Q7). Control questions (Q3 and Q6) show a correct behaviour as they indicate a low score, corresponding to a positive perceived learning experience. Students offered a mild score to the experiment ranking (Q8).
Table 2

Satisfaction questionnaire answers

 

Q1

Q2

Q3

Q4

Q5

Q6

Q7

Q8

Score

3.80±0.16

3.82±0.15

1.61±0.16

3.76±0.15

3.85±0.15

1.47±0.18

3.89±0.15

1.60±0.11

Quasi - Var

0.81

0.76

0.82

0.72

0.78

1.05

0.75

0.42

t

6.66

7.02

3.23

6.73

7.27

3.90

7.70

p

P < 0.0001

P < 0.0001

P < 0.0001

P < 0.0001

P < 0.0001

P < 0.0001

P < 0.0001

This can be interpreted in agreement with the results shown in the previous section, where no objective positive reward has been achieved by the experiment. In other words, although students perceived that the multisensory educational game we proposed is interesting, motivating and with potential to improve and facilitate learning, there is also an underlying perception that the tool could be improved. This is also reflected in the questions score, where none of the questions achieved an average rank above 4. However, it is worthy to note that the responses to all questions were statistically significantly different from the neutral position, as detailed in Table 2.

When investigating the open answers and comments in the questionnaire, we found that players enjoyed the multisensory experience offered by Fragrance Channel: “It’s fun”, “I hope to play the game another time”, “It's a new and wonderful experience”, “I enjoyed the game”, “Great”, “The multisensory experience is very special” and that they want to repeat it also for other classes: “I want more courses like this”, “I hope to learn more”. Students suggested also the game could be improved through the addition of artificial intelligence context-sensitive behaviours or of game characters.

5 Discussion

In this paper, we proposed to expand history learning beyond the traditional methods by using a multi-sensory game. Our results showed that stimulating various senses in an educational game engages the users and has potential to improve the learning process. In the following sections, we will discuss the affordances and limitations for designing an educational multisensory game, before concluding with thoughts on how to improve users’ performances and engagement in similar setups.

5.1 Affordances and limitations

The results we obtained are encouraging and indicate the potential of an olfactory enhanced educational game when studying certain aspects of history. Considering the knowledge gain pre-post test, all the four conditions were effective and produced similar results. After using the Fragrance Chanel game, users improved their knowledge about the countries that played a significant role during the Age of Discovery, the origin of different spices, the geographical location and the components that influence the speed or the capacity of a boat. This information acquisition processed took place under all the different setups: on mobile or desktop, in the presence or absence of olfactory feedback. The lack of significance between the differences in performances on mobile and desktop shows that the type of interface does not influence students’ achievements. This is encouraging, since it indicates that similar educational games can be experienced on any device. It would be interesting to explore how performance would be affected after a traditional board game, where users use actual cards and build boats to navigate on the map.

In designing the olfactory feedback we chose the smell to be congruent with the other modalities involved in the task (e.g., smell of cinnamon was associated to the image of cinnamon). We opted for this implementation because congruence leads to stronger integration and more robust synchrony while enhancing the attentional control. In Fragrance Chanel this type of olfactory feedback did not trigger an improvement in performance. Possible interpretations for this could be related to participants’ ability to discriminate among the odour stimuli, the fact they did not verbalise the odour [30] or to the significance of the imagined scenario. Further exploration is needed to determine the type of information that can be linked to a certain smell, or to how olfactory feedback can be personalised and related to the students’ culture, emotions, or gaming events. Moreover, we aim to explore how the cognitive performance of the students is impacted by an incongruent olfactory design. This is motivated by interesting results that showed how memory was enhanced in the presence of the peppermint smell [41] as well as by our previous work [22]. Another interesting approach in designing the olfactory feedback would be to take into account crossmodal associations existent between modalities (e.g., deliver a citric smell corresponding to the hexagonal design of the map) [51].

When analysing the self-reported engagement and satisfaction we observe that students consider multisensory experiences as a valuable and enjoyable tool. Moreover, they are eager to repeat them and they suggest features for improvement. In conclusion, although the subjective evaluation shows enjoyment, objective measures do not yet offer an obvious improvement in student achievements under multisensory exposure. This is in line with other reported results in the multisensory arena [55, 57], which concluded that whilst users enjoyed such multisensory experiences and found that they heighten the sense of relevance, the presence of olfaction did not have a significant impact on the task performance of participants in an information recall task [2]. Moreover, the relatively low score obtained for Q6 is in agreement with previous work [23] which highlighted that users are tolerant to interference and distortion effects caused by olfactory effects in multimedia environments.

5.2 Possible extension

Multisensory systems proved to be effective in the learning process in different areas. STEM subjects were shown to benefit from the multisensory dimension that had a positive impact on the student’s learning experience [40]. In this study, users were exposed to an applied learning session on concepts about the Quality of Experience and demonstrated a better performance under the multisensory exposure. We envisage that using a multisensory game in learning STEM subjects could lead to even better performances because it can improve the understanding of different mechanism and concepts. For instance, mathematics and physics skill acquisition could be targeted in building the ship. This part of the game could be extended to encompass equations and physical principles like Archimedes principle. The olfactory feedback in this case will consists of aromas of rosemary of peppermint, shown to have a positive effect on cognition and mood [39].

6 Conclusion

Classic (digital) multimedia learning assumed that only audio-visual channels are available to potential learners [38]. Whilst this might have been the state of the art, digital technology has now evolved so that alternative forms of media and stimuli - the effect of which in the non-digital world is well studied – can now evolve to the digital mainstream. Towards this end, in this paper, we have presented the results of a study whose results reveal that, whilst the potential for multisensory effects to deliver a better educational experience is unmistakeable, we still have some way to go in order to understand how to integrate multisensory experiences - involving new media types such as olfaction - in games, in order to be able to achieve better cognoscible targets.

We are planning to run experiments also with a future version of the game, a hybrid board game with the same storyline but offering different ways of interaction that allow for cooperation and for engaging different sensory cues. Students will receive olfactory feedback while building their boats out of 3D models. This approach builds on conclusions from [39] that training transfer is improved by pleasantness of the odour displayed while assembling.

Footnotes

Notes

Acknowledgements

The work reported in this paper has received funding from the European Union’s Horizon 2020 Research and Innovation programme under Grant Agreement no. 688503 (NEWTON – Networked Labs for Training in Sciences and Technologies).

References

  1. 1.
    Ademoye OA, Ghinea G (2009) Synchronization of olfaction-enhanced multimedia. IEEE Trans Multimedia 11(3):561–565CrossRefGoogle Scholar
  2. 2.
    Ademoye OA, Ghinea G (2013) Information Recall Task Impact in Olfaction-enhanced Multimedia. ACM Trans Multimed Comput Commun Appl 9(3):17:1–17:16CrossRefGoogle Scholar
  3. 3.
    Alais D, Cass J (2010) Multisensory perceptual learning of temporal order: audiovisual learning transfers to vision but not audition. PLoS One 5(6):1–9CrossRefGoogle Scholar
  4. 4.
    Antunes M, Pacheco MAR, Giovanela M (2012) Design and implementation of an educational game for teaching chemistry in higher education. J Chem Educ 89(4):517–521CrossRefGoogle Scholar
  5. 5.
    Bellotti F, Kapralos B, Lee K, Moreno-Ger P, Berta R (2013) Assessment in and of serious games: an overview. Advances in Human-Computer Interaction, p 1.  https://doi.org/10.1155/2013/136864
  6. 6.
    Boyle E, Connolly TM, Hainey T (2011) The role of psychology in understanding the impact of computer games. Entertain Comput 2(2):69–74CrossRefGoogle Scholar
  7. 7.
    Boyle EA et al (2016) An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Comput Educ 94:178–192CrossRefGoogle Scholar
  8. 8.
    Braghirolli LF, Ribeiro JLD, Weise AD, Pizzolato M (2016) Benefits of educational games as an introductory activity in industrial engineering education. Comput Hum Behav 58:315–324CrossRefGoogle Scholar
  9. 9.
    Broadbent H, White H, Mareschal D, Kirkham N (2017) Incidental learning in a multisensory environment across childhood. Dev Sci.  https://doi.org/10.1111/desc.12554
  10. 10.
    Childers C, Coleman M (2010) The role olfactory technology in serious gaming, mental training, and therapy. https://pdfs.semanticscholar.org/ec84/13e464c80d7a10acafd4302bc3b46a67a1c0.pdf. Accessed 02 Jan 2018
  11. 11.
    Chittaro L, Buttussi F (2015) Assessing knowledge retention of an immersive serious game vs. a traditional education method in aviation safety. IEEE Trans Vis Comput Graph 21(4):529–538CrossRefGoogle Scholar
  12. 12.
    Coller BD, Scott MJ (2009) Effectiveness of using a video game to teach a course in mechanical engineering. Comput Educ 53(3):900–912CrossRefGoogle Scholar
  13. 13.
    Connolly TM, Boyle EA, MacArthur E, Hainey T, Boyle JM (2012) A systematic literature review of empirical evidence on computer games and serious games. Comput Educ 59(2):661–686CrossRefGoogle Scholar
  14. 14.
    Cruz S, Carvalho AAA, Araújo I (2016) A game for learning history on mobile devices. Educ Inf Technol:1–17Google Scholar
  15. 15.
    Egenfeldt-Nielsen S, Smith JH, Tosca SP (2015) Understanding video games: the essential introduction. Routledge, LondonGoogle Scholar
  16. 16.
    Escobedo L, Tentori M, Quintana E, Favela J, Garcia-Rosas D (2014) Using augmented reality to help children with autism stay focused. IEEE Pervasive Comput 13(1):38–46CrossRefGoogle Scholar
  17. 17.
    Ferdig RE (ed) (2008) Handbook of research on effective electronic gaming in education. IGI Global, HersheyGoogle Scholar
  18. 18.
    Fifer JM, Barutchu A, Shivdasani MN, Crewther SG (2013) Verbal and novel multisensory associative learning in adults. F1000Res 2:34Google Scholar
  19. 19.
    Forsyth C, Pavlik P Jr, Graesser AC, Cai Z, Germany ML, Millis K, Dolan RP, Butler H, Halpern D (2012) Learning gains for Core concepts in a serious game on scientific reasoning. In: Proceedings of the 5th international conference on educational data mining, pp 172–175Google Scholar
  20. 20.
    Franzwa C, Tang Y, Johnson A (2013) Serious game design: motivating students through a balance of fun and learning. In: 2013 5th International conference on games and virtual worlds for serious applications (VS-GAMES), pp 1–7Google Scholar
  21. 21.
    Gardner H (1983) Frames of mind. Basic Books, New YorkGoogle Scholar
  22. 22.
    G. Ghinea, and O.A. Ademoye," User perception of media content association in olfaction-enhanced multimedia." ACM Trans Multimed Comput Commun Appl (TOMM), 8(4), 52, 2012Google Scholar
  23. 23.
    Ghinea G, Ademoye O (2012) The sweet smell of success: enhancing multimedia applications with olfaction. ACM Trans Multimed Comput Commun Appl 8(1):2:1–2:17CrossRefGoogle Scholar
  24. 24.
    Ghinea G, Timmerer C, Lin W, Gulliver SR (2014) Mulsemedia: State of the art, perspectives, and challenges. ACM Trans Multimed Comput Commun App 11(1s):17Google Scholar
  25. 25.
    Gingras G, Rowland BA, Stein BE (2009) The differing impact of multisensory and unisensory integration on behavior. J Neurosci 29(15):4897–4902CrossRefGoogle Scholar
  26. 26.
    Hancock PA, Mercado JE, Merlo J, Erp JBFV (2013) Improving target detection in visual search through the augmenting multi-sensory cues. Ergonomics 56(5):729–738CrossRefGoogle Scholar
  27. 27.
    Huang WD, Johnson TE, Han SHC (2013) Impact of online instructional game features on college students’ perceived motivational support and cognitive investment: a structural equation modeling study. Internet High Educ 17:58–68Google Scholar
  28. 28.
    Hung C-M, Huang I, Hwang G-J (2014) Effects of digital game-based learning on students’ self-efficacy, motivation, anxiety, and achievements in learning mathematics. J Comput Educ 1(2–3):151–166CrossRefGoogle Scholar
  29. 29.
    Hwang G-J, Yang L-H, Wang S-Y (2013) A concept map-embedded educational computer game for improving students’ learning performance in natural science courses. Comput Educ 69:121–130CrossRefGoogle Scholar
  30. 30.
    Jönsson FU, Møller P, Olsson MJ (2011) Olfactory working memory: effects of verbalization on the 2-back task. Mem Cogn 39(6):1023–1032CrossRefGoogle Scholar
  31. 31.
    Jordan KE, Baker J (2011) Multisensory information boosts numerical matching abilities in young children. Dev Sci 14(2):205–213CrossRefGoogle Scholar
  32. 32.
    Kapp KM (2012) The gamification of learning and instruction: game-based methods and strategies for training and education. Wiley, New YorkGoogle Scholar
  33. 33.
    Kruijff E, Marquardt A, Trepkowski C, Schild J, Hinkenjann A (2017) Designed emotions: challenges and potential methodologies for improving multisensory cues to enhance user engagement in immersive systems. Vis Comput 33(4):471–488Google Scholar
  34. 34.
    Li KF, Sevcenco AM, Yan E (2013) Telerehabilitation using low-cost video game controllers. In: 2013 Seventh international conference on complex, intelligent, and software intensive systems, pp 136–143Google Scholar
  35. 35.
    Ma M, Oikonomou A, Jain LC (eds) (2011) Serious games and edutainment applications. Springer, LondonGoogle Scholar
  36. 36.
    Mangen A (2010) Point and click: theoretical and phenomenological reflections on the digitization early childhood education. Contemp Issues Early Child 11(4):415–431Google Scholar
  37. 37.
    Matassa A, Console L, Angelini L, Caon M, Khaled OA (2015) Workshop on full-body and multisensory experience in ubiquitous interaction. In: Adjunct proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2015 ACM international symposium on wearable computers, New York, NY, USA, pp 923–926Google Scholar
  38. 38.
    Mayer RE (2005) The Cambridge handbook of multimedia learning. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  39. 39.
    Moore AG, Herrera NS, Hurst TC, McMahan RP, Poeschl S (2015) The effects of olfaction on training transfer for an assembly task. In: 2015 I.E. virtual reality (VR), pp 237–238Google Scholar
  40. 40.
    Moss M, Cook J, Wesnes K, Duckett P (2003) Aromas of rosemary and lavender essential oils differentially affect cognition and mood in healthy adults. Int J Neurosci 113(1):15–38CrossRefGoogle Scholar
  41. 41.
    Moss M, Hewitt S, Moss L, Wesnes K (2008) Modulation of cognitive performance and mood by aromas of peppermint and Ylang-Ylang. Int J Neurosci 118(1):59–77CrossRefGoogle Scholar
  42. 42.
    Murray MM, Michel CM, de Peralta RG, Ortigue S, Brunet D, Andino SG, Schnider A (2004) Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging. NeuroImage 21(1):125–135CrossRefGoogle Scholar
  43. 43.
    Nussbaum M, Beserra VdS (2014) Educational Videogame Design. In: 2014 I.E. 14th International Conference on Advanced Learning Technologies, pp 2–3Google Scholar
  44. 44.
    Pichierri G, Murer K, de Bruin ED (2012) A cognitive-motor intervention using a dance video game to enhance foot placement accuracy and gait under dual task conditions in older adults: a randomized controlled trial. BMC Geriatr 12:74CrossRefGoogle Scholar
  45. 45.
    Pieretti RA, Kaul SD, Zarchy RM, O’Hanlon LM (2015) Using a multimodal approach to facilitate articulation, phonemic awareness, and literacy in young children. Commun Disord Q 36(3):131–141CrossRefGoogle Scholar
  46. 46.
    Psotka J (2013) Educational games and virtual reality as disruptive technologies. Educ Technol Soc 16(2):69–80Google Scholar
  47. 47.
    Şar E (2012) The role of history-themed non-educational computer games on primary school children’ (at grades 6th, 7th and 8th) perceptions of history. Procedia Soc Behav Sci 55:776–781Google Scholar
  48. 48.
    Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417CrossRefGoogle Scholar
  49. 49.
    Šisler V, Brom C, Cuhra J, Činátl K, Gemrot J (2012) Stories from the history of Czechoslovakia, a serious game for teaching history of the Czech lands in the 20th century – notes on design concepts and design process. In: Herrlich M, Malaka R, Masuch M (eds) Entertainment computing - ICEC 2012: 11th international conference, ICEC 2012, Bremen, Germany, September 26–29, 2012. Proceedings. Springer Berlin Heidelberg, Berlin, pp 67–74Google Scholar
  50. 50.
    Sisler V, Selmbacherova T, Pinkas J, Brom C (2014) Teaching contemporary history to high school students: the augmented learning environment of Czechoslovakia 38-89. Masaryk UJL Tech 8:99Google Scholar
  51. 51.
    Spence C (2011) Crossmodal correspondences: a tutorial review. Atten Percept Psychophys 73(4):971–95Google Scholar
  52. 52.
    Tijou A, Richard E, Richard P (2006) Using olfactive virtual environments for learning organic molecules. In: International Conference on Technologies for E-Learning and Digital Entertainment. Springer, Berlin, Heidelberg, pp 1223–1233Google Scholar
  53. 53.
    Tortell R, Luigi D, Dozois A, Bouchard S, Morie JF, Ilan D (2007) The effects of scent and game play experience on memory of a virtual environment. Virtual Real 11(1):61–68CrossRefGoogle Scholar
  54. 54.
    Verduin ML, LaRowe SD, Myrick H, Cannon-Bowers J, Bowers C (2013) Computer simulation games as an adjunct for treatment in male veterans with alcohol use disorder. J Subst Abus Treat 44(3):316–322CrossRefGoogle Scholar
  55. 55.
    Yuan Z, Chen S, Ghinea G, Muntean G-M (2014) User Quality of Experience of Mulsemedia Applications. ACM Trans Multimed Comput Commun Appl 11(1s):15:1–15:19CrossRefGoogle Scholar
  56. 56.
    Zhang L, Sun S, Xing B, Fu J, Yu S (2016) Exploring olfaction for enhancing multisensory and emotional game experience. In: El Rhalibi A, Tian F, Pan Z, Liu B (eds) E-learning and games: 10th international conference, edutainment 2016, Hangzhou, China, April 14–16, 2016, revised selected papers. Springer International Publishing, Cham, pp 111–121CrossRefGoogle Scholar
  57. 57.
    Zou L, Tal I, Covaci A, Ibarrola E, Ghinea G, Muntean G-M (2017) Can multisensorial media improve learner experience?. In: Proceedings of the 8th ACM on multimedia systems conference, New York, NY, USA, pp 315–320Google Scholar

Copyright information

© The Author(s) 2018

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Alexandra Covaci
    • 1
  • Gheorghita Ghinea
    • 1
  • Chang-Hsin Lin
    • 2
  • Shu-Hsien Huang
    • 2
  • Ju-Ling Shih
    • 2
  1. 1.Brunel UniversityLondonUK
  2. 2.National University of TainanTainanTaiwan

Personalised recommendations