The unceasing advances of technology as well as the pervasiveness of an increasing number of devices providing a variety of capabilities in intelligent environments are urging innovative and advanced interaction techniques. The aim is offering users ubiquitous, transparent, and natural interaction patterns in different settings, as an essential part of a new way of interpreting the relationship between humans and technology. In particular, a plethora of ubiquitous and pervasive devices and their embedded sensors provide new, and sometimes unforeseen, multimedia and multimodal support to build intelligent smart environments. The papers selected for the first part of this Special Issue outline some recent trends, in particular in the area of multimedia/multimodal applications. Some of these papers are the revised and extended versions of invited selected papers presented at the CHItaly 2017 conference, the biannual conference of the Italian Chapter of ACM Special Interest Group on Computer-Human Interaction (SIGCHI). They are representative of the work by researchers and practitioners in the Italian HCI community. Among the issues addressed in this first part, it is possible to mention the challenges associated with hand gesture recognition, and those raised by augmented interaction. Moreover, strategies for effective communications/conversations have also been analyzed from two perspectives: how to help people learn culturally-based communication protocols, and how to teach children managing group conversations through effective turn-taking. Another area that has been covered is related to the needs of people with cognitive disabilities, focusing on how to enhance the accessibility of web pages for people affected by dyslexia, and how to better support cognitive training of people with Alzheimer Disease.
The second part of this Special Issue is devoted to an exciting research topic that so far has not received the interest it deserves. Video games design is an intriguing and quite complex field, since it melts together results and findings deriving from many, widely different, disciplines, e.g., computer science, physics, music, visual art, industrial design, project management, marketing, mathematics, literature, anthropology, social sciences, medicine, storytelling, psychology and economy. A “melting pot” that emerges as especially suited to deal with new, emerging technologies, and which challenges both the academy and the industry: virtual and augmented reality, tangible interfaces, sensors, biometry, etc. Actually, these innovations can be exploited to figure out new game mechanics, and can influence all those aspects related to the interaction paradigms with the users and the related user experience. The latter holds both for games produced for pure entertainment, and for those which, instead, aim at addressing education, teaching or even more delicate fields such as rehabilitation. In many cases, the lack of a solid theoretical ground hinders the production of key guidelines and good practices directly deployable in the companies. Hence, the authors of the papers presented in the second part of this Special Issue have been invited to contribute in the advancement of the video game design field, by submitting works focusing on the complete spectrum of the computer science areas affecting the interaction patterns in the dyad human being – video game. In particular, a subset of the papers presented in the “game section” of this Special Issue are the revised and extended versions of selected papers presented at the 1st Workshop on Games-Human Interaction (GHItaly 17), held in conjunction with CHItaly 2017. The aim of the workshop has been both to bring together scholars belonging to very different disciplinary areas and establish a common ground on the topic of video game design, and to propose a meeting venue for researchers in a field still too underestimated, especially in Italy.
The six papers in the first section of the Special Issue are shortly presented in the following.
Hand gesture is still one of the most efficient and natural ways for human communication and, as a result, hand gesture recognition is becoming increasingly important. The article “Hand Gesture Recognition using Topological Features”  presents a topological approach to address the hand gesture recognition problem, by defining a hand gesture descriptor based on a Growing Neural Gas graph. The proposed algorithm uses the boundary of such a graph as a coarse approximation of the boundary of the object, and combines the information captured from the boundary with information of the region of the hand. The approach is able to extract the features that best capture the main geometrical/topological properties of the graph. The proposed subject-independent algorithm is evaluated on NTU Hand Digits dataset and compared against state-of-the-art methods: the achieved results are comparable with methods that use depth and color. The authors also prepared a comprehensive dataset (SBU-1) for different hand gestures containing 2170 images and including many possible deformations and variations, as well as some articulations which currently are not captured by existing datasets. The proposed algorithm is robust to scale, rotation and noise, while preserving similar recognition rate in comparison with the state-of-the-art results.
The new devices and “things” increasingly used in ambient intelligent environments, allow the combination of digital and physical worlds – potentially retaining the benefits of both. This has spurred the work of several researchers. In particular, the paper “Augmenting natural interaction with physical paper in Ambient Intelligence Environments”  builds on the idea of bridging the use of physical paper with state-of-the-art technologies that characterize Ambient Intelligence environments. It presents InPrinted, a framework supporting physical paper augmentation and user interaction in Ambient Intelligence environments. InPrinted provides context-aware and anticipation mechanisms, as well as tools and interaction techniques that support the development of applications incorporating printed matter augmentation. As a proof of concept, four applications have been implemented using the framework, each one supporting different interaction techniques and augmentation types, and addressing different activities. These applications have been assessed following a multi-step approach, involving heuristic evaluation, user testing, comparative evaluation and large-scale in-situ evaluation. The evaluation highlights the naturalness of touch-based interaction, and also suggest that users can easily expand their knowledge from digital interactions to interactions with augmented paper, achieving a positive User eXperience (UX), while the potential benefits of this technology are well accepted by users of all ages.
The paper “Learning communication from first- and third-person POVs: How perceptual differences influence the interpretation of conversations whilst waiting”  analyzes the importance of mitigating communication issues that can occur due to, e.g., different cultures, backgrounds, conversational styles and experience of participants. The authors propose a communication learning assistance system that can help people grasp culturally-based communications by focusing on cultural non-verbal behaviour related to interaction and communication. In particular, the authors study how participants learn and understand different behavioral patterns during interactions, by using experiments on the differences that exist on perceived communications in simulated crowds, both in first- and third-person points of view. In the study, participants interact with autonomous agents and experimenters via avatars in a shared virtual space, to obtain multiple tickets from two service counters. The gathered results demonstrate that the participants in the first-person group tend to focus on the interactions and their feelings while waiting for being served, whereas participants in the third-person group tend to see events and sequences more clearly because they are outside the conversation, thereby emphasizing fairness in waiting.
Conversations are typically structured for turn taking, whose practice is also pursued since primary schools for managing group conversations. However, turn taking is a rather abstract concept for children: concrete “operational” options have to be investigated to help primary school classes grasp the sense and rules of it. Among them, one possibility is using physical objects (tokens) to convey turn taking, in line with past research that indicates that it can positively affect children conversation behaviours. The paper “Turn Taking with Turn-Talk in Group: Actions and Reflections with Children and Teachers”  shows how a meta-design approach, based on action research, can help design tangibles for conveying turn taking in groups. The paper focuses on several actions with TurnTalk in a primary school classroom with 9-10 years old children, and it shows how actions led to benefits for the class conversational dynamics. In particular, the play-card mechanism of Turn-Talk helps in scaffolding an orderly conversation. The provided feedback makes children jointly reflect on their conversation, with visual metaphors adequate to them. In addition, the provided rules for rewards make everybody’s engagement in conversation perceived as a group responsibility. According to the available results, thanks to Turn-Talk, children engaged in conversation in an orderly manner, making an effort to include all their group members in the conversation, and replicating positive behaviours over time. Even when the technology “disappeared”, some positive effects were observed to remain. Finally, Turn-Talked helped teachers to be more objective and aware of what was happening in the classroom, highlighting who needed support or had positive behaviors to be reinforced.
The paper “Testing web-based solutions for improving reading tasks in dyslexic and neuro-typical users”  deals with the difficulties that students with dyslexia experience when they interact with web-based applications. Based on an extended survey of the accessibility guidelines issued by scholars and institutions, this paper aims to empirically evaluate if and to what extent it would be possible to further improve the accessibility/readability of web sites for students with dyslexia as well as neuro-typical readers. The aim of the paper is twofold: on the one hand, it wants to assess whether and how to improve the level of web page accessibility for users with dyslexia, and to determine new requisites to add to the current ones associated with the WCAG guidelines for web accessibility. On the other hand, authors would like to discover if the use of a specific font designed for users with dyslexia would be also welcomed by neuro-typical users in reading tasks. In order to achieve these goals, the authors designed an experiment involving students diagnosed with dyslexia. Results show that further modifications to the page style beyond those considered by WCAG (e.g., font type, size and column width) can be appreciated by students with dyslexia and may be considered as additional personalization options for this kind of users. In addition, the results of a second experiment highlight that both neuro-typical users and users with dyslexia prefer the proposed font for reading tasks, compared to a regular sans-serif font.
The research reported in the paper “MemHolo: Mixed Reality Experiences for Subjects with Alzheimer’s Disease”  investigates the potential of Wearable Mixed Reality technology, particularly HoloLens, for persons with Alzheimer’s Disease (AD). The authors collaborated with a team of AD specialists to develop MemHolo, an innovative HoloLens-based application for senior subjects affected by an initial stage of AD. MemHolo is intended as a cognitive training tool that complements regular therapeutic interventions and enables patients to practice short-term and spatial memory skills in a safe and controlled virtual environment. The authors exemplify how traditional brain training activities can be translated into Wearable Mixed Reality experiences for this target group and shed a light on how people with AD perceive HoloLens applications They also pinpoint potential benefits and drawbacks for these persons, paving the ground for new forms of treatments in this field. One of the main pros of Mixed Reality that AD experts acknowledged was the possibility to interact, at the same time, with the virtual and the real world. This reduces the risk of the isolation effect that may occur in pure wearable Virtual Reality. On the contrary, especially for people with AD, the presence of a caregiver or a family member in a familiar environment avoids getting scared or confused. The study reported in the paper shows that MemHolo created a surprising level of curiosity and engagement in participants, and improved socialization. None of the patients manifested confusion or real-virtual cognitive mismatch during the tests. Even though a person per time used HoloLens, presenting the research as a group activity created a collaborative environment where the others could help the subject during the activities and performed themselves a form of cognitive training. In addition, the interest in the application persisted after the sessions, as MemHolo was the subject of conversations among participants and with families in other moments of their life at the center.
The eight papers in the second section of the Special Issue are shortly introduced below.
A first group of two works – mainly theoretical – has used approaches partially borrowed from cognitive studies to understand the impact of specific features of video games on the gaming experience. In particular, they have tackled the meaning-making in games and the difficulties that human brain has to overcome when mapping human movement on traditional interfaces for games. In the paper “An exploratory study of the relationship between meaning-making and quality in games” , the authors investigated an intriguing topic: trying to identify and understand relationships among players’ perceptions of meaning-making affordances in games, and the quality of the experience lived by the player, her rating of the game and the experts rating. The underlying assumption is the conceptualization of gameplay as “an iterative and contextualized activity driven by meaning-making processes that integrate rational interpretation and affective valorization of key game aspects”. Adopting these lenses, the authors asked to some experienced players to evaluate the quality of several games, and then they compared the outcomes with scores provided by external sources. As a result, they noticed that an association between meaning-making affordances and critics’ evaluations may exist, but the same does not hold true for players’ game experience and player ratings.
A slightly different perspective is that adopted in the paper “Fighting the game. Command systems and player-avatar interaction in fighting games in a social cognitive neuroscience framework” . This work builds on the observation that video games in many cases require the player to interact using an avatar whose body shares many features with the player’s biological body. Examples of these are its aspect and movements, often very detailed and mimicking real humans’ ones. In this regard, models and theories in social cognitive neuroscience have tried to understand how humans understand their actions and other humans’ actions by putting in tight relation perception, imagination and execution of actions. These studies had an impact not only in the field of human cognition in itself, but also on the design of modern mimetic interfaces for games (such as virtual reality and the like). In spite of this, still too few efforts have been put in understanding how these theories can be applied to more traditional interfaces (such as gamepads), even if they represent the majority of the devices used by players to interact with the game. To understand the implications of this, the author has studied fighting games, which mostly use non-mimetic interfaces to move a human avatar, providing an effective framework to better understand interactions with avatars.
A subsequent group of works mainly focuses on techniques and approaches borrowed from different fields to improve game design. The aim is to provide tools useful to support the heavy and complex work of designing personalized games, gamified activities or HCI applications for research. Creating good gamified designs is effort intensive per se. The paper “A Process for Designing Algorithm-Based Personalized Gamification Demonstrated for a Computer-Supported Collaborative Learning Environment”  builds on the observation that, given the demonstrated importance of taking personal characteristics and preferences into account when designing gamified systems, “tailoring system interactions to each user will only add to this workload”. Hence, the authors propose to partially address this problem by producing personalized contents with the aid of appropriate machine learning algorithms. They exemplify the process by implementing personalized gamification for a computer-supported collaborative learning environment.
Similar assumptions underlie the work presented in the paper “FANTASIA: A Framework for Advanced Natural Tools and Applications in Social, Interactive Approaches” . The authors try to overcome several evident limitations of modern game engines (i.e., complex software applications, providing some already-ready functionalities, such as the physics simulation, used as development environments for video games) which, in several occasions, hinder the proper addressing of Human-Computer Interaction issues. Examples are those related to knowledge representation, probabilistic reasoning and voice synthesis. As a possible solution, the authors propose the FANTASIA framework, able to integrate a graph database, a dialogue manager, a game engine and a voice synthesis engine, and supporting the rapid design and implementation of interactive applications for HCI studies.
The third group of papers in this section stands in between theoretical and applicative approaches, by exploring novel applications of game-related paradigms as tools for transparently conveying knowledge, training and experiences. The work “Fostering Computational Thinking through Collaborative Game-based Learning”  digs into the importance of understanding an algorithmic solution to a problem as a driver to improve end-users’ involvement. Actually, from automatic checkouts, to e-banking and online shopping, the relevance of algorithms in our everyday life is increasingly growing. Hence the importance of acquiring Computational Thinking skills is evident when involving end-users in the design process of many systems. To tackle this issue, the authors propose a game-based system to foster CT skills, which they have tested in an exploratory study with real users.
The scope of the next paper with title “Developing and evaluating a BCI video game for Neurofeedback Training: The case of Autism”  is definitely much narrower and more precise. Its aim is to test the efficacy of a Brain Computer Interface (BCI) game as a mean to foster neurofeedback training in autistic children. The authors underline the lack of scientific evidence supporting the efficacy of BCI video games in brain training and try to contribute in filling this gap. To achieve this goal they describe a use case and the pilot-testing of a BCI game aimed at supporting autistic children while attending neurofeedback training sessions. Actually, caring for people with this type of impairment imposes cognitive, motor, behavioral, and attention challenges that current solutions targeted for other populations may not address.
The last, more applicative, set of works in this group addresses the issue of measuring the efficacy of approaches rooted into game design principles as a way to support teaching and learning activities. In “Playing to play: a piano-based user interface for music education video games”  the authors discuss a case-study where a video game is exploited to teach to the player how to use a complex and not-intuitive control interface (i.e., the keyboard of a piano). The authors’ approach is rooted into the tangible interface paradigm, which adopts everyday objects to control the user interface, and the encouraging outcomes of their research show that the subjects have been able to improve their knowledge of the piano keyboard after only two gaming sessions, with no significant differences between children with or without pre-acquired knowledge of the keyboard.
An approach quite similar to the previous one is adopted in the work described in the paper “BashDungeon: Learning UNIX with a video game” . The authors present an experimental video game designed to teach fundamentals of computer science to undergraduate students. The game consists of a dungeon designed to resemble the topology of a Unix file system: its rooms contain puzzles aimed at teaching the basics of Unix. The game has been tested on a class of B.Sc. students, demonstrating both a good acceptance of the metaphor and its effectiveness in supporting the learning process.
We want to close this editorial with thanks to both the authors, who have been willing to share their experience and research, and to reviewers, who contributed with their suggestions to clarify and improve the papers. We wish the readers a fruitful enjoyment of the presented papers, able to leverage new ideas and research directions.
Carmelo Ardito, Maria De Marsico, Davide Gadia, Dario Maggiorini, Ilaria Mariani, Laura Ripamonti, Carmen Santoro
Aruanno B, Garzotto F (2019) MemHolo: mixed reality experiences for subjects with Alzheimer’s disease. Multimed Tools Appl:1–21. https://doi.org/10.1007/s11042-018-7089-8
Corda F, Onnis M, Pes M, Spano LD, Scateni R (2019) BashDungeon: learning UNIX with a video game. Multimed Tools Appl:1–16. https://doi.org/10.1007/s11042-019-7230-3
Damiano R, Gena C, Venturini G (2019) Testing web-based solutions for improving reading tasks in dyslexic and neuro-typical users. Multimed Tools Appl:1–27. https://doi.org/10.1007/s11042-019-7273-5
Fabricatore C, Gyaurov D, Lopez X (2019) An exploratory study of the relationship between meaning-making and quality in games. Multimed Tools Appl:1–26. https://doi.org/10.1007/s11042-019-7232-1
Gennari R, Melonio A, Rizvi M (2019) Turn taking with turn-talk in group. Multimed Tools Appl:1–27. https://doi.org/10.1007/s11042-018-7090-2
Knutas A, van Roy R, Hynninen T, Granato M, Kasurinen J, Ikonen J (2018) A process for designing algorithm-based personalized gamification. Multimed Tools Appl:1–20. https://doi.org/10.1007/s11042-018-6913-5
Margetis G, Ntoa S, Antona M, Stephanidis C (2019) Augmenting natural interaction with physical paper in ambient intelligence environments. Multimed Tools Appl:1–47. https://doi.org/10.1007/s11042-018-7088-9
Mattiassi AD (2019) Fighting the game. Command systems and player-avatar interaction in fighting games in a social cognitive neuroscience framework. Multimed Tools Appl:1–27. https://doi.org/10.1007/s11042-019-7231-2
Mercado J, Espinosa-Curiel I, Escobedo L, Tentori M (2018) Developing and evaluating a BCI video game for neurofeedback training: the case of autism. Multimed Tools Appl:1–38. https://doi.org/10.1007/s11042-018-6916-2
Micheloni E, Tramarin M, Rodà A, Chiaravalli F (2018) Playing to play: a piano-based user interface for music education video-games. Multimed Tools Appl:1–18. https://doi.org/10.1007/s11042-018-6917-1
Mirehi N, Tahmasbi M, Targhi AT (2019) Hand gesture recognition using topological features. Multimed Tools Appl:1–26. https://doi.org/10.1007/s11042-019-7269-1
Origlia, A., Cutugno, F., Rodà, A., Cosi, P., & Zmarich, C. (2019) FANTASIA: a framework for advanced natural tools and applications in social, interactive approaches. Multimed Tools Appl, 1-36. https://doi.org/10.1007/s11042-019-7362-5
Thovuttikul S, Ohmoto Y, Nishida T (2019) Learning communication from first-and third-person POVs: how perceptual differences influence the interpretation of conversations whilst waiting. Multimed Tools Appl:1–25. https://doi.org/10.1007/s11042-019-7272-6
Turchi T, Fogli D, Malizia A (2019) Fostering computational thinking through collaborative game-based learning. Multimed Tools Appl:1–25. https://doi.org/10.1007/s11042-019-7229-9
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ardito, C., De Marsico, M., Gadia, D. et al. Special Issue on Advances in Human-Computer Interaction.
Multimed Tools Appl 78, 13353–13359 (2019). https://doi.org/10.1007/s11042-019-7690-5