This Special Issue covers some important aspects of the overlap between MultiMedia (MM) and Human-Computer Interaction (HCI). The authors of the 16 papers were invited to revise and expand conference presentations. Each expanded paper underwent an additional round of rigorous peer review and revision. Invitations went to the authors of the best papers shown at the 11th Edition of CHItaly, the biannual Conference of the Italian SIGCHI Chapter, whose theme was, “Public, Private, and Community-Based interaction”, and thus aimed at encompassing the different souls underlying the new trends in HCI, from interaction with public large displays to interaction with mobile devices, to crowdsourcing/community-based kinds of interaction. However, the conference has also hosted innovative contributions regarding advanced interaction paradigms and models, new ways of interpreting human-computer interaction and new “natural”, “transparent”, “ubiquitous” and most of all “inclusive” interfaces.

In the extremely engaging scenario of advanced HCI, new and often unforeseen technological achievements play a fundamental enacting role. In particular, multimedia tools provide a unique opportunity to increase expressivity, usability, and enjoyment of computer interfaces, and this should be taken into account when designing them. However, this also requires an even greater attention to user needs and extensive field studies to investigate the most suited solutions for each kind of application and user category. Only in this way it is possible to achieve a really advanced HCI through MM.

We are glad to offer in this Special Issue some very interesting examples of a smart use of MM, and also examples in which multimedia tools are both the object and the instrument in the education of future software designers. It is interesting to notice how teaching/learning (through) MM has become one of the fields where the most intensive experimentation is carried out. As a matter of fact, a number of papers presented here tackle related issues, especially when children are involved. Affective and emotional multimedia, and multimedia for improving the interaction with elder and disabled people, are further examples of application fields considered herein. Finally, the special issue deals with frameworks and conceptual tools for the design of advanced multimedia applications in ambient intelligence, cultural heritage, and system recommendation.

The paper “Designing visualizations of temporal relations for children: action research meets HCI”, by Tania Di Mascio, Rosella Gennari, Laura Tarantino, and Pierpaolo Vittorini deals with children-oriented Technology Enhanced Learning (TEL) systems. The EU project TERENCE developed an adaptive learning system supporting 7-11 years old children characterized by “poor text comprehension”, a cognitive disability related to problems in carrying out reading activities. The work relies on the definition of a novel children-oriented data gathering technique that allowed the authors to carry out a study on the children mental model of time. User Centered Design (UCD), especially tailored on children needs, and Action Research (AR) are the design tools used to develop a children-oriented integrated “read-n-play” visual environment able to force the acquisition of the capability of reasoning about temporal events within a story. MM is the key to obtain high flexibility, adaptability, and possible reusability of game-based modules.

As already claimed, MM can be the tool, but also the object of education. The paper “HCI and education: a blended design experience” by Fabio Pittarello and Tommaso Pellegrini deals with problems related to teaching HCI in an undergraduate course. It reports some of the results of educational experiences where both learners and teachers were actively involved in a process of knowledge construction and design, using multimedia tools and strategies. In particular, the students were presented with the importance of using multimedia components for designing engaging interfaces, capable of attracting the users for informing them.

MM especially support gamification activities, which are deemed to represent an effective way to attract and motivate learners. The paper “Gamified probes for cooperative learning: a case study” by Rosella Gennari, Alessandra Melonio, and Santina Torello focuses on gamified probes for promoting a sense of progression and control, as well as social relations in a cooperative learning process in classroom. The exploited probes have limited ad-hoc functionalities, tested in the field, and are designed to be flexible enough to enable different usages. Such gamified probes can be further enhanced with technology when needed, to enable specific functionalities. In practice, they act as technology probes: they are endowed with micro-controllers, sensors and actuators for transparently improving interaction of children with gamified probes, for promoting interactions among children and educators, and both for logging relevant data according to the expected functionalities of probes, and for storing interaction data relevant for the learning process as well.

A further example of gamification for social activities is presented in the paper “Fighting exclusion: a multimedia mobile app with zombies and maps as a medium for civic engagement and design” by Catia Prandi, Marco Roccetti, Paola Salomoni, Valentina Nisi and Nuno Jardim Nunes. It presents an interesting study focusing on accessibility in an original way. The paper presents a study on urban data crowdsourcing driven by the proposed application, named Geo-Zombie. This is a multimedia mobile application aimed at engaging pedestrians in collecting urban accessibility data through pictures and multimedia content. Five different initial multimedia mobile applications are submitted to a number of evaluation activities and to field trials to refine the design and to select the best app.

Of course, game design goes well beyond gamification of specific user activities. It has an autonomous role, and of course the use of multimedia elements and their conceptual interpretation is crucial. The paper “An ontology for videogame interoperability” by Janne Parkkila, Filip Radulovic, Daniel Garijo, María Poveda-Villalón, Jouni Ikonen, Jari Porras, and Asunción Gómez-Pérez presents the Video Game Ontology (VGO), a model for enabling interoperability among video games and enhancing data analysis of gameplay information. The authors describe the process and stakeholders involved in the creation of the ontology, the ontology conceptualization and its evaluation. They also present a practical demonstration of the applicability of the Video Game Ontology with three example games that take advantage of the created ontology. The use of the VGO enables better interoperability among the example games, which is an aspect in video game designed that was quite neglected notwithstanding the increasing popularity of this field.

Along a similar line, the paper “Procedural content generation for platformers: designing and testing FUN PLEdGE” by Laura Anna Ripamonti, Mattia Mannalà, Davide Gadia, and Dario Maggiorini, tackles the issues related to video game design. In the presented work, the authors discuss the design, prototyping and testing of FUN PLEdGE, a general-purpose automated level generator and editor for platform video games (Platformers). Its main goal is to assist designers by producing – unassisted – game levels and guaranteeing the possibility to modify and personalize by hand the generated levels. The obtained level generator can be integrated in Unity 3D, a diffused state-of-the-art game engine. The results obtained in creating a prototype of Platformer were assessed by carrying out both tests with players and with game developers.

In modern research, MM plays a novel and important role also in affective computing and emotional analysis of HCI. The paper “Stress in interactive applications: analysis of the valence-arousal space based on physiological signals and self-reported data” by Alexandros Liapis, Christos Katsanos, Dimitris Sotiropoulos, Nikos Karousos, and Michalis Xenos explores an original way of using multimodal channels of human-computer communication. In particular, the presented experiments aim at measuring users’ emotional reaction to interactive multimedia and hypermedia. The authors exploit a particularly popular self-reported method for emotion assessment, namely the Valence-Arousal (VA) Scale, which is a 9x9 affective grid. Actually, the paper aims at going beyond self-reporting. This is obtained by identifying specific stress region(s) in the VA space through the combination of self-reported ratings and physiological signals, namely skin conductance. The results allow HCI and interactive MM researchers and practitioners to employ the Affect Grid in their Users’ Emotional Experience (UEX) evaluation studies, knowing a priori that a specific VA region is associated with both perceived and physiologically experienced stress.

Ambient Assisted Living is one of the fields where affective computing plays its role. The paper “Simulating empathic behavior in a social assistive robot” by Berardina De Carolis, Stefano Ferilli, and Giuseppe Palestra, describes how an affective reasoning has been implemented in the NAO robot for simulating empathic behaviors in the context of Ambient Assisted Living. This kind of reasoning entails the capability of recognizing the user’s affective state to be able to trigger the most appropriate communicative behavior. Speech and facial expression are used as sources of cues for affective reasoning. The authors exploit a combination of several classifiers, used to detect the features that come into play when deciding the affective state to be triggered. Dynamic Belief Networks are used to reason on them and take a decision.

The design of interactive products that improve users’ experience and feelings is discussed in the paper “Experience over time: evaluating the experience of use of a squeezable interface in the medium term” by Patrizia Marti and Iolanda Iacono. The authors developed a soft rubber cover for tablets, called “Squeeze Me”, which is able to communicate with different software applications. The coating material embeds pressure sensors to detect input from the user and actuators to provide a haptic feedback. Squeeze Me has been used to implement a mobile application, called “Squeeze to zoom”, which allows the user to zoom in and out while taking a photograph without removing the hands from the tablet. The experience of use of this application has been compared with zooming modalities commonly available on tables and smartphones, both in a short-term field study (with 67 participants) and over a four-week study (with 8 participants). Interesting results have been discovered about the impact that the type of task can have on the experience of use and about the evolution over time of the relationship between users and products, namely about how user experience may change over time.

The use of MM is of course a powerful tool to facilitate user interaction, and also to promote accessibility of applications and services. The paper “FATCHA: biometrics lends tools for CAPTCHAs” by Maria De Marsico, Luca Marchionni, Andrea Novelli, and Michael Oertel presents a novel strategy to implement a CAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart). The approach underlying FATCHA aims at recognizing a human user by exploiting multimodal interaction. Real time capture of human actions is entailed instead of human ability to recognize visual or auditory items, in order to propose a challenge that is difficult for an automatic responder but easy for a human. In particular, compared with traditional CAPTCHAs based on text, this approach is intended to better support users with visual or auditory impairment. FATCHA design borrows from detection/recognition techniques developed in the biometric field. In fact, it is also possible to extend the service with a second module that allows the user authentication by face recognition, instead or besides using passwords. In this context, FATCHA gesture challenge can be used as a liveliness test to avoid biometric spoofing.

The main focus of the paper “Analyzing visually impaired people’s touch gestures on smartphones” by Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Amaury Trujillo is instead to study whether and how gestures on smartphones could be improved, in order to make touch-based interaction by visually impaired users more accessible and usable. An experiment involving 36 participants, blind from different stages of life or with low vision, is described in the paper. A very detailed analysis of participants’ gesture performance and their preferences regarding touch-based smartphones is presented, in order to finally offer a set of recommendations for choosing and designing gestures that are easier for visually impaired people to perform. An additional contribution of this paper is also the novel approach and the related system developed to capture touch-based gestures from several users simultaneously; this system is based on web technologies and wireless communication. Overall, this paper represents an interesting stepping stone in the research about accessibility of handheld touchscreen devices.

Elder adults constitute another category of users that require the design of suitable software tools. The paper “Interactive Multimedia Content Fruition in Older Adults: The Case of SeniorChannel” by Valeria Orso, Anna Spagnolli, Luciano Gamberini, Francisco Ibanez, and Maria Elena Fabregat is aimed at demonstrating that the interactive TV model, in its various platforms (smart TV devices, PCs, tablets) may be used to involve elder adults actively in content fruition and provide them with an opportunity to increase their social involvement in the local community. The paper contributes with a list of design guidelines for interactive multimedia systems devoted to this kind of users, extracted from literature and further elaborated to develop SeniorChannel TV, an application that allows users to access and interact with TV content. The paper then describes the results of two test studies, which provide very positive results and suggest that interactive multimedia systems can foster social inclusion of the elder population.

MM plays an important role also in the formulation of models of advanced design. The paper “A layered structure for a design space dedicated to rich interactive multimedia content” by Augusto Celentano and Emmanuel Dubois builds on the core approaches of focus + context and overview + detail. The authors propose and discuss a layered organization for a design space suitable for the visualization of and interaction with rich information environments. These environments rely on multimedia and multidimensional information. Furthermore, the proposal aims at supporting multi-device deployment. Different information categories are identified and correspond to peculiar user goals and exploration spaces. In order to implement a flexible and efficient structure, each information category is associated to a different design space layer.

Along a similar line, the paper “Rule-based tools for the configuration of ambient intelligence systems: a comparative user study” by Federico Cabitza, Daniela Fogli, Rosa Lanzilotti, and Antonio Piccinno presents the first stage of a research agenda that concerns the development of a conceptual framework for ambient intelligence environments. In the paper, the authors focused on the user layer of the framework, which represents that part of the system where inhabitants interact with the environment, and are called on to modify its behaviors through the definition of event-condition-action (ECA) rules. An analysis of end user-oriented tools for ECA rule creation is thus proposed, along with a choice of two visual tools, namely Atooma and IFTTT, to be more deeply compared through a user study. The obtained quantitative and qualitative results not only provide indications on how to pursue the research agenda, but also about the design, in general, of end user-oriented tools for ambient intelligence.

Another framework for multimedia systems is presented in the paper “Supporting the consumption and co-authoring of locative media experiences for a rural village community: design and field trial evaluation of the SHARC2.0 framework” by Keith Cheverst, Helen Turner, Trien Do, and Daniel Fitton. This framework supports the authoring and consumption of Locative Media Experiences (LMEs) and has been developed in the frame of a ‘research in the wild’ project aimed at creating tools that enable residents of a rural village to co-curate LMEs related to the village’s local history. It comprises a web-based tool and a mobile app for authoring multimedia content, and a web-based tool and a mobile app for LME consumption. Personal cloud storage is used to facilitate both local and distributed ownership of LMEs and to support synchronization of multimedia content. The results of the framework evaluation with visitors and residents provide insights into the potential value of supporting the co-authoring of locative media, as well as into the challenges associated with the approach, such as the burden on users with moderation responsibilities and the implications for long-term participation and sustainability.

In the paper “User interface patterns in recommendation-empowered content intensive multimedia applications” by Paolo Cremonesi, Mehdi Elahi and Franca Garzotto, design patterns are chosen as powerful design tools to improve the quality of user interfaces of multimedia recommender systems. Through a systematic analysis of 54 recommendation-empowered content intensive multimedia applications, the authors have discovered the occurrences of existing general (i.e., domain independent) user interfaces patterns and have identified recurrent user interfaces design solutions. The results of the inspection have been discussed and evaluated with a team of experts, thus leading to a consolidated set of 14 new design patterns, some of which specific of recommendation systems, and others useful also in different contexts. Finally, the paper contributes to the research in interface design with a general procedure for pattern definition, which is a mix of pattern mining and expert validation.

We hope the papers presented here may inspire new research and new achievements. It is important to take in mind that technology must always be supported by careful consideration of user needs, and by search for inclusion. If this guideline becomes a fixed point in MM design, new exciting possibilities can be opened to all users.