Dear reader,

figure a

at present, we observe a rapid growth in the development of increasingly complex “intelligent” systems that serve users throughout all areas of their daily lives. They range from classical technical systems such as household appliances, cars, or consumer electronics through mobile apps and services to advanced service robots in various fields of application. While many of the rather conventional systems already provide multiple modalities to interact with, the most advanced are even equipped with cognitive abilities such as perception, cognition, and reasoning. However, the use of such complex technical systems and in particular the actual exploitation of their rich functionality remain challenging and quite often lead to users’ cognitive overload and frustration.

Companion Technologies bridge the gap between the extensive functionality of technical systems and human users’ individual requirements and needs. They enable the construction of really smart—adaptive, flexible, and cooperative—technical systems by applying and fusing techniques from different areas of research.

In our special issue we present interesting pieces of work—quite a number of new technical contributions, ongoing and completed research projects, several dissertation abstracts, as well as an interview—that are related to, or even fundamental for, Companion-Technology. In the community part of this issue, there is also a conference report on the first International Symposium on Companion-Technology [59].

1 Content

This issue starts with an introduction to Companion-Technology [48]; it reviews relevant research areas; possible fields of application of the technology; and related research projects including those that have a particular view on the notion of a technical Companion. The technical contributions span a wide range of areas: Worch et al. [73] introduce a robotic perception framework for perceiving and understanding everyday human tasks in a household environment. Schwarze et al. [68] introduce a system that assists visually impaired people to avoid obstacles in their close vicinity via sonification. The article by Gugenheimer et al. [58] shows how Companion-Technology can enhance a multi-screen TV system by placing projected screens anywhere in a user’s home. Mehlmann et al. [62] present a modeling approach that enables the realization of natural interaction in artificial social companions. The multimodal interaction between a human and a machine is empirically studied by Rösner et al. [65]. Demberg et al. [54] discuss challenges in natural language generation and show how this field can benefit from AI planning techniques.

Further articles present three excellent research programs, which are all funded by the German Research Foundation: The Transregional Collaborative Research Centre (CRC) “Companion-Technology for Cognitive Technical Systems” [50]; the CRC “Information Density and Linguistic Encoding (IDeaL)” [52]; and the CRC “Spatial Cognition” [64], which recently completed its 12 years research program.

Several dissertation abstracts from different fields of research show a variety of contributions to Companion-Technology. These include safety issues of intelligent and autonomous systems in open environments [55], multimodal behavior analytics [67], multimodal emotion recognition [69], and hierarchical planning [45, 66].

The issue ends with an interview with David E. Smith, a senior researcher in the Intelligent Systems Division at NASA Ames Research Center [47].

1.1 Survey

  • Companion-Technology: An Overview

    Susanne Biundo, Daniel Höller, Bernd Schattenberg, and Pascal Bercher

1.2 Technical Contributions

  • Perception for Everyday Human Robot Interaction

    Jan-Hendrik Worch, Ferenc Bálint-Benczédi, and Michael Beetz

  • A Camera-based Mobility Aid for Visually Impaired People

    Tobias Schwarze, Martin Lauer, Manuel Schwaab, Michailas Romanovas, Sandra Böhm, and Thomas Jürgensohn

  • How Companion-Technology can Enhance a Multi-Screen Television Experience: A test bed for adaptive multimodal interaction in domestic environments

    Jan Gugenheimer, Frank Honold, Dennis Wolf, Felix Schüssel, Julian Seifert, Michael Weber, and Enrico Rukzio

  • Modeling Grounding for Interactive Social Companions

    Gregor Mehlmann, Kathrin Janowski, and Elisabeth André

  • Desiderata for the design of companion systems: Insights from a large scale Wizard of Oz experiment

    Dietmar Rösner, Matthias Haase, Thomas Bauer, Stephan Günther, Julia Krüger, and Jörg Frommer

  • Search Challenges in Natural Language Generation with Complex Optimization Objectives

    Vera Demberg, Jörg Hoffmann, David M. Howcroft, Dietrich Klakow, and Álvaro Torralba

1.3 Research Projects

  • Companion-Technology for Cognitive Technical Systems

    Susanne Biundo and Andreas Wendemuth

  • Information Density and Linguistic Encoding (IDeaL)

    Matthew W. Crocker, Vera Demberg, and Elke Teich

  • Cognitive Space and Spatial Cognition: The SFB/TR 8 Spatial Cognition

    Marco Ragni, Thomas Barkowsky, Bernhard Nebel, and Christian Freksa

1.4 Doctoral Dissertations

  • Safety of Autonomous Cognitive-oriented Robots

    Philipp Ertle

  • Multimodal Behavior Analytics for Interactive Technologies

    Stefan Scherer

  • Emotional and User-Specific Acoustic Cues for Improved Analysis of Naturalistic Interactions

    Ingo Siegert

  • Hybrid Planning and Scheduling

    Bernd Schattenberg

  • Search Complexities for HTN Planning

    Ron Alford

1.5 Interview

  • Interview with David E. Smith

    Pascal Bercher and Daniel Höller

2 Service

Although Companion-Technology is an intrinsically interdisciplinary field, we restrict our review on publication media to those of AI and related areas.

2.1 Conferences, Symposia, and Workshops

Companion-Technology, Cognitive Technical Systems

  • Advances in Cognitive Systems [1]

  • International Conference on Advanced Cognitive Technologies and Applications (COGNITIVE) [2]

  • International Conference on Cognitive Systems (CogSys) [3]

  • International Symposium on Attention in Cognitive Systems (ISACS) [4]

  • International Symposium on Companion Technology (ISCT) [5]Footnote 1

  • International Symposium on Resilient Cognitive Systems (ISRCS) [6]

  • International Workshop on Emotion Representations and Modelling for Companion Systems (ERM4CT) [7]Footnote 2

  • International Workshop on Multimodal Pattern Recognition of Social Signals in Human–Computer-Interaction (MPRSS) [8]

Robotic Companions, Social Robots, Robot/Human-Interaction

  • International Conference on Human–Robot Interaction (HRI) [9]

  • International Conference on Human–Robot Personal Relationships [10]

  • International Conference on Social Robotics (ICSR) [11]

  • International Symposium on New Frontiers in Human–Robot Interaction [12]

  • International Symposium on Robot and Human Interactive Communication [13]

  • Symposium on Robot Companions: Hard Problems and Open Challenges in Human–Robot Interaction (2005) [14]

Human/Computer-Interaction

  • Australian Conference on Human–Computer Interaction (OzCHI) [15]

  • Conference on Human Factors in Computing Systems (CHI) [16]

  • HCI International Conference [17]

  • Intelligent User Interfaces (IUI) [18]

  • International Conference on Human–Computer Interaction (INTERACT) [19]

  • International Conference on Intelligent Environments (IE) [20]

  • International Conference on Multimodal Interaction (ICMI) [21]

  • International Conference on Pervasive Computing and Communications (PerCom) [22]

  • International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp) [23]

  • Nordic Conference on Human–Computer Interaction (NordiCHI) [24]

  • The ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS) [25]

2.2 Journals

Companion-Technology, Cognitive Technical Systems

  • Advances in Cognitive Systems [1]

  • Cognitive Systems Research [26]

  • Cognition, Technology and Work [27]

  • Cognitive Technology Journal [28]

  • IEEE Intelligent Systems [29]

  • IEEE Transactions on Human–Machine Systems [30]

  • Journal of Cognitive Engineering and Decision Making (JCEDM) [31]

  • Special Issue on Cognition for Technical Systems (in Künstliche Intelligenz 2010) [46]

  • User Modelling and User-Adapted Interaction [32]

Robotic Companions, Social Robots, Robot/Human-Interaction

  • International Journal of Social Robotics (IJSR) [33]

  • Journal of Human–Robot Interaction [34]

  • Special Issue on Human–Robot Interaction (in IEEE Transactions on Robotics 2007) [61]

  • Special Issue on Personal Robotics (in Autonomous Robots 2001) [51]

Human/Computer-Interaction

  • ACM Transactions on Computer-Human Interaction [35]

  • ACM Transactions on Interactive Intelligent Systems (TiiS) [36]

  • Communications of the ACM [37]

  • Human–Computer Interaction [38]

  • i-com: Journal of Interactive Media [39]

  • Interacting with Computers [40]

  • International Journal of Human Computer Interaction (IJHCI) [41]

  • International Journal of Human–Computer Studies [42]

  • Journal of Ambient Intelligence and Smart Environments (JAISE) [43]

  • Journal on Multimodal User Interfaces (JMUI) [44]

2.3 Books

  • Affective Computing (1997) [63]

  • Close engagements with artificial companions: key social, psychological, ethical and design issues (2010) [71]

  • Cognitive Behavioural Systems (2012) [57]

  • Companion Technology—A Paradigm Shift in Human-Technology Interaction (2016, forthcoming) [49]

  • Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (2012) [60]

  • Joint Cognitive Systems: Patterns in Cognitive Systems Engineering (2006) [72]

  • Socially intelligent agents: Creating relationships with computers and robots (2002) [53]

  • The Oxford Handbook of Cognitive Engineering (2013) [70]

  • Toward Autonomous, Adaptive, and Context-Aware Multimodal Interfaces (2011) [56]