Everyday automation experience: a research agenda

Introduction

Automation is finding its way into many parts of everyday life. This is manifested by the increasing opportunities for end users to offload decisions to their home appliances, to hand over control to their cars, to step into automated trains, or to go for shopping at self-checkout stores. Supported by artificial intelligence, emerging automated services are getting closer towards human cognitive functions, as they are integrating analysis and decision based on the processing of large information sets [48]. In anticipation of the broad impact that these emerging technologies will have on our sphere of life, a reflective and systematic consideration is necessary that leverages their full potential in terms of user experience. While there is a long human factors research tradition on automation, scholars have for long concentrated on highly specialized professional work tasks for trained personnel, such as control center operators or pilots. With technological innovations and new use cases in domains such as transport, home automation, and retail, the goal of designing automation for a broader population has become more important. This transition of automation technology towards the everyday life thus has put the experience of “naïve users” into the center of attention.

This special issue contains selected scientific contributions from two workshops at the CHI 2019 and CHI 2020 conferences [17, 20]. The goal of these workshops was to investigate the requirements and design approaches for everyday automation experience. Participants within these workshops came from different domains of science and practice, which led to a first systematic discussion of the scope, already pre-existing knowledge and remaining issues of this newly involving field. The papers have a broad scope in terms of application fields, but they all have in common that they seek to deeply understand specific aspects of user experience and the design of everyday automation systems. The selection of papers within this special issue reveals the breadth of experience aspects that need to be considered when dealing with automation experience.

In the first paper, Thomas Lindgren, Vaike Fors, Sarah Pink, and Katalin Osz investigate the anticipatory aspect of automation experience, drawing on a 1.5-year ethnographic study where five families used research cars with evolving automated driving functions in their everyday lives. Their analysis focuses on people’s gradual adoption to automated cars. Lindgren et al. impressively uncover the involved processes that involve anticipating if users can trust automated driving functions, what the ADAS features will do, and what the longer term technological possibilities will be. They also show that this anticipatory UX occurs within specific socio-technical and environmental circumstances, which would not be easy to derive in experimental settings. This form of studying anticipation therefore offers us new insights into how people adopt automated driving in their everyday commuting routines.

Elçin Hancı, Joyca Lacroix, Peter A. M. Ruijten, Antal Haans, and Wijnand IJsselsteijn investigate the role of personal attitudes and motivation for the use of self-tracking technologies as a new form of automation experience that impacts our understanding of body and behavior. An analysis of the survey data together with data from motivation for self-tracking scale indicates that commitment to self-track positively correlates with autonomous motivation for tracking and negatively with controlled motivation. The research shows how self-tracking technology, as a novel automation experience, impacts users’ everyday behaviors. Furthermore, the paper discusses the relevance of interpersonal differences in attitudes and motivations towards continuous automated collection of personal data, and it highlights the methodological challenges for measurement.

Holger Klapperich, Alarith Uhde, and Marc Hassenzahl explore the trade-offs of the comfort achieved by everyday automation with the joyful experiences of interaction with a product. As an interesting result of their field research, they show that manual interactions have experiential benefits over more automated processes. Using the example of coffee making, Klapperich et al. present a method that considers potential experiential costs of everyday automation and strategies of how to design interaction with automation. They conclude that full automation may not necessarily be the ultimate goal for the design of everyday artifacts, as many activities can nurture enriched and more meaningful experiences. Furthermore, they propose to restart the discussion on levels of automation and thereby to bring in more strongly the aspect of the users’ experience instead of focusing on technological sophistication and unreflected efficiency.

Muneeb Imtiaz Ahmad, Ingo Keller, David Robb, and Katrin Lohan investigate cognitive load as a correlate of automation experience. They propose a framework to detect cognitive load by non-intrusively measuring physiological data from the eyes and the heart. Using purely such physiological data sources, Ahmad et al. show that different levels of cognitive load can be accurately predicted by means of their presented approach, and they outline how these insights could be applied to different application contexts in automation, robotics, and special vehicle operation. Attentive interfaces could build on this and further scientific accounts, in order to monitor the time-varying capabilities of users, in order to propose suitable manual interventions and handovers.

Carmelo Ardito, Giuseppe Desolda, Rosa Lanzilotti, Alessio Malizia, Maristella Matera, Paolo Buono, and Antonio Piccinno demonstrate and evaluate an approach how non-programmers can be supported to configure and orchestrate automated devices through the Internet of Things (IoT) for expressive and immersive end-user experiences. Ardito and colleagues experiment with different approaches that enable domain experts to identify and specify properties of IoT devices through user-defined semantics and thereby to facilitate them to automate the IoT devices’ behavior. In their study, they compare three different interaction paradigms that support the specification of user-defined semantics through a “transparent” use of ontologies. Furthermore, a set of conclusions is proposed with regard to how the proposed paradigms help domain experts express their semantics, which in turn facilitates the creation of interactive applications enabling smart interactive environments.

Alexander Georg Mirnig, Magdalena Gärtner, Elisabeth Füssl, Karin Ausserer, Alexander Meschtscherjakov, Vivien Wallner, Moritz Kubesch, and Manfred Tscheligi direct their attention to moments where automated operation comes to its limits. In their study on incident management in fully automated public transport, they explored participant’s behavior and willingness to assist in solving problems in a variety of scenarios where the bus suddenly stops for technical reasons or a hypothesized situation of harassment. The authors found that suggesting responsible behaviors is an existent barrier, when passengers are involved in the resolving of potentially dangerous situations in a public transport setting. The authors provide recommendation on how to address this challenge, by the multimodal design of incident-relevant instructions that are explicit, brief, timely, and distinguishable from regular on-trip information,

In the remainder of this editorial, we draw an agenda that shall function as a guiding frame for future everyday automation experience research endeavors. First, we highlight the special features of everyday automation experience that research should address, and then we look at the way how experience can be captured. We also highlight the benefits and potential limitations of investigating automation experience across various application domains. We then point towards promising design approaches that could help advance this field.

Addressing the special features of everyday automation experience

Human experience in general as well as user experience is a highly complex and entangled phenomenon. Thus, many scholars from different fields and research traditions have explored and conceptualized the various facets and the broad spectrum of experiences with technology in different contexts and applications domains. This richness and variety of research is, for example, represented in the subcommittee “User Experience and Usability” of the annual CHI conference. The experience of automated systems by non-professional users in everyday situations must be described and analyzed in a qualitatively different way than both conventional forms of human-machine interaction and professional automation use cases. In the following, we introduce into special features of automation experience (see Fig. 1), and for each of them, we highlight related open questions that need to be addressed to improve our understanding of automation experience.

Fig. 1

Special features of everyday automation experience

Uninitiated encounters

People increasingly encounter systems in their surroundings that they are not habituated or trained for. While first-time uninitiated usage is already a challenge with standard interfaces, this gets more delicate when a service or system works on its own, without explicit user commands. One of the involved challenges is that their behavior may not be as easily foreseeable as when operated by a human. For example, an automated vehicle approaching may be driving differently than what a pedestrian may have expected. Likewise, the automated handling of processes in the public, such as increasingly in airport check-ins or in supermarkets, is something that can strongly impact the overall experience. In order to improve intelligibility, automotive human factors research has investigated ways to realize awareness and intent communication, towards other road users or passengers have been investigated (see, e.g., Mirnig et al. from this special issue or [19]).

Implicit interaction

In everyday automation usage situations, continuous interaction is typically not as central as with manually operated systems. Relationships of users to automation in everyday contexts can be rather described as forms of “implicit interaction” [29, 56, 59] or peripheral interaction [4]. But while implicit and peripheral interaction is often thought to be in the direction that users provide inputs to a system, automated systems can often go further by taking initiative by addressing users with their requests and actions. How to make users understand “when it is their turn”, when they can override, and where they better follow the recommendation of a system are corresponding fundamental research questions. More in general, the question arises, what it means to experience ubiquitous automated systems if they are increasingly encountered indirectly or on the periphery of users’ attention.

Ambient embeddedness

Supported by new forms of implicit interaction, many automated systems do not anymore feature prominent displays, as they are merging into the users’ surroundings. In a sense, everyday automation is thus a strong driver for putting into practice the visions of “ubiquitous computing” [58, 73], the “disappearing computer” [66], as well as ambient intelligence [10]. The ambient nature of automated systems and their interwovenness in mundane, repetitive routines also supports the ordinariness of the involved user experience [11, 40]. Modern building management systems are a good example for underlying automation that frees users from complexity, but that can also be intransparent [1]. Also, here it is important to provide, when needed, cues on how to access more detailed information about what is going on and how building tenants can intervene in case they want to change the system behavior.

Decoupledness

Automation has been around us below the level of noticeability for a long time already, interweaving with critical infrastructure such as water and electricity supply, escalators, or conveyor belts. The now evolving forms of automation incorporate higher cognitive functions and through their behavior are more pervasively entering our sphere. Household energy management is a good example: many of us have become used to some form of heating regulation processes in the background. However, when demand-response systems remote control our boiler, the electric car, and other appliances in the household, we are being faced with a central instance that impacts more than just a subsystem, as it may have more overview over the own house than ourselves. However, even when appliances, transactions, and processes are completely automated, without direct user control, there is often still some remote monitoring of multiple devices. Examples for this are automated mobility services with control centers that intervene in case of an irregular situation. Questions related to this are how to provide a sense of potential remote coordination of automated services. Given the processing of data is done at other places, as is the case with today’s distributed and cloud-based smart assistants, research should also more systematically address privacy, security, and trust concerns of such forms of remote automation approaches.

Perceived system adaptation

With the rise of artificial intelligence, systems in the user’s surroundings are enabled to learn and consequently change their behavior. While the adaptation to user preferences can increase comfort, this may aggravate the above-mentioned challenge of intelligibility. Related issues that should be addressed are thresholds for perceivability and acceptance factors of system adaptation over longer time periods and the possibility of modifying adaptations when they are not effective.

Substitutive experiences

When an activity does not have to be performed any more, since it has become automated, people can do other things. For example, in a highly automated car, different types of replacement activities have been developed, reaching from relaxation and entertainment to work. In such safety-critical situations, replacement activities bear the risk of demanding too much attention, in case a manual handover is needed. While there are many promising approaches, more research needs to be undertaken to enable for engaging and comfortable replacement tasks while maintaining situation awareness. In other domains, such as the home and public areas, replacement activities have been addressed not so intensely. A more reflective consideration of the substitutive experiences evoked by replacements of activities by automation should be enabled also in these areas. For less safety-critical situations, requirements and design research could encompass ways to inspire people to turn to more challenging and engaging activities than passive entertainment or to recommend activities that fit to personal profile and interests.

Interventions

In many usage contexts, manual interventions have to be supported, in order to better adapt the system behavior to the respective context and preferences. Such interventions could make a system more useful, such as when the window blinds in the smart home are not behaving in the right way such that comfort is compromised. This way one more or less abruptly has to swap from a passive mode towards an expert both in using the system and the domain the system is operating in. In other, more safety-relevant cases, these interventions will call for quick and reliable human responses. The prime example for this is the control handover in automated driving, where the situation must be quickly grasped. Related challenges are to increase the user’s understanding of which interventions can be performed and which consequences they will entail.

End users as programmers

Apart from situation-dependent interventions, people can also get into a quite active role when it comes to the customization of automated systems. In fact, automation enables the potential for the creative planning and customization of system behaviors to people who have never programmed before. Many services, such as Amazon Alexa, offer means for end-user programming, such as the configuration of routines by If-This-Then-That methods. A central question in this regard is how to enable people with no or little programming skills to customize the behavior of a system. The possible solutions should allow users to understand whether what they specify actually corresponds to the desired behavior. Such solutions should be available for immediate and easy use, for example, by allowing users to specify their desired personalization rules through direct interactions with the objects involved in the automation. However, when non-programmers are supported, there also need to be mechanisms in place that safeguard the correct functioning of the system. Additionally, one needs to address the paradox challenge of both enabling informal end-user programming languages while still maintaining formality on the system’s side.

These special features of everyday automation each comprise their own research challenges. But it is also necessary to enable a more comprehensive understanding of their interrelationship and their dependence on contextual factors. When it comes to conceptualizing and capturing the specific forms and facets of everyday automation experience, challenges arise from these specific issues. Research on everyday automation experience has to address these specifics of everyday automation encounters characterized by interactions that are less direct and explicit as well as rather unobtrusively and peripherally perceived. For example, we need to develop and readily make available methods to capture user experience within the implicitness of everyday automation interactions, the peripheral encountering of automated systems, as well as the ordinariness of experiences with automation technologies. In the following section, approaches for investigating these issues are discussed.

Unpacking and capturing the facets of everyday automation contexts

A main challenge for future research on everyday automation experience is to explore and unpack the full spectrum of experiences with automation technology that might be specific for different domains and usage contexts. To unpack and appropriately capture the varying facets as well as the complexity of everyday automation experience, conceptual and methodological work is needed to provide an understanding of everyday automation experience across domains, especially as current predominant conceptions of user experience [25, 74] might fall short in framing and conceptualizing everyday automation experience and its special features.

Research on everyday automation experience needs to explicitly reflect on and articulate which kind of everyday automation experience is captured. Certain studies and research projects deploy different research designs, different approaches, different theoretical backgrounds, and different methods. This inherently leads to different facets, features, and forms of everyday automation experience that are captured. For this purpose, we refer to the framework introduced by Kuutti and Bannon [33] that distinguishes between an “interaction perspective” and a “practice perspective” in human-computer interaction and transfer it to the field of automation experience in order to articulate and structure different analytical perspectives on everyday automation experience.

Interaction perspective on automation experience

Building on Kuutti and Bannon’s [33] framework, automation experience research taking an “interaction perspective” centers around the very specific interactions between humans and automated technologies. This perspective focuses on the momentary and ahistorical experiences that arise from human-automation encounters and usually comes along with methods from psychology, for example, conducting controlled experiments in the lab and/or short-term studies focusing on users’ tasks. An interaction perspective might be particularly challenged by the “disappearing interaction” phenomenon related to ubiquitous automated systems but even so fruitful to address this phenomenon.

Practice perspective on automation experience

On the opposite, Kuutti and Bannon [33] propose a so-called “practice perspective,” which can be characterized by “decentering the human and the computer” [51]. Transferred to the exploration of everyday automation experience, research taking a practice perspective focuses on processes and long-term actions that are interwoven with the physical, social, and cultural surroundings. This perspective is often related to the deployment of qualitative methods, observational studies, long-term engagement with users in natural settings, and studies of activities surrounding the specific interaction of a human and an automated system. A “practice perspective” on automation experience might be specifically suitable to address the ordinary and mundane facets of everyday automation experience or to explore experiences related to the ambient embeddedness of automated systems.

We suggest relying on this framework of “interaction” and “practice perspective” to frame different analytical positions regarding everyday automation experience. It enables to distinguish and cluster the different angles of research on everyday automation experience as well as to articulate which analytical stance is taken to address, capture, and conceptualize certain features of everyday automation experience, thus rendering this differences visible. Certain research methods are particularly suitable for capturing specific forms of everyday automation experiences, and authors should reflect on how specific methods are applied and can be deployed to unpack and capture different features and nuances of everyday automation experience. This might provide a basis for further reflections and conceptual as well as methodological work on everyday automation experience.

Design concepts and development approaches for everyday automation experience

Researchers have started to work on generic design guidelines for automated systems. Examples include the concept of intervention user interfaces by Schmidt and Herrmann [57] and corresponding design principles derived from Shneiderman’s well-known golden rules [64]. Amershi et al. [3] proposed 18 design guidelines for human-AI interaction gathered from literature. While these principles and guidelines are generally applicable, we argue for an increased consideration of the characteristics of everyday automation in order to enhance the user experience of such appliances. In the following sections, we elaborate on promising research directions for improving interaction with, intelligibility of, and trust in ubiquitous automated systems.

Improved interventions

Despite advances in artificial intelligence, there will be situations, in which an automated system should sporadically enable interaction with their users. Such interactions include its configuration, potential interventions during the automated process, the confirmation of recommended actions, and rating and giving feedback on completed tasks. Fundamental for designing beneficial and user-accepted automated applications for everyday life, we consider a deeper understanding of the various application domains and users’ respective expectations and requirements. Application domains for everyday automation might widely differ regarding desired interaction opportunities. For example, public transport users might appreciate an automated service selecting and buying the most suitable ticket for their journey. However, this fully automated “action implementation” [50] might not be desirable in a restaurant or shopping context where users might probably experience a loss of control and feel overruled by system. Here, rather an “action selection” [50] in form of recommendations of a meal or product might be useful, however, the user might like to make the final confirmation based on additional context parameters not known to the system. Still, in both application domains, user interaction in the form of feedback after system usage is of value to further learn about the user preferences and adapt in future cases.

Improved intelligibility

The reasoning of (AI-driven) automated services and their underlying data processing are complex and hard to understand for non-expert users. This situation is worsened by systems involving a multitude of automated appliances. An example is the smart home domain, where users are able to combine virtually an unlimited number of smart gadgets which exchange data. An overall understanding of the mechanisms and reasoning of such smart systems seem a crucial factor for their adoption by non-expert users and for effective interactions with them. While the research area of “Explainable AI” aims at improving the transparency of AI-driven processes and at making them more comprehensible, recently introduced approaches mainly address expert users and are still too complex to support non-professional users.

In order to design for intelligible automated systems for everyday life, we argue that the users’ mental models of automated everyday objects need to be investigated. In related work, Tabassum et al. [67] explored users’ mental models of the data practices of their smart home devices. Today, many smart home gadgets and household robots come with mobile applications for their configuration. It seems worth exploring how such apps can provide meaningful supportive information during usage beyond simple status indicators. For example, augmented reality views might provide contextual information and guidance for automated systems with a physical presence appropriate for non-experts.

Maintained integrity

With regard to many of the above aspects, important questions on integrity and trust arise. This concerns trust into an external entity or body that may have too much insight or control into one’s own sphere of action but also trust in the automation system capabilities themselves. Researchers have pointed out the relevance of appropriate trust calibration to avert misuse, disuse, or abuse of an automated system [26, 49]. In case of an inappropriate level of trust, people may not accept or inadequately rely on autonomous systems in their everyday life [34]. Trust calibration is not an easy task for users, and therefore previous studies in professional have recommended that training on the task may be required for interacting with limited automation systems [41]. One central question is whether the factors affecting trust in professional contexts are similar as those in everyday automation. One of the important differences is that in everyday, contexts training is usually not considered to be important or practical. Furthermore, while users are often obliged to use certain technologies in professional settings, they can take advantage of opt-out opportunities in private life.

Opening the view across application domains

The above described trend towards everyday automation implies that it encompasses a broad spectrum of our experiences. Consequently, research on human-automation interaction will also have to continue to expand both regarding the breadth of investigated areas and the depth of knowledge about them [28]. In this regard, a key challenge is to find ways how a comprehensive and consistent experience for users can be enabled, without having to switch to, learn, and monitor many different paradigms. A prime example is the smart home, where the best way for managing the multitude of entertainment, security, energy, and housekeeping devices has not yet been found. For some aspects of automation experience, such as trust formation, an integrative perspective has been taken on by some scholars (see especially [34]).

However, there is still much potential with transferring principles drawn from professional usage scenarios (such as the handling of large aircraft automation systems) to less intensively trained usage (such as light aircrafts with amateur pilots). Furthermore, the question arises which design patterns proven successful in already intensively researched domains and can be applied to other application fields. Such efforts of holistic consideration will have to extend previous approaches of categorization into levels of automation across domains [15, 30, 60, 62, 70]. Some have started to implement these levels of automation in applications domains, such as in manufacturing [22], and more widely in the field of automated transport. Table 1 provides a brief and non-exhaustive overview of the status of everyday automation in some of the relevant application domains, and it lists respective topics and issues being focused in current investigations.

Table 1 Overview of application domains of everyday automation, the respective status of everyday automation, and the everyday automation aspects focused so far in research

Towards a design space for everyday automation experience

Exploring and structuring the manifold facets of everyday automation seem necessary in order to provide a common basis for analyzing and discussing existing and upcoming appearances of everyday automation, for identifying research gaps as well as for facilitating cross-domain innovation and thus for inspiring future automated systems for everyday life. To this end, Shneiderman [63] recently presented a human-centered artificial intelligence framework that categorizes system types into a space along the levels of human and system control. He concludes that one should design both for high levels of human control and of system control. Bongard et al. [6] propose the definition of an “Automation Space,” a design space for everyday automation. Based on existing related categorizations [e.g. 49] and a structured review of scientific studies on automated systems in everyday life and its subsequent qualitative content analysis, the researchers identified eight core dimensions for categorizing ubiquitous automated systems: the presence of the system, the functionality, the context, the application domain, the type of automated task, the available interaction modalities, the required user expertise, and the level of automation. Figure 2 depicts these dimensions and their potential characteristics: e.g., the presence of the system might be either virtual or physical, and application domains include shopping, transportation, home, and user interactions that might be possible through voice, gesture, etc. For visualizing this design space, the authors propose a so-called morphological box: this matrix presentation features all dimensions and their characteristics in rows; thus, each automated system can be represented by a vertical path through these rows, and each novel path might inspire a future ubiquitous automated system. Figure 2 contains two example systems from literature such as an automatic vacuum cleaner robot.

Fig. 2

“Automation Space,” a preliminary version of a design space for everyday automation with two examples of Bongard et al.

Summary and conclusions

In this introduction of the theme issue, we have presented eight special features of everyday automation experience and propose related promising research directions. We highlight that users get into contact with automation experience technology in the form of uninitiated encounters and that they are exposed to implicit ambient interactions. Living with such systems requires people to trust in calm, often unnoticeable, distributed system operation and adaptation, but they also may necessitate abrupt manual interventions and knowledgeable customization activities. We argue that, while important theoretical concepts have been proposed for related aspects, a more comprehensive conceptual framing is needed to grasp the qualities of everyday automation experience in their entirety and their various manifestations.

Of central importance is how to collect and analyze everyday user experience data, and thus we also need to come up with suitable methodologies to unpack and capture them. We juxtaposed interaction and the practice perspectives, and we state that both are needed to achieve meaningful insights. Building up on this broader empirical foundation, we highlight three avenues for design, which shall improve intelligibility, interventions, and integrity of everyday automation experience. Thereby, a broad and informed view should be adopted, which takes account of approaches and achievements in other application domains. Ideally, the achieved design approaches should feed into further instances of the above introduced preliminary design space that could help shape everyday automation experience into something desirable for the future.