Keywords

1 Introduction

Advances in Artificial Intelligence (AI) and Information & Communication Technology (ICT) have been manifested in various automated systems, such as sensing technology, machine learning modules, Internet of Things, conversational agents and cognitive robotics. The embodiment in artificial, virtual or physical, agents enables automation to evolve as a member of mixed human-agent teams. A major challenge is to combine, automate and embody the information processes in such a way that agents really become full-fledged team-members, complementing and collaborating with the human team-members.

The coordination and collaboration in human-agent teamwork requires intelligent reactive and anticipatory behaviors of the agents. More specifically, they require a shared knowledge representation, methods to comply with policies and agreements for responsible teamwork, and the learning and effectuation of successful patterns of joint activities (e.g., [1, 11, 18, 25]). In addition, according to the fifth challenge of Klein et al. [17], the agent should be able to make pertinent aspects of their status and intentions obvious to their teammates. In our view, this means that there is a need for mutual human-agent exchange of the reasons and foundations of actions, and that the agent should provide explanations for adequate human understanding and appreciation (incl. trust) of its performances.

Symbolic AI, such as BDI-agents with built-in Beliefs, Desires and Intentions based on folk psychology, provides explicit opportunities for the generation of explanations that are understandable and useful for the human team-member. For example, there have been developed explanation methods for fire-fighting teams [10], and tactical air combat teams [12, 28]. However, intelligent agents will be embodying more-and-more sub-symbolic machine learning methods for which it is far from clear how to derive an explanation logic for the human-agent collaboration.

So, for the envisioned human-agent teamwork, we need to develop methods for explainable AI (XAI) for adequate human understanding and appreciation (including trust) of symbolic and sub-symbolic agent performance. Such explanation should support human-in-the-loop (co-)learning of the human-agent teams.

To meet this challenge, our research focuses on the development of complementary explanation methods for agents using a holistic approach which covers all three phases of an explanation (see Fig. 1).

Fig. 1.
figure 1

Different phases of an explanation (ε refers to “the concept of explanation”, cf. [29])

Full size image

The first phase concerns \( \varepsilon \)-generation. In this phase, we distinguish between Perceptual XAI and Cognitive XAI. Perceptual XAI aims to explain the perceptual foundation of the agent behavior, and is usually connected to the sub-symbolic reasoning parts of an AI architecture. This paper discusses two types of perceptual XAI: The construction of an Intuitive Confidence Measure and the classification of facts and foils for contrastive explanations [30,31,32]. The second type is cognitive XAI, which can explain why a certain action was chosen (e.g., by relating them to goals or beliefs; [13,14,15]). This part interacts with the sub-symbolic part of the AI behavior engine to ground its belief base. The construction of goal-based deliberative explanations is an example of cognitive XAI.

The second phase in the explanation process is \( \varepsilon \)-communication. This process is characterized by the form of explanation (e.g. textual, or using dedicated images as proposed by Beller et al. [2]), and the content of the explanation (i.e. what is communicated). We will propose the use of ontologies to standardize the language which is used to provide explanations [27, 29]. For the design of the form of the explanations (which need to be adaptive and interactive), we are developing interaction design patterns (cf., [23, 33, 34]).

The third phase of explanation, \( \varepsilon \)-reception, concerns how well the human understands the explanation. With respect to XAI reception, some user studies (e.g., [22]) have been conducted, but there is a significant lack of empirical research with actual human task performers who need explanations in realistic human-agent settings (e.g., [20, 21]). This paper summarizes some first results of our evaluations.

We are developing a general, perceptual-cognitive, explanation framework that can be applied and refined in different projects, crossing domains. In this way, we can provide technological progress and empirical grounding, building general models and methods for explanation that can be instantiated in different domains. First prototypes have been developed and tested with end-users in transportation, healthcare and defense domains. Results show that human needs and preferences for explanations depend on their individual characteristics (e.g., age and experience) and on the operational context. The paper will present an overview of the explanation framework and the first prototype designs and evaluation outcomes.

2 Perceptual XAI

We propose two methods for perceptual XAI: (1) The construction of an Intuitive Confidence Measure (ICM) and (2) the identification of the counterfactual reference (i.e., the foil that is set against the fact in a contrastive explanation).

2.1 Intuitive Confidence Measure

Waa et al. [30, 31] developed a generic confidence or certainty measure for agent’s machine learning (ML) that can be understood by the human. It is model-agnostic, i.e., the measure can be used for any machine learning model as it depends solely on the input and output of a trained model and future feedback about that output. The confidence (or uncertainty) reflects machine learning model’s expected performance on a single decision or classification. Our Intuitive Confidence Measure (ICM) should be easily understood by humans without ML-knowledge, and behave in a predictable way. We designed ICM to be intuitive by basing it on the notion of similarity and previous experiences: Previous experiences with the ML model’s performance directly influence the confidence of a new output, and this influence is based on how similar those past data points are to the new data point (similarity can be represented as a distance in an n-dimensional space). The ICM is applicable to any (semi-)supervised machine learning model, that is either trained online or offline, by treating it as a black box. One of the use cases is an agent that predicts if its remote teammate is required to be at location in the near future (e.g. at a work-desk in a dynamic positioning ship). The agent can explain how likely it is that this single prediction (output) is correct [31].

We have applied the ICM explanation method to the domain of monitoring dynamic positioning systems, i.e. highly automatic systems which aim to maintain a vessel’s position and heading using dedicated propellers and thrusters [6]. Occasionally, the system requires human intervention, for example to warn the user about potential problems that are predicted by a machine learning model. Because these predictions may be wrong, the user must have an appropriate level of trust in this type of advice. We used the intuitive confidence measure to provide this kind of advice. The message below is an example of this type of \( \varepsilon \)-communication, where the ICM measure in the advice is communicated to the user. Furthermore, the particular design pattern behind the message also allows extra information to be provided on request of the user (in the example on the type of changes, and the type of conditions) (Fig. 2).

Fig. 2.
figure 2

Advice accompanied by the ICM XAI method.

Full size image

To test the \( \varepsilon \)-reception of the ICM XAI method, we performed a user experiment [31], where we compared ICM with two different methods of computing confidence measures. The results confirmed the expectation that the intuitiveness of the XAI method is an important factor to consider when designing explainable smart systems. The user study showed large individual variations concerning user’s interest in a confidence value and its utility in different situations. In comparison with other confidence measures, the ICM proves to be relatively easy to understand although not always preferred. The main reason for this was that the users, without ML-knowledge, attributed desired properties to other measures they did not fully understand. Whereas, with the ICM they understood the measure well enough to even identify its weaknesses; the users quickly thought to understand complex methods while, in fact, they did not (see [31], for more details of the user study).

2.2 Counterfactual Reference

For humans, explanations are most often contrastive, i.e., they refer to specific counterfactual cases, called foils [20]. When asking why an agent made a specific prediction or decision, humans would like to be informed about the contrast against the prediction or decision. So, an explanation should compare agent’s output to an alternative counterfactual output (i.e., the foil). In other words, a good explanation considers both the fact (output) and the foil (alternative output).

Therefore, the generic PeCoX method for automated explanation extraction (i.e., \( \varepsilon \)-generation) includes the identification of the foil for a contrastive explanation. This method should be able to deal with the context-dependency by learning which foil matches with which input-output pair based on interaction feedback from the user (e.g., through one-shot learning in combination with high generalization). This method is an extension of past work that constructs an explanation with the help of a localized interpretable machine learning model, such as a decision tree, through error weighting based on the distance between the data point of interest and the rest [24]. This way, the explanations can become more focused with less redundant information as the ‘why’ question is answered in a more precise manner.

More specifically, the PeCoX foil identification method is inspired by the LIME method, which provides an algorithm that learns an interpretable model locally around the prediction and a method to explain models by presenting representative individual predictions [24]. Further, like the Intuitive Confidence Measure of Sect. 2.1, LIME is model-agnostic. In our framework, it is further important to classify the foils for the generation of adaptive explanations (i.e., attuned to a specific user group).

The content and format of the explanations should be well-tailored to the user group. Ontology design patterns are constructed (see Sect. 4) that specify the high-level feature set, such that feature-based explanations are easy to comprehend by the user. Furthermore, interaction design patterns are generated that specify the corresponding multimodal dialogues for contrastive explanations.

We are applying the contrastive explanation method to the domain of type1 diabetes mellitus (T1DM). T1DM is a chronic condition where insufficient insulin is produced by the pancreas, affecting blood glucose levels. Daily self-management is needed for a balanced glucose level, harmonizing insulin doses (via pen or pump), food intake (amount of carbohydrates) and activities (e.g. sport). High and low glucose levels may lead to a hypo or hyper, and on the long term to serious health complications. The Personal Assistant for a healthy Lifestyle (PALFootnote 1) project develops a system for children aged 8–14, their parents and health care professionals that advances child’s diabetes self-management. The PAL system comprises an Embodied Conversational Agent (ECA) robot and (mobile) avatar (Fig. 3), a set of mobile health (mHealth) applications (e.g., diabetes diary, educational quizzes), and dashboards for the caregivers (i.e., health care professionals and parents). All parts are interconnected with a shared knowledge-base and reasoning mechanism. health care professional, parent or child). Example contrastive explanations concerns PAL predictions that a child will not complete his or her self-management tasks. Such predictions should be explained to the child, parent and healthcare professional in different ways. Section 3 will provide more concrete explanation examples of the PAL system.

Fig. 3.
figure 3

Example of PAL quiz-question for learning to act on a low blood sugar level.

Full size image

3 Cognitive XAI

In this part of the framework we consider explanations from the intentional stance [5]. When taking the intentional stance, you assume the action is a consequence of the intentions of the agent performing the action. You then explain the action by giving the reasons for the underlying intention. An explanation like such typically consists of beliefs, goals, and/or emotions [4, 5, 7, 9, 19]. For example, a support agent that tells its user to eat vegetables every day might provide the following explanation: ‘I hope (emotion) that you will take my advice to eat vegetables every day because I want (goal) you to adopt a healthy lifestyle, and I think (belief) that you currently do not eat enough vegetables’. A method for explaining the reasons is being developed for a cognitive-affective (BDI-based) agent that updates its beliefs, goals and emotions based on events perceived in the environment [13, 14]. This agent can explain and justify its actions by communicating (a) the beliefs that underpin the actions, (b) the goals that inform the human of the agent’s desired state when acting, and (c) the emotions that trigger or shape the actions.

3.1 Goal- and Belief-Based Explanations

The reasoning of a BDI based agent often consists of many beliefs and goals. If we use all of those for the explanation then this might overflow the user with information, which would thus not help us to make the behaviour intelligible [16]. Current work in XAI for artificial agents has thus focused on finding guidelines for which beliefs and goals are most valuable to use in an explanation towards end-users [3, 10]. This work confirms that a good explanation is short and thus contains few beliefs and goals. Similar to the evaluation of the Intuitive Confidence Measure in Sect. 2.1, a user study showed individual differences in explanation preferences. Particularly, adults showed a stronger preference than children for goals over beliefs in the explanations [14]. So, individual characteristics of the user must be taken into account when choosing which beliefs and goals should be selected as content for the explanation (Fig. 4).

Fig. 4.
figure 4

Example of goal-based explanation in PAL.

Full size image

3.2 Emotion-Based Explanations

Many artificial agents use computational modelling of emotion (based on cognitive appraisal theory) for their behaviour [15]. Here it is proposed that computational modelling of emotion can enhance XAI in several ways. The previous section mentioned the difficulty of selecting beliefs and goals as content for the explanation. The first use of emotions in XAI is as heuristic to identify the most important beliefs and goals for explanation. For example, the goal with the strongest influence for computing the desirability of an event can be used to explain the action the agent did after perceiving the event. The second use of emotions in XAI addresses that humans often use emotions when explaining behaviour [7]. The simulated emotions can be used to provide or enrich the action explanation (e.g., I was disappointed and therefore stopped my action). The third use addresses that emotions themselves can require to be explained (e.g., I was disappointed, because I had to do the same task over and over again) (Fig. 5).

Fig. 5.
figure 5

Example of emotion-based explanation in PAL.

Full size image

4 Ontology and Interaction Design Patterns

In our research and development projects, human-agent teams are modeled as joint cognitive systems, in which ontologies define the knowledge structures of these systems (i.e., the concepts, with their relationships, which underpin the cognitive processes of the teamwork). The explicit modeling of key concepts in explanations, enhances knowledge sharing (cf., [27]). These ontological models can be specified at different levels of generality [26]. The PeCoX top-level ontologies model general concepts that are domain independent, including explanation and human factors concepts (such as confidence and emotion, respectively). Specific domain and task aspects are captured in corresponding (lower level) ontological sub-models. For the construction of generic adaptive explanations, the Perceptual-Cognitive XAI framework is providing an extendable PeCoX-ontology in the form of Ontology Design Patterns (ODP, [8, 29]).

A formal definition of an explanation, like ODP, should specify its components, their roles and their interactions. Two ontologies provide starting points for such a definition. First, Su et al. [27] present an explanation ontology for constructing explanations for two agents that need to come to a partial shared understanding. They distinguish two knowledge types for agents that are engaged into an explanation process: (1) The domain knowledge entails representations of the to-be-explained concepts with the explanation parameters, and (2) the explanation knowledge entails the concepts that describe the explanation process and the permitted interactions in this process. Second, Tiddi et al. [29] studied approaches of Cognitive Sciences to model explanations with the instantiations in each of the analyzed disciplines. Their ontology intends to provide an abstract description which can be applicable to any context where an agent automatically produces explanations.

The ontologies of Su et al. [27] and Tiddi et al. [29] do insufficiently address (1) the perceptual XAI foundation of \( \varepsilon \)-generation, and (2) the situated human needs for explanation, i.e., the \( \varepsilon \)-reception (see Fig. 1). Section 2 describes the perceptual XAI concepts that are being specified an included in the PeCoX ontology. For the \( \varepsilon \)-reception, the social sciences provide theories and “models” for specific human information processes, for example, on the perception and activation of intentional and affective behaviors [20]. We are focusing and formalizing these “sub-models” into ODPs that support the predictions of \( \varepsilon \)-reception outcomes.

Whereas OPDs are used to support the design and reasoning of the communication processes and content of explanations, we use Interaction Design Patterns (IDP) for the specification of the form or shape of this communication [23, 33, 34].

5 Conclusions

This paper presented an integrative development approach for explainable AI in human-agent teams, addressing the perceptual and cognitive foundations of an agent’s behavior during the explanation generation, communication and reception. The proposed PeCoX framework is being developed and tested in the domains of transport, health-care and defense. This framework distinguishes between perceptual and cognitive explanations. On a technological side it provides tools for generating explanations from perceptual components (often implemented using sub-symbolic or connectionist approaches, such as neural nets), and cognitive components (often implemented using symbolic approaches, such as rule-based and ontology-based knowledge systems). On a human-side it provides tools for understanding the different types of explanatory support that a human would want in different contexts on both a cognitive as well as a perceptual level. Ontology Design Patterns are being constructed for the reasoning and communication of explanations, whereas Interaction Design Patterns are being constructed for the shaping of the adaptive, multimodal communications.

First results of the PeCoX framework development were acquired in the domains of transport, health-care and defense. At the perceptual level, first, the Intuitive Confidence Measure (ICM) proves to be a good candidate to enhance human’s understanding of the perceptual and cognitive foundation of agent’s behavior in a dynamic position ship. The ICM-evaluation showed the need to attune the explanations to the context and the biases of human’s perception of their own understanding. The participants in the evaluation of different confidence measures quickly thought to understand complex methods while, in fact, they did not. The explanation should be formulated in a “human-aware” way that minimizes the risk for such misbeliefs. Furthermore, the “foil” identification seems to be an effective method to generate desired contrastive explanations and situate them in a relevant context.

At the cognitive level, the goal, belief and emotion models provide the analytical knowledge foundation of the explanations. Similarly to the “ICM evaluations” at the perceptual level, the cognitive evaluations showed a need for further adaptation of explanations to the momentary context (e.g., age and role of the user). The user studies provide the empirical foundation of the explanation adaptation (i.e., the harmonization of belief-, goal- and emotion communication to the momentary user state and context).

It should be noted that, for effective and efficient explainable AI, a balance must be found between technological possibilities (which types of explanations can be provided by the underlying control logic of the autonomous system?), and user needs (given the current context, which type of explanations must be given for the human to establish an appropriate level of trust in the system?). This might involve making the technology better explainable, or accepting that the user cannot be explained in every aspect, and trying to mitigate possible negative consequences.