Increasing the Value of XAI for Users: A Psychological Perspective

This paper summarizes the psychological insights and related design challenges that have emerged in the field of Explain - able AI (XAI). This summary is organized as a set of principles, some of which have recently been instantiated in XAI research. The primary aspects of implementation to which the principles refer are the design and evaluation stages of XAI system development, that is, principles concerning the design of explanations and the design of experiments for evaluating the performance of XAI systems. The principles can serve as guidance, to ensure that AI systems are human-centered and effectively assist people in solving difficult problems.


Introduction
The general objective of Explainable AI (XAI) is to develop methods that enable people to achieve a satisfying and useful understanding of an AI system's capabilities and vulnerabilities [see 70].This objective pertains to stakeholders such as policy makers, system integrators, and trainers, but it pertains especially to end-users.Understanding how to use an AI system makes it possible for people to act appropriately, • The need to settle the matter of "fragile credibility" or trust and reliance (e.g., users often prefer to pursue their own goals rather than follow the given advice), • The need to enable users to actively test the explanations and explore the boundary conditions of the AI, • The need to fully integrate a help system with the application, • The need to be able to deal with user misinterpretations of explanations, via a dialog, • The need to deal with "tangled errors" when one mistake leads to another and the situation becomes uncorrectable, • The need to reveal usefulness and usability challenges (e.g., help systems can trigger annoyance rather than help users).
The topic of XAI has ramped up quickly.Researchers have created Machine Learning systems for object recognition that "explain" their categorization of objects or images using saliency or "heat" maps showing what the AI system "is looking at."Researchers have created software agent systems that explain their choice of courses of action using such displays as histograms or matrices of probabilities [see 30].There has been a wave of review articles and a wave of attempts to taxonomize machine-generated explanations (qualities, formats, data types, purposes, etc.) [7,12,62,64,72,79].Many issues have emerged in this research.For instance, there have been lamentations about the ambiguity of key concepts including transparency, interpretability and explainability [2,6,8,15,46,47,53,61].Interpretability tools are advertised as exposing machine learning interpretability algorithms, and thereby explain the output of machine learning systems.But the "explanations" that these systems provide are code-intensive representations of the results from statistical or game-theoretic modeling.They serve more to justify system architectures to other computer scientists [1] than to explain AI systems in ways that make sense to people generally [57].Recent research has begun to address the issue of experimental adequacy and rigor [4,9,37,40,50,54,67,73].
A program established in 2018 by the US Defense Advanced Research Projects agency was a major impetus for the field (although the concept of XAI pre-dates it [see 69]).What has been accomplished in the field of XAI most recently, what assumptions have been revealed, and what remains to be done in order to ensure that AI systems are "human-centered"?XAI research has identified some principles that should be appreciated [12,64].These represent the findings of a broad range of research and scholarship at the nexus of AI, automation, and human-machine systems [31,60].We highlight the ways in which these principles have been invoked and in some cases have been integrated into AI research.

Organization
This article is organized as a set of twelve principles, some of which have recently been instantiated in XAI systems.The purpose of the principles is to express guidance for XAI system developers, not to solve immediate problems of the design of explanations, displays or experimental evaluations.The principles might be thought of as cautionary tales, or more strongly as necessary considerations.Details concerning the design of explanations and evaluations, and that are in accordance with the Principles, are presented in [42,47,66].
The Principles fall into these groupings: • Cognition as a determiner of what it means to explain, • Orchestration of the process of explaining, • Designing explanations, • Designing empirical evaluations.

The Cognitive Perspective: What does it
Mean to Explain?

Principle 1: All Explanation Involves Self-Explanation
This principle is about how XAI system developers should think about the cognitive process of explanation.The principle might be deemed the Golden Rule of XAI systems: Explain unto others in such a way as to help them explain to themselves.From the moment the user receives their first instruction on how to use an AI system, there is an active process of self-explanation.It may be brief and superficial, resulting in knowledge that is fragmentary and inconsistent [22] or the self-explanation process may be drawn-out and deliberative.Sometimes users/learners simply do not care "how it works" and just want to get on with their task or job.But self-explanation is often a highly motivated desire to understand.This assertion derives from psychological experimentation [13] and empirical research on how people explain complex systems to other people [51,52].
Explanations can help learners form and refine their mental model of the AI and the task.But the achievement of an understanding is not just the ingestion of information.Self-explaining involves changing preconceptions by both assimilation and accommodation, to use Piagetian terminology.Thus, it can be useful for the learner to see what happens when the AI system fails, or is at its boundary conditions.
Failure cases can be a signal of a model mismatch [38], especially if the AI system fails in ways that humans would never fail (e.g., confusing a turtle with a rifle) [14,41].
The self-explanation process can be seen in research that has used cognitive interviews to assess explanatory efficacy [5].A number of researchers have noted that surprising events or violations of expectations trigger a need for an explanation.This has been considered in psychology, especially in research on curiosity [58].A model of self-explanation is emerging in XAI based on the psychological investigation of explanatory reasoning [52] and XAI research on levels of intelligibility [56].
This first principle entails a reconsideration of the basic conceptual model that motivated some early XAI research, including the DARPA Explainable AI Program, which was a major impetus for the field in 2018-2021.This model is depicted in Fig. 1.This model approached the topic from a programmatic, computer science point of view.The explanation is generated and then is delivered, (ideally) to good effect.The model highlights the things that would have to be measured in an evaluation of an XAI system (the shaded nodes in Fig. 1): (1) Is the explanation good in the sense that it accords with the criteria that have been espoused in the literatures on explanatory reasoning (succinct, understandable, etc.)? (2) Is the explanation satisfying in the judgment of the users?(3) Do the users really understand the explanation?(4) Does performance improve because of the explanation?
This model expressed the premise that explanation of how the AI system works is presented early, at the point where the user is being instructed on the task and the tool.A further assumption was that subsequent human-computer interaction would involve the generation and presentation of specific machine decisions [62].These assumptions had a positive impact in that they highlighted the distinction between local (Why did it decide that?) and global (How does it work?)explanations, a distinction that many XAI researchers embraced [28,77].
In parallel with the computer science work, empirical analysis was conducted of a large corpus of cases where a person was the recipient of an explanation of how some sort of complex system works [52].Many of the cases illustrated how global explanations often include local information (e.g., instances that exemplify rules or principles), and local explanations often include some global information (e.g., a generalization over a class of instances).Thus, in actual explanation events, the global-local distinction gets blurry.A second revelation from the empirical research was that the "spoon feeding" paradigm expressed in Fig. 1 is blind to the fact that users engage in a motivated, deliberative attempt to make sense of the AI system and any explanatory material that may be presented.This is diagrammed in Fig. 2. In this psychological model, explanatory systems benefit by providing information that empowers users to self-explain, rather than just delivering some sort of representation of the output of an algorithm, a representation that is believed to be adequate as an explanation.
The Fig. 1 model is Programmatic, that is, it was a description of an approach to evaluation methodology (i.e., what to measure and when to measure it).The emphasis Fig. 1 2018 model of the XAI process, from the perspective of computer science formal) information.XAI researchers have recognized the value of narrative explanations that describe causal relations, and the value of narrative explanations [31].
As was discovered in the research on Intelligent Tutoring Systems (ITS), an XAI system's explanation capabilities need to be designed on the basis of a pedagogical model-an instantiated model of the instructional process of structured interaction, which in turn is based on a cognitive model of learning [18].XAI systems might attempt to identify the user's current understanding (so that it can better predict how to transform this knowledge), and support the information that will help make these transformations.This may be asking a lot of XAI system developers who might not think of XAI as a form of ITS.But for XAI to work as intended, it needs to accord with the key findings from ITS research [see 18].Recent research has demonstrated that XAI systems can support the user in a process of active exploration [37].

Principle 3: Explanation Relates to Trust, but Trust must be Considered a Dynamic Process, not a State
This Principle is about how XAI system developers should conceive of the relation between explanation and trust/ reliance in an AI system.The notional model of the XAI process (Fig. 1, above) assumed that good explanation will result in trust.But trust is not a single state that develops to some level of calibration.In work contexts in which humans rely on computational technologies, people always feel some mixture of justified and unjustified trust and justified and unjustified mistrust.These attitudes are in constant flux and rarely develop in a progression to some ideal and stable point [4,35,39,55].Trust can come and go in a flash.When was on the performance of the user, given that the user benefits from the machine-generated explanations.As such, the model made limiting assumptions about the psychological aspects of explanation.While the measurement nodes of Fig. 1 might be ported into the Fig. 2 model, the Fig. 2 psychological model refers to the performance of the human-XAI system.Additionally, the model situates performance in a loop with the process of self-explanation, which can be thought of as the user's refinement of their mental model.

Principle No. 2: Explanation is an Exploratory Activity
Following from the notion of progressive refinement that is illustrated in Fig. 2, Principle 2 is also about how XAI system developers should think about the process of explaining.Most taxonomies of types of explanations or features of explanations are predicated on the view that the property of being an explanation is a property of text, visualizations, etc.The mantram Explanation is Exploration is another lesson from the XAI research.It harkens back to John Dewey's theory of inquiry [70].Explanations should not force learners to adopt the AI's model, a model that is not their own, and is arguably not human, and can only be assimilated with difficulty.Explanations can actually impair learning if they do not encourage the learner to discover patterns.Explanations should enable people to diagnose situations, predict or anticipate the future and justify decisions or actions.Explanations should enable people to notice apparent inconsistencies, and the XAI system should enable the user to restore consistency between their mental model and empirical reality.Thus, the explanation process must involve the exchange of meaningful (and not just computationally AI decision invalid?XAI systems might benefit from considering the long-term interaction with users, even in simple ways like recognizing that once learned, an explanation may not need to be given again unless something important has changed.Such design decisions are contextual (see Principle 7) and specific to individuals (see Principle 10).
As was said above, XAI research initially assumed that the property of being an explanation is a property of text, images, etc. XAI research initially assumed that explanation only involves providing an explanation to the user, on the assumption that the explanation is good and sufficient.XAI researchers are escaping these assumptions.Explaining depends on the ability of the human and the machine to interact.Explanation as collaboration is another prime lesson from XAI research [25,32,64].Some XAI systems have come to regard explaining as a two-way process in which the XAI program provides explanatory information, and the user advises the program in one way or another [28,44,49,75,82].Thus, explanations allowing for user input might lead to improvements of the computer model.More work has to be done to extend this basic idea.Explanations work better when people can interact with them.

Principle 5: Explanations are most Needed when there is a "Trigger."
Explanations are not needed all the time, but in some early XAI research it was assumed that explanations should always be presented.The displays created for some early XAI always included a field showing "reasons" why the AI system made its determination (e.g., a list of key features, or a "saliency map" for classifier systems).It remains unclear as to whether the persistence of an explanation display detracts from performance by virtue of the dedication of display real estate to information that is not always needed.This is often because users have seen something interesting or surprising or they have some sort of goal that they strain to achieve.Some triggers have been mentioned above, but Table 1 lists those that have been referenced in work on XAI systems [see 62].Furthermore, the triggers are manifest in interviews with stakeholders about their explanation requirements [38].That said, not all XAI researchers may see it as the goal of their XAI systems to provide explanatory answers to some of these trigger questions.
Advances in AI (and XAI) will come when systems begin to understand and anticipate situations that are likely to engender surprise and violate user expectations [69].
the AI fails in a way that a human would never fail, reliance can collapse.

Principle 4: Exploration is Interaction; it is never a "One-Off"
This Principle is an extension of Principle 2. For many XAI systems, explaining has to be thought of as a process that extends over multiple uses and interactions.Explaining could extend over multiple sessions if the user wants to check back on a previous case.Or, the user might want to test the coverage of the AI by presenting it various counterfactual test cases.
Explanation cannot be a one-off if only because the AI system is a machine learning system which itself can continually evolve.Especially for AI systems that learn or are applied in dynamic contexts, users often need repeated explanations and re-explanations.How has the algorithm changed?How are AI system's decisions affected by the new data, and if the new data themselves are not valid, is the

Principle 8: Explanation Occurs in a Context, Carrying with it the Goals of the Work System
Explanation is an interaction among the user, the XAI system, and their activity in a task context.Different explanations support different information needs for different tasks within a broader set of work goals.Explanation needs of the user are also related to role of the user within that broader work context [38].Stakeholders sometimes need access to others (e.g., trusted engineers, trusted vendors) in order for them to be able to develop satisfying mental models of AI systems.Trainers need to know how the AI system fails and how it misleads as much as they need to know how it works.Some stakeholders need to develop an understanding that enables them to explain the AI to someone else and not just satisfy their own sensemaking requirements.

Principle 9: Explanations Tend to Work Better if they Include Demonstrations of Differences and Contrasts
This Principle is about the design of machine-generated explanations.A central lesson of XAI research is the utility of contrastive and counterfactual explanations in understanding the boundary conditions of a system [5,25,28,60].XAI researchers have begun to consider the importance of interactive explanations and counterfactual or contrastive explanations.These goals can reflect curiosity, or a need to know: What did the AI system just do?Why didn't the AI system do z?What would the AI system have done if x had been different?Explanation of "why something is what it is" entails an explanation of "why it is not something else." In other words, explanation and contrastive reasoning are co-implicative.The importance of counterfactuals to explainability and to the automatic generation of explanations has been noted by XAI researchers [3,19,67,79] (see [48] for a review).However, contrasts and counterfactuals might only be useful if the user already has in mind a mental model of typical behavior.That is, the user needs to understand what "right" is before they can understand what "not-right" is or what "wrong" is.Pedagogical use of examples requires understanding a failure in order to produce benefit-otherwise examples may end up simply frustrating the user and fostering distrust because of the salience of failures.
It has been argued that counterfactuals lack explanatory value because they do not provide a causal model [80].This claim hinges on the assumption that an explanation is a resolution: an explanation has served its purpose once it has been delivered and understood.However, the delivery

Principle 6: Explanation Occurs in a Context, Carrying with it the Needs of the User
Some people prefer explanations that refer to single, necessary, or "focal" causes and that are both simple and yet broadly applicable.Some people will believe explanations to be good even when they contain flaws or gaps in reasoning.Some people are not satisfied with simple, superficial explanations.Some people are more deliberative and reflective in their explanatory reasoning.Some people are eager to learn how the system works.Expert users can prefer less detail, presumably because they have enough knowledge about causal relationships in the domain that they can link together the focal causes.Non-expert users may need more detailed information for their sensemaking.
Explanations can be desired for a number of reasons, that depend on the goals of the user [36,75].This includes the need to self-explain, discussed above, but also pragmatic, ask-relevant goals: How will this AI system improve my performance?How much effort does it take to use this AI system?How can I trust it?How do I recover from the AI system's mistakes?How will I know when there has been an anomaly or a failure?How can I repair the AI system?For want of satisfactory answers to these sorts of questions, the user might resort to trying a workaround.

Principle 7: People do not Necessarily Engage with Explainability Tools
It should not be taken for granted that people will engage with XAI.As outlined in Fig. 1 above, there was an assumption of the sequence: 'make decision' → 'explain decision' → 'understand' that people will use information from XAI tools to understand the AI's decisions.However, recent research suggests that this assumption should be questioned.For example, in a study involving judgments about nutrition [27], explanations were about the AI system's recommendations, but hid the actual recommendations themselves.The research showed that this resulted in better decisions and better incidental learning on their participants compared to giving just recommendations, and compared to giving both recommendations and explanations.The conclusion was that withholding decisions leads to higher cognitive engagement and therefore better decisions.
as an evaluation of the interactive performance of the user and the XAI system [64].

Principle 11: The Evaluation of Human-XAI Task Performance should Rely on Representative Users and Domain Experts
Also by implication from Principle 8, XAI evaluation should involve representative users as research participants.Workcentered design principles suggest involving representative intended beneficiaries of the AI system early and throughout the system development process [17,20,21,29,36].Early evaluation of explainability interventions may rely upon non-experts on simplified tasks, but the findings may not transfer to the real context.Furthermore, a work-centered approach necessarily involves domain experts in the system design and development activity, from the beginning and throughout the development process-not just a one-off at the initial design stage or some sort of culminating evaluation.Recent XAI research attests to this [27,75,78].

Principle 12: Research Evaluating XAI Systems Needs to "Get Inside the Heads" of the Users
This Principle is about methodology in the evaluation of XAI systems.XAI researchers have come to recognize the value of cognitive task analysis to reveal learners' mental models [23,24,28,33,45,73,75,78]. Valuable findings have come from post-experimental structured interviews, asking the participants such questions as What problems did you find? and Why do you think the AI system did that?[5,59].Results are the most informative about how machinegenerated explanations influence user reasoning, that is, the interview data help the developers make sense of the performance data.Presentations on XAI projects that include quotations from users in post-experimental inquiry demonstrate the kind of awareness that XAI evaluation research requires-awareness of the user's reasoning as they actively try to understand the AI system. of an explanation is not a terminus in the sensemaking process.Indeed, counterfactuals have a very important purpose: They show the user that exploration is possible, and they show how exploration can be conducted.This is powerful, as it supports exploration of situations when the AI is operating at the boundaries of its competence envelope-instances that fall inside a class but nearly do not, and instances that do not fall outside the class but nearly do.Such cases show when a small change to a case makes a difference to the categorization.

Principle 10: The Evaluation of Human-XAI Task Performance should Rely on Ecologically Valid Tasks
It might be taken for granted that XAI systems need to support task-essential capabilities that people cannot exercise well, or at all, without computational aids [71,75].Recent XAI research has shown that an XAI system is not interesting for people to use, or try to understand, if the user is always more skilled than the AI [74,75,77].The task that is set before a human-AI work system has to be one that presents a genuine challenge.But the evaluation of XAI systems often involves human judgment of the acceptability or usefulness of algorithms.Such bench-testing may provide useful early guidance, but the benefits may not occur in situ.By implication from Principle 8, XAI evaluation should maintain the context of actual work.It is inadvisable to develop an explanation system by just looking at a convenient proxy task (e.g., bird classification) [9].While proxy tasks may arguably be ecologically relevant, explanations have to relate the AI tool to the context of actual tasks and goals tasks [4,9,42].
Most real-world applications of AI systems involve work situations bearing multiple tasks and multiple goals.Explanations have to relate the AI tool to the user's knowledge in the context of those goals and tasks.Much XAI research has assumed a one-person, one-task problem situation.Failure to build ecological relevance into an XAI system makes research easy because it sidesteps the complexity of actual work settings, in which a group of people using an AI tool may have different roles and tasks, with conflicting constraints to satisfy.Additionally, the ecologically sparse settings and tasks used in XAI system design and evaluation assume that the user of the AI will only be interacting with one program.In most real-world settings, workers juggle multiple computational systems.Furthermore, the user and the AI system act interdependently [43].Thus, the evaluation of the performance of an XAI system is to be thought of interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of AFRL or the U.S. Government.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material.If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.

Discussion
The Principles are an attempt to capture the key ideas and challenges that have recently emerged, and in particular in the work that was conducted with the support of the DARPA XAI Program.Of course, more research, development and evaluation activity needs to be done to deepen and extend the application of the Principles.
The Principles speak to the value of cross-disciplinary synthesis and collaboration.The evaluation of an XAI system, or any AI system, is essentially a large-scale psychological experiment.Rigor is necessary to justify the investment of time and effort required to develop AI systems.The development and deployment of useful and useable XAI systems requires: • A multi-disciplinary collaboration (philosophy, psychology, and education as well as computer science); • A design process that embeds the human-XAI system in a realistic and relevant task context; • A view that the good XAI system needs to benefit the machine as well as the human.
The Principles and their applications in XAI express the perspectives of experimental psychology and empirical philosophy.XAI research has certainly also resulted in insights from the perspective of computer science.For example, there has been a recognition of the idea that feedback from the user might be used to improve the computational model, that is, the benefits of explanatory reasoning work in both directions [37].User feedback to an XAI system can improve the Machine Learning model that needs to be explained/understood.As XAI research matures there will no doubt be additional achievements and insights.

Acknowledgement and Disclaimer
The Authors would like to thank Timothy Cullen (Col., US Army, Ret.) for his consultation on this project.The Authors would like to thank the anonymous Reviewers for their many helpful comments on the submission.This material is approved for public release.Distribution is unlimited.This material is based on research sponsored by the Air Force Research Lab (AFRL) under agreement number FA8650-17-2-7711 and by Australian Research Council (ARC) Discovery Grant DP190103414: Explanation in Artificial Intelligence: A Human-Centered Approach.The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon.The views and conclusions contained herein are those of the authors and should not be

Fig. 2
Fig. 2 Psychological Model of the XAI process

Table 1 "
Triggers" of the need for an explanation, mapped onto the