After training as a psychologist with a specialization in assessment, in 2010, I began to serve as an independent reviewer of disability documentation. I was initially surprised to see that some psychological evaluation reports were written from a clear position of advocacy. In part, this was evident from the tone and style; some evaluators not only stated conclusions and recommendations but also made demands and pleas. Some reports also made clear errors in reasoning, presented evidence in a one-sided manner, or ignored accepted standards for interpreting data, all in an apparent effort to assist clients. In the present paper, I consider why these types of advocacy exist, why it is problematic, and what we can do about it. I start by acknowledging that advocacy for clients is a natural motive for psychologists and sometimes even an official part of the job. I then discuss two different mindsets that psychologists can adopt when conducting evaluations, one that is motivated to reach a certain conclusion, and one that is more open to following the assessment data wherever they lead. I conclude with recommendations for how to fit appropriate, ethical, limited forms of advocacy into psychologists’ work.

The Psychologist as Advocate

In November 2019, the New York Times reported the story of Monifa Cannady, a single mother raising two young sons (one with autism) in New York City after a period of homelessness spent living in the subway system (Coleman, 2019). It’s an emotionally moving story, but it interested me for an additional reason: Ms. Cannady was trying to obtain employment as a teacher’s aide, but she had failed the exam required for the job. The article mentioned that Ms. Cannady was working with a social worker who was trying to help her get accommodations for a learning disability, so that she could retake the exam. As the reporter noted, “For Ms. Cannady, passing that exam is vital — to getting off Social Security, to becoming more financially stable, to doing more for Elijah and Aiden.” Imagine that a client in Ms. Cannady’s situation comes to you for a diagnostic evaluation, but imagine further that the evaluation data do not support the presence of a learning disability. All of the client’s academic skills are in the average range, when the diagnostic criteria clearly state that there must be evidence of below-average academic skills (American Psychiatric Association, 2013). There is no evidence of any other disorders either. Should you find some way to make a diagnosis and recommend accommodations, to help the client pass the exam? A psychologist would need to have a heart of stone not to be tempted.

Since Ms. Cannady’s story was covered, a major world event occurred: the COVID-19 pandemic, which has brought a rise in mental health problems (e.g., Hawes et al., 2021). Recently, I saw a post on a listserv for disability services professionals at colleges. A student with an ADHD diagnosis had requested a testing accommodation: permission to listen to a soundtrack of thunderstorm noise on their phone while taking an exam. Listserv members replied with thoughts about whether this was an appropriate ADHD accommodation, whether it was truly needed for access to tests, and whether the student having access to their own phone raised test security issues. Then one member responded that she saw the request as a mental health accommodation rather than an ADHD accommodation. She continued: “Anything which may help mitigate the current mental health crisis, I am going to advocate for. I've had students observed on camera cutting and breaking down too many times. I want to make a difference for those students, bottom line.” Although no information had been presented regarding the level of distress or impairment of the specific student requesting the accommodation, it is easy to sympathize with the perspective of this listserv member. Who would want to be in the position of saying “no” to a student who wants to make taking their tests a bit easier during a stressful pandemic? After all, even before the pandemic, research had found repeatedly that teams in K-12 schools consider students’ anxiety, comfort, and self-esteem as primary factors when making testing accommodation decisions (Crawford & Ketterlin-Geller, 2013; Rickey, 2005). If you, as a psychologist, were advising a disability services professional regarding a student who had an ADHD diagnosis and was in distress over needing to take exams without thunderstorm noise, what should you do?

These two cases raise the topic of this paper: the tension between advocacy and objectivity in psychoeducational assessment. Advocacy is often considered a fundamental skill and activity of psychologists. Defined most broadly as “a process of informing and assisting decision makers” (Lating et al., 2009, p. 106), advocacy has been especially promoted in recent years and specifically associated with advocating on behalf of marginalized populations (e.g., Hailes et al., 2021). Advocacy includes work at the level of systems and public policy, but the American Psychological Association (2011) includes other work as well; its most recent guidelines for psychologist competencies define advocacy as “Actions targeting the impact of social, political, economic or cultural factors to promote change at the individual (client), institutional, and/or systems level” (p. 17).

Client-level advocacy is often done by providing decision-makers with information about a client who the psychologist has evaluated or is working with. For instance, to help a client access a substance abuse rehabilitation program with limited spots, a psychologist might describe the client as having a high motivation to change and no past relapses. To assist a client who is charged with a crime, the psychologist may characterize the client as having poor decision-making skills or even as being “insane” under relevant legal standards. Decisions like these make obvious the potential ethical quandary that psychologists find themselves in, when the available psychological data does not yield the conclusion that the client would like to have. A psychologist whose client has low motivation to stop using illicit drugs or who clearly can make reflective decisions and chose to nonetheless commit a crime is in a difficult position, wanting to advocate for the client, while also needing to avoid misrepresenting facts. In part, the need to avoid misrepresentation comes from an ethical duty that goes beyond individual clients. To retain credibility, psychologists must be viewed as having some degree of objectivity in their judgments. Even when a psychologist is not employed specifically as an independent professional by a decision-maker (by, say, a court or drug abuse rehabilitation program), they must try to remain aware of concerns larger than the client’s desires and even their well-being.

The Scout and Soldier Mindsets

A useful framework for thinking about the role of advocacy and objectivity in psychological practice (and elsewhere) is found in a recent book, The Scout Mindset, by Galef (2021). Galef discusses two “mindsets,” mental stances that we can adopt when engaging in reasoning and judgment.

The Soldier Mindset

Galef (2021) describes the soldier mindset as trying to find evidence in one direction, for or against a particular conclusion. The soldier asks “Can I believe this?” (i.e., is there a way that I can find any evidence to allow me to reach conclusion X?) or “Must I believe this?” (i.e., can I find any evidence to allow me to dispute conclusion X?). This mental posture is an intense one, where we try to defend a belief or position, in the same way that a soldier would guard something of value. As Galef notes, in adversarial situations, descriptions of argumentation frequently use metaphors from the world of military battle: we talk about claims being “challenged” or “destroyed,” or alternatively being “bolstered” and “supported.”

A large body of psychological research has long suggested that when the stakes are high, the soldier mindset is our default approach to reasoning (for an early review of relevant literature, see Kunda, 1990). This is famously the case when people think about moral and political matters (Haidt, 2001; Leeper & Slothuus, 2014), but is also present in many other settings. Indeed, this process can even be seen in the psychoanalytic defense mechanism of rationalization (Clark, 1998). Epley and Gilovich (2016) identified two sets of mechanisms in what they termed motivated reasoning (reasoning done with a strong motive to reach a particular conclusion). The first set of mechanisms involves bias in recruiting evidence. In short, we seek evidence that supports our preferred beliefs and avoid evidence that undermines it. For instance, if we anticipate getting negative feedback about our work from a particular coworker, we may never show our work to that person, even if their feedback is likely to be accurate and helpful. The second set of mechanisms involves bias in evaluating evidence. In essence, we shift our standards for the credibility of evidence, such that we have very rigorous standards for any evidence that might undermine our preferred conclusions, while being quite kind and lenient toward any evidence that supports those conclusions. For instance, if a cheese-lover reads a news article claiming to find health risks associated with cheese intake, they are more likely to find some fault with the article’s logic, whereas a similar article claiming benefits of cheese consumption will be uncritically believed.

Although psychologists have training in mental processes, they are still vulnerable to biased reasoning like that seen in the soldier mindset. Indeed, biases in recruiting and evaluating evidence are very much a worry in psychoeducational assessment. The use of performance validity tests (PVTs; Boone, 2017) provides a helpful example. PVTs have been shown to catch many cases of noncredible effort in cognitive testing, but a clinician may choose not to use a PVT (a bias in recruiting evidence) because of not wanting to obtain any data suggesting that the client is not trying hard enough during the evaluation. Similarly, even if a PVT is used, the clinician may try to explain away the results if the client fails the test (a bias in evaluating evidence), even to the point of proposing implausible alternative explanations of the test failure (Green & Merten, 2013).

Motives for the Soldier Mindset in Psychological Assessment

Why do psychologists adopt the soldier mindset? Looking at the research literature in the areas of clinical ethics and cognitive psychology, several possible motives present themselves.

Financial Conflicts of Interest

Financial conflicts of interest have long been recognized as one factor that can cloud the judgment of health care professionals, including psychologists (Cristea & Ioannidis, 2018). Financial conflicts of interest often have influence that health care providers are not even aware of (Sah, 2012). In the case of psychological assessment, several financial conflicts of interest may be operating. First, if clients (or their families, who may be the paying customers) are seeking particular diagnoses or recommendations, and a psychologist declines to provide those (due to lack of appropriate evidence), the psychologist may receive negative reviews, complaints, and even in extreme cases go unpaid by unhappy customers. Such customers are less likely to refer others in their social circle, and over time, a psychologist can gain a reputation within a community of not providing “helpful” documentation (i.e., the documentation that the customers want). Second, in some cases, evaluators also provide treatments or further consultation/advocacy directly, and so a diagnosis may generate income for the evaluator in the form of a continued relationship with that client. For instance, when an independent evaluator diagnoses a child or adolescent with a disability, the evaluator may be asked to attend meetings with the client’s school, or to provide remedial services. Finally, in many cases, a positive diagnosis is needed for insurance reimbursement. Assessment services are already poorly reimbursed relative to treatment, and if an assessment fails to identify any conditions, the client’s insurance may not pay anything at all.

Financial conflicts of interest may also explain why important research findings in the area of psychological assessment do not make their way into practice. For instance, despite ample research showing that it is common for healthy, nondisabled clients to get a few low scores in the course of completing a cognitive battery (see, e.g., Brooks et al., 2011), clinicians still routinely interpret a random low score as a sign of disability. Similarly, despite considerable evidence that young adults being evaluated for ADHD often exaggerate their symptoms or otherwise generate noncredible data (e.g., Nelson & Lovett, 2019), it is still rare to see validity tests incorporated into ADHD batteries (Nelson et al., 2019). I have led trainings and workshops on these issues for clinicians, and although there are occasionally principled disagreements over the issues, what seems more common is that clinicians who take this research seriously would not be able to make enough diagnoses, triggering the financial and reputational consequences discussed earlier. As the activist and organizer Upton Sinclair observed long ago, “it is difficult to get a man to understand something, when his salary depends upon his not understanding it” (Sinclair, 1935/1994, p. 109).

Desire for Positive Interpersonal Interactions

When clients come seeking a particular diagnosis or recommendations, or have any hoped-for outcomes at all, it is awkward for a psychologist to offer feedback that dashes the client’s expectations. Clients and their families may become upset, openly challenge the psychologist’s conclusions, and even claim that the psychologist lacks competence. This is a particular risk in cases where clients may interpret the feedback as containing negative character judgments. For instance, if a client fails a performance validity test or triggers a symptom validity index, delivering this feedback may make the client distressed or even angry (Carone et al., 2010). Another instance would be if the lack of a diagnosis suggests that a client’s trouble in functioning is due to factors under their control, leading them to feel blamed. An example would be citing heavy marijuana use, as opposed to ADHD, as a likely explanation of inattention symptoms that started at age 19. Even receiving feedback that one’s reported symptoms are in fact typical and not indicative of disability (e.g., reporting attention problems that are preventing achievement in a high-performance setting, but that yield average-range ratings on a norm-referenced self-report symptom questionnaire) may lead to distress and defensiveness. Psychologists often have training in resolving interpersonal conflict, but still typically strive to avoid it in professional interactions.

Clinical Empathy and Adoption of Client Goals

Many psychological assessment procedures involve attempts at entering into the client’s perspective—trying to understand their self-perceptions as well as their interests and goals. Very often, the client is seeking an evaluation for a purpose that relates to an educational, occupational, or personal goal, and good clinicians seek to understand that goal and gather assessment data relevant to it. Natural human empathy helps when adopting the client’s perspective, but without caution and care, a clinician may adopt the client’s goals as their own, even to the point of being tempted to draw conclusions and make recommendations that are unawarranted but that seem to assist the client in reaching the goal.

Part of the problem here is that even psychologists who specialize in assessment typically have a therapeutic end in mind when doing their work. Clients come to us in distress, and whether by evaluation or treatment, a reduction in distress is typically the ultimate goal of our work. Indeed, even official professional ethics guidelines state that “Psychologists strive to benefit those with whom they work and take care to do no harm” (American Psychological Association, 2017, p. 3). It can seem that helping clients to reach their immediate goals (e.g., obtaining desired medication or disability accommodations) is a way of providing benefit to them, whereas withholding such recommendations could be a form of harm. But the same ethics guidelines state that “Psychologists base the opinions contained in their recommendations, reports, and diagnostic or evaluative statements…on information and techniques sufficient to substantiate their findings” (American Psychological Association, 2017, pp. 12–13). Therefore, making claims that go beyond the available data, even in an effort to help a client, would seem to be unethical and unwise.

The tension between a therapeutic role and an objective one has long been noted by forensic psychologists (e.g., Greenberg & Shuman, 1997; Younggren et al., 2020). An explicit hallmark of forensic practice, enshrined in official practice guidelines (American Psychological Association, 2013), is impartiality—representing all data, including those on both sides of a question, fairly. In contrast, when conducting psychotherapy, it may be less important to determine the objective truth of everything that a client says. For instance, when a client reports a traumatic experience, the therapist does not typically need to investigate to find out whether the reports are precisely correct before beginning therapy. To be sure, there are times when clients’ inaccurate beliefs are at the root of their symptoms (Beck, 1976), but when someone is only seeking psychotherapy, there is not typically a strong motivation to lie about the symptoms themselves. Therefore, the therapist can afford to invest a degree of trust that would be foolishly credulous in a forensic context.

The problem here is that even when psychological evaluators do not think of themselves as forensic psychologists, they are often doing work of a quasi-forensic nature (cf. Lovett & Davis, 2017). When evaluators go beyond a clinical diagnosis to draw a conclusion about whether the client meets standards for disability, qualifies for accommodations, needs special education, or requires an emotional support animal, among other conclusions, the evaluators are determining whether the client meets a legal standard, not just a clinical one. And even clinical diagnoses often lead to outcomes such as ADHD medications that are highly desired by people with and without legitimate disabilities. An objective mindset akin to forensic practice is therefore needed in evaluations.

Confirmation Bias

An often-overlooked source of motivated reasoning is the simple desire to prove one’s own hypotheses correct, also known as confirmation bias (Nickerson, 1998). It is impossible, particularly for experienced clinicians, to avoid developing hypotheses about a client based on initial referral information. Very often, a client’s history (even in brief referral notes) includes a prior diagnosis, or at least symptoms that seem to match a particular disorder. Confirmation bias then leads evaluators to search for information that would support the hypothesis (biased recruitment of evidence) and to interpret any information obtained with an eye toward justifying the hypothesis (biased evaluation of evidence). For instance, if a college student presents with a prior diagnosis of ADHD, and reports needing only a re-evaluation to get documentation for accommodations, the psychologist may think of it as an easy case, needing only a quick confirmation of the prior diagnosis. This can lead the clinician to only obtain a self-report of ADHD symptoms. Such an evaluation lacks critical evidence such as third-party symptom reports (Sibley et al., 2012) and a ruling out of anxiety and depression problems that can also cause inattention (Harrison et al., 2013).

A confirmation bias becomes particularly acute under conditions of moral polarization, where the initial hypothesis is taken as necessarily true. When evaluators view themselves as advocates who need to assist their clients by proving the presence of a disorder or a need for particular accommodations or services, this moral fervor can lead to a neglect of the standards typically endorsed for competent practice. The moral mandate effect is social psychologists’ term for the tendency of moral concerns to lead people to abandon procedural integrity to gain outcomes that are viewed as morally correct and indeed imperative (cf. Skitka, 2002). Often, the judgments underlying such moral mandates are themselves driven by emotions rather than careful reasoning (Haidt, 2001). My opening case studies illustrate this well; sympathy felt for clients in difficult situations can drive psychologists to slant evaluation results to assist those clients, particularly if the psychologists see it as their moral duty.

The Scout Mindset

The alternative to motivated reasoning is what Galef (2021) calls the scout mindset. Instead of asking the soldier’s questions (“Must I believe this?” and “Can I believe this?”), the scout asks simply, “Is it true?” For Galef, the scout’s job is essentially that of a mapmaker, and the most accurate evidence is needed for a useful map. Whether information is “good news” or “bad news” is immaterial, as both types of information are equally important. A more theoretically sophisticated presentation of this mindset is found in the concept of actively open-minded thinking (AOT; Baron, 2019, 2020). As psychologist Jonathan Baron describes it, AOT has three core elements when making decisions: (a) there must be a sufficiently thorough search for evidence, (b) the search must not be biased toward a particular conclusion, and (c) the confidence in the conclusion should be adjusted based on the evidence.

The applicability of AOT to psychological evaluations is clear. With regard to the first element, evaluations should be comprehensive and thorough, testing multiple alternative explanations of symptoms and concerns, and obtaining data from multiple sources and using different types of techniques. This suggestion does not neglect concerns of efficiency; it does not promote unnecessarily thorough assessments. As Baron (2020) notes, at some point the costs of further search for evidence outweigh the benefits. At that point, search should cease. It is reasonable to consider financial costs and logistical difficulties in developing an assessment battery for each client, based on the certainty/confidence needed for the conclusion (i.e., the “stakes” of the assessment).

With regard to the second element of AOT, assessment tools should be selected and implemented in such a way as to avoid bias toward a particular conclusion regarding a diagnosis or recommendation. Although few psychologists conducting an ADHD evaluation would ask obviously leading questions (e.g., “You get distracted by a lot of things, right?”), a psychologist may only about examples of problems in past work settings rather than examples of good performance.

Finally, with regard to the third element of AOT, the confidence with which diagnostic and other conclusions can be made should be based on the evidence available and reflected in any oral feedback or written report from the evaluation. Expressing certainty in a diagnostic conclusion when the evidence is mixed is inconsistent with AOT, and there are times when (through no fault of the client or clinician) confidence must be necessarily low, because much relevant evidence is inaccessible. For instance, particularly during the early parts of the COVID-19 pandemic, psychological evaluations were being conducted under conditions that were far from optimal. Appropriately, experts recommended that clinicians widen their mental “confidence intervals” for interpretations and conclusions made under such conditions (Wright et al., 2020). This is actually appropriate advice whenever the quality of evidence is poor.

Some of these principles are found, or at least implied, in the APA (2017) ethics code. Earlier, I quoted the code’s statement that opinions from an assessment should be based in adequate information and techniques. The code also notes that when validity and reliability of an assessment tool has not been established, both “the strengths and limitations of test results and interpretation” should be described (p. 13). Finally, the code insists that any evaluations:

take into account the purpose of the assessment as well as the various test factors, test-taking abilities, and other characteristics of the person being assessed, such as situational, personal, linguistic, and cultural differences, that might affect psychologists’ judgments or reduce the accuracy of their interpretations. (APA, 2017, p. 13)

However, the code does not directly address the types of bias that typically drive Galef’s (2021) soldier, nor the typical sources of that bias in psychological evaluations. Instead, the code’s only discussion of psychologists’ biases relates to forms of social identity prejudice such as racism. Discussions of the cognitive biases described in the present paper can cause just as much threat towards accurate, and therefore ethical assessment practices, but practitioners must look beyond the ethics code for guidance.

Promoting the Scout Mindset

Once psychologists are out in the world working, it may be too late to expect many practitioners to push back against the network of incentives motivating the soldier mindset. Every effort should therefore be made to reach future practitioners during their training. Openly discussing the financial, interpersonal, and cognitive incentives that lead to poor assessment decisions before students encounter those incentives stands the best chance of changing reasoning and behavior without defensiveness or resistance. Further experience employing the scout mindset during closely supervised assessment cases in practica, fieldwork, and externship/internship settings reinforces these principles in a relatively “low stakes” environment where the supervisor has ultimate responsibility for the assessment report.

A particular issue that can be discussed in coursework and applied in practice training settings is the ethics of granting disability status without sufficient evidence and recommending services when they are unneeded. Recent discussions regarding professional psychology training frequently emphasize issues of equity and social justice (e.g., Council of Chairs of Training Councils, 2021), but there has been little attention paid to the equity and social justice issues raised by overdiagnosis and overaccommodation (but see Lovett, 2021 for a discussion of the latter). Students, many of whom are deeply committed to social activism (cf. Galán et al., 2021), may benefit from connecting this interest to soldier-mindset assessment practices that actually further inequality. Obviously, there is a danger that being so focused on equity-related outcomes may lead students to simply recommend diagnoses and accommodations to further that goal, but nonetheless it can be helpful to note that the ethical issues around these decisions are complex, and that the decisions have ethical consequences go far beyond the individual clients who are seeking evaluations. (For instance, recommending testing accommodations may help a client, but hurt other examinees who are competing with the client.)

Other ways of promoting the scout mindset would require much larger changes to our assessment system. Clear and detailed operational definitions of disabilities would eliminate the flexible thresholds that give evaluators so much room to consider almost anyone to be disabled (for further discussion, see Harrison, 2017). A similarly detailed decision model, endorsed by relevant professional bodies, for recommending particular accommodations or services would take pressure off of individual clinicians to make inappropriate recommendations. (For an initial attempt at such a model for making recommendations about emotional support animals, see Ferrell & Crowley, 2021.) Publicly funding, or even partially subsidizing psychological evaluations (for those individuals who first go through inexpensive screenings), would reduce financial incentives for problematic clinical practices and help to separate therapy work from neutral, dispassionate evaluation work. In the United States, we provide such evaluations in public schools for free and could make them part of health care more generally. Again, these may be large-scale changes to be sought long-term, but are likely not realistic to expect in the short term.

The Proper Role of Advocacy

Where does all this leave advocacy? Throughout the present paper, I have implied an association between advocacy and the soldier mindset on one hand, and objectivity and the scout mindset on the other. Interestingly, Galef (2021) argues that a scout mindset is helpful even for advocates. For instance, to be an effective attorney, before building the strongest arguments to advocate for your client, it is helpful to have the most accurate information about the facts of the case. An analogy can be made to psychological evaluations, where evaluators can best advocate for clients after obtaining the most accurate understanding possible of what the client’s problems are. In some cases, this is doubtless true, but in general, I find this argument unpersuasive. It is too easy to think of examples where clients’ interests may be served through poor evaluation practices, or even by lying. Indeed, my opening case studies may be among these examples. Promoting these clients’ welfare at all costs would require misstating facts. In sum, then, certain types of advocacy are in fact antithetical to objectivity.

However, a different conception of advocacy is consistent with the scout mindset. In this conception, psychologists and clients enter into an explicit agreement where the psychologist will assist the client by providing accurate information to the client—information that the client can share with any relevant third parties. The psychologist openly lets the client know that the information may not be what the client expects—some of it may be surprising or even distressing, but the psychologist will do their best job to interpret all of the evaluation data with the goal of drawing the most accurate conclusions possible based on the available evidence. If the client chooses to pursue the evaluation, they know that the psychologist will be devoting considerable effort to being accurate, not necessarily finding what the client expects or wants. This does not rule out roles where psychologists work directly for courts, testing agencies, or schools, but when psychologists are contracted by clients or their families, the psychologist makes clear that providing accurate information is their route to helping the client.

There is a second type of advocacy that does not violate the scout mindset, this type being at the systems level. Psychologists can advocate for system changes that permit more objectivity and lead to fewer incentives for making inappropriate diagnoses and recommendations. For instance, a psychologist who feels pressure to recommend extended time on tests can advocate that a testing entity consider whether its tests really are designed to measure speed or fluency of skills; if not, extended time may be made available to all who would like it. Similarly, a psychologist who feels pressure to write letters for emotional support animals can engage in policy advocacy when the regulations supporting these animals are being reviewed and public comment is solicited. Admittedly, policy advocacy must be done with caution and care, to keep psychologists from going beyond their areas of technical expertise and including their own political opinions and biases (Suedfeld & Tetlock, 1992), but psychologists can share relevant information with policy makers to help to make policies more empirically based. The scout mindset is again relevant here, since careful and ethical policy advocacy requires the same principles of actively open-minded thinking that appropriate clinical practice does: sufficient search for relevant evidence, unbiased presentation of evidence, and clear statements regarding how confident one can be based on the available evidence.

Conclusions

Carving out space for advocacy, both at the level of clients and of systems, can be professionally fulfilling for psychologists, and in certain limited ways advocacy can fit ethically into psychologists’ work. However, objectivity should be an overriding goal when conducting psychological evaluations. Objectivity often requires pushing back against a network of incentives that promote biased, motivated reasoning towards conclusions that clients may want to hear. Therefore, psychology trainees should be alerted to that network early in their education and be given strategies for promoting an objective scout mindset.