Cognitive Reflection and Decision-Making Accuracy: Examining Their Relation and Boundary Conditions in the Context of Evidence-based Management

Abstract

Evidence-based management (EBM) is an increasingly advocated yet rarely adopted framework to assist in making organizational decisions. We investigate how cognitive reflection – the ability or disposition to question an intuitive response and rely instead on a more analytical process—influences decision-making accuracy in the context of EBM. Across three experimental studies (N = 332, 166 and 99), we determine the effects of using actively sought and passively sought evidence, in boundary conditions of cognitive and emotional load. Results of study 1 show that cognitive reflection is generally associated with higher decision-making accuracy, particularly in passive pathways to EBM, and when exposed to low cognitive load. Results of studies 2 and 3 support that passive pathways to EBM will result in higher accuracy, and that negative emotion inducing stimuli can strengthen the effect of cognitive reflection on evidence collection. The findings demonstrate that a contingency approach to EBM is critical, with the impact of cognitive reflection differing for situational constraints and for actively and passively sought evidence.

Introduction

Initially conceptualized as an antidote to the research-practice gap, evidence based-management (EBM) has evolved into a comprehensive decision-making framework that draws on multiple sources of evidence. EBM is defined as the systematic collection and judicious use of the best available evidence from multiple sources when making organizational decisions (Barends & Rousseau, 2018, emphasis added). Thus, EBM comprises collecting and critically applying evidence from scientific research, but also from organizational data, practitioners’ expertise, and the stakeholders’ interests and concerns. In doing so, EBM has the potential to reduce bias, educate managers, and facilitate more accurate decisions (Pfeffer & Sutton, 2006). The aim of this paper is to advance knowledge in decision-making accuracy when engaging in different EBM pathways and conditions.

Searching for quality evidence and using it to make informed and accurate decisions has become a critical aspect of responsible and effective leadership (Lord & Hall, 2005) with research supporting beneficial outcomes for employees and firms when evidence is collected and applied judiciously (Camuffo et al., 2019; Ghasemaghaei et al., 2017; Song et al., 2018). Further, advancements in technology are facilitating further generation of evidence and its availability, making the need for decision-making accuracy in EBM increasingly relevant.

Research on EBM builds on the preceding movement towards evidence-based medicine, which originated three decades ago and inspired a shift towards more scientific and informed decision-making in medicine, as well as policy and education (Djulbegovic & Guyatt, 2017; Sackett et al., 1996). Research to date on EBM has mostly focused on conceptual papers (e.g., Barends et al., 2014; Briner et al., 2009; Rousseau, 2006) and advocacy articles arguing its importance in education and practice (e.g., Rousseau, 2020; Rynes et al., 2014). The empirical research conducted to date has generally focused on identifying the level of awareness of, and disbelief in, research findings from the academic literature which address organizational issues (e.g., Caprar et al., 2016; HakemZadeh & Baba, 2016; Rynes et al., 2018; Tenhiälä et al., 2016). However, EBM use involves more than the application of research findings, because it requires decision-makers to collect and apply the best available evidence from multiple sources.

In recent years, cross-sectional research has highlighted a positive attitude towards EBM among managers, yet a lack of EBM use in practice (Barends et al., 2017; Criado-Perez et al., 2020a). This gap between attitude and behavior in using EBM suggests that there are contextual and ability-related enablers and constraints on EBM. Indeed, participants from these studies reported a lack of time and cognitive resources as pervasive barriers to EBM use, and the accuracy of decisions in the context on EBM were not explored. Additional insights about enablers of EBM come from two EBM review papers that draw on related literature: Criado-Perez et al. (2020b) suggested a multi-level model of enablers for EBM by drawing on related strategy literature about absorptive capacity; Rousseau and Gunia (2016) reviewed the medical literature on evidence-based practice. Both papers highlighted individual differences as an important precursor of EBM, notably a preference for engaging in learning tasks, and an ability to evaluate and synthesize information to guide one’s decisions. Thus, there is emerging evidence that EBM use requires decision-makers to think critically and avoid a premature decision before evaluating the available evidence, yet the factors influencing accurate decision-making in the context of EBM require further investigation.

This paper focuses on how and when the accuracy of managerial decisions is enhanced or constrained by consulting with evidence which is differently presented. We draw on the dual-process theories of cognitive processing (Evans & Stanovich, 2013). The ability or disposition to monitor and override intuitive responses, referred to as cognitive reflection, is explored as an enabler of decision-making accuracy because this is associated with critical thinking and proper evaluation of evidence prior to making a decision. To explore the nuances of this relation we examine and compare active involvement in the collection of evidence and passive presentation of relevant evidence. Further, managers are often under pressure when making decisions, which may influence the appropriate degree of evidence collected and analyzed to inform a decision. Drawing on cognitive load theory (Sweller, 1988, 2011) we consider the moderating effect of cognitive and emotional load on the relationship between cognitive reflection and decision-making accuracy, thereby identifying boundary conditions and contextual opportunities.

The present paper makes important and multiple contributions to EBM and decision-making research. First, most research on EBM is conceptual (e.g., Barends et al., 2014; Briner et al., 2009), or survey focused with managers self-reporting the use of evidence (e.g., Barends et al., 2017; Criado-Perez et al., 2020a) and there is a scarcity of experimental work. This is surprising given the strengths of experimental investigation which include high internal validity and more accurate understanding of enablers and outcomes when using EBM. We examine decision-making accuracy for different approaches to EBM and draw from dual processing theory to understand their relation with cognitive reflection. Second, we draw links between cognitive reflection and cognitive load theory in the prediction of decision-making accuracy. Although research on judgements and decision-making associated with cognitive reflection is abundant (Campitelli & Labollita, 2010; Stanovich & West, 2014), much less attention has been placed on the conditions when this individual difference is most impactful. Individuals high on cognitive reflection may be most impacted by unfavorable contexts or most prepared to perform under conditions of cognitive and emotional load. In examining these two contexts we contribute to recent work on cognitive load theory, which has called for the need to consider the role of affect and arousal (Plass & Kalyuga, 2019; Young et al., 2021), as well as highlighting additional boundary conditions for the examined relations.

An Active and a Passive Pathway to Evidence-based Management

The judgement and decision-making literature has largely focused on studying how decision-making by fully informed individuals differs from normative models (Moore & Flynn, 2008). However, studies on decision-making have also proposed different heuristics or rules that individuals may apply for evidence acquisition based on the perceived validity of available cues (e.g., Coenen et al., 2019; Lee & Cummins, 2004; Newell & Lee, 2011), thus providing a basis for expecting individual differences in evidence acquisition. Moreover, evidence-based managerial decisions are characterized by uncertainty, as opposed to calculable risk and probabilities, and the available evidence can often be conflicting and of unknown validity. Less is known about how individuals collect and use information in such circumstances (Griffin & Grote, 2020; Hausmann & Läge, 2008). The cost of information and framing of choices has been shown to influence evidence collection and decision-making strategies (Bro̊der, 2003; Szaszi et al., 2018), yet research is lacking in regards to how factors such as individual differences and the active or passive seeking of evidence may influence decision-making accuracy and EBM (Curley et al., 2019).

Picture the following two scenarios. On the one hand, managers may walk into a meeting room to find all the relevant information from multiple sources collected, synthesized, and presented for them to make a decision that aligns with the best available evidence. On the other hand, managers may be presented with a problem to solve, and actively engage in the collection of information from multiple sources, until they make a decision based on the evidence collected. In this paper, we aim to examine the predictors of each of these two conditions under which EBM may be used resulting in what we refer to as active and passive trials of EBM. The differences between these two scenarios are important because situations that call for a purposeful collection of evidence may influence the amount of evidence collected as well as the way it is used, and each process may be predicted by different individual and contextual factors (see Case & Given, 2016).

By measuring decision-making accuracy in active and passive trials we can examine whether these two EBM pathways influence the use of available evidence. A large body of research on decision-making and information-seeking behavior provides evidence that the cost associated with information will deter individuals from collecting and using such information (Ashford & Cummings, 1983; Miller & Jablin, 1991; Morrison & Vancouver, 2000). This aligns with the Conservation of Resources theory (Hobfoll, 1989), which contends that individuals will aim to acquire new information as well as conserve resources that are perceived as helpful to reach their goals, such as information, time, or support (Halbesleben et al., 2014). Given the additional effort and time associated with the active collection of evidence, we expect individuals to consult with less evidence and therefore their decision-making accuracy to be lower in active trials compared to passive trials of EBM.

• H1: Individuals will make more accurate decisions when they are provided with the relevant evidence than when they have to actively collect that evidence.

Cognitive Reflection as an Underlying Capability Supporting EBM

Cognitive reflection is characterized by a tendency to question an intuitive response, relying instead on a more systematic analysis of the available evidence (Epstein et al., 1996; Stanovich & Stanovich, 2010; Stanovich & West, 2014; Toplak et al., 2014). Frederik (2005) conceptualized cognitive reflection as being associated with dual-system theories (Evans & Stanovich, 2013; Sloman, 1996). These theories propose that human cognition is governed by two systems: System 1 for fast, effortless and intuitive thinking, and System 2, which may override System 1 through more slow, analytic, and effortful thinking. High scores on cognitive reflection are associated with inhibition of an inappropriate impulsive response (Campitelli & Gerrans, 2014; Frederick, 2005) and a disposition towards considering alternatives through the available information (Campitelli & Labollita, 2010; Cokely & Kelley, 2009; Toplak et al., 2011). Unlike motivational differences such as curiosity and need for cognition, cognitive reflection is an ability. The ability to rely on an analytic process that considers the information available is essential for EBM. Thus, we expect individuals high on cognitive reflection to make more accurate decisions than individuals low on cognitive reflection.

• H2: Cognitive reflection is positively associated with decision-making accuracy.

Further, we argue cognitive reflection will be particularly important for decision-making accuracy when individuals are presented with all the relevant information. This aligns with the evidence that cognitive reflection is associated with overriding an impulsive incorrect decision and making an accurate decision considering the information presented. As such, individuals who are high on cognitive reflection will benefit the most from situations in which they have more information (Fig. 1)

• H3: Cognitive reflection interacts with the active/passive pathway to EBM, such that cognitive reflection will have a stronger effect on decision-making accuracy when individuals are provided with relevant evidence than when actively collecting it.

Fig. 1

Proposed hypotheses

Next, we turn to hypotheses specifically for the active pathway of EBM. Decision-making accuracy hinges on the effective synthesis of the relevant evidence. Decision-makers can only synthesize available evidence to inform one’s decisions as far as they collect the relevant evidence. If not provided with all the relevant evidence, it is difficult to determine inconclusive, misleading, or inconsistent guidance regarding the most appropriate course of action. Such variety in information accuracy is common when analyzing complex management problems (Scherer, 1998; Tranfield et al., 2003). Following a single lead can be misleading and conducive to poor decision-making. Thus, individuals who actively collect more evidence will be more likely to have higher decision-making accuracy.

• H4: Evidence collection is positively associated with decision-making accuracy.

Considering H4, and considering the association of cognitive reflection with inhibition of a premature response and the analysis of alternatives solutions (Campitelli & Labollita, 2010; Stanovich & West, 2014), we expect cognitive reflection to indirectly predict decision-making accuracy through an increase in evidence collection. That is, cognitive reflection is used to engage in behaviors of evidence collection, which facilitates decision-making accuracy.

• H5: Cognitive reflection is positively associated with decision-making accuracy through increased evidence collection.

Cognitive Load as a Distractor of the EBM Process

Cognitive load theory builds on our knowledge of human cognitive architecture, working memory (Miller, 1956; RepovŠ & Baddeley, 2006), and schema production (Sweller, 1988, 2011) to explain our limited capacity to engage in cognitive tasks that involve new information. The cognitive processing system of working memory is limited in duration and capacity and is used for temporarily storing, manipulating, and learning of novel information (Sweller, 2011). Cognitive load can be understood as the taxation of cognitive resources (Deck et al., 2021), or more precisely the taxation of working memory, and is increased by the intensity of tasks such as shifting attention, processing or abstracting information, and integrating alternative sources of information (Plass & Kalyuga, 2019).

Cognitive load can be categorized into intrinsic and extraneous cognitive load. Intrinsic cognitive load refers to the complexity and interactivity of knowledge being processed and can only be reduced by increasing the expertise of the learner. Extraneous cognitive load is unproductive for learning and performing a task, such as additional load produced by a lack of clear instructions or by the processing of information that is not relevant to the task at hand (Sweller, 2011). Processing of information and learning during tasks that involve high intrinsic cognitive load can be compromised by extraneous cognitive load as both intrinsic and extraneous load may burden the same resource -working memory (Plass & Kalyuga, 2019; Sweller, 1994). Given our limited capacity for cognitive load, a situational context that induces high (extraneous) cognitive load will likely impact the resources allocated to an effortful process like EBM, and as a result, influence the relation of cognitive reflection with evidence collection.

Individuals high on cognitive reflection have a disposition to engage in a more reflective and analytical decision-making style, which is more likely to involve higher intrinsic cognitive load compared to a superficial and heuristically guided analysis. These individuals may find extraneous cognitive load interferes with decision-making as they intend to collect and appraise the available evidence, compared to those who do not engage in these cognitive effortful tasks. Given the limitations of human cognition in information processing, cognitive reflection may only predict EBM use in situations when decision-makers have the capacity to engage in a thorough and analytical decision-making process. We propose individuals high on cognitive reflection will generally consult with more evidence, but will withhold from doing so when under cognitive load. Under high cognitive load the attention and resources focused on a task will be hindered and divided. Therefore, compared to individuals who score low on cognitive reflection, those who score high may reduce evidence collection when under cognitive load.

• H6: Cognitive load interacts with cognitive reflection in predicting evidence collection, such that high cognitive load weakens the effect of high cognitive reflection on evidence collection.

Alternatively, cognitive load could influence the relation between evidence collection and decision-making accuracy. We do not expect this to be the case. Under extreme conditions of cognitive load, the lack of available resources may interfere with the processing of the evidence collected. However, we expect individuals high on cognitive reflection to adapt the degree of evidence collection, instead of collecting evidence that they have no capacity to process.

Negative Emotional Load as a Distractor of the EBM Process

Emotional load – understood as high levels of emotion inducing stimuli—can also load our limited cognitive capacity. We examine the effect of emotional load on evidence collection through the lens of recent developments in cognitive load theory (Moreno, 2010; Plass & Kalyuga, 2019), which incorporates the influence of affect on effective cognitive load capacity.

Similarly to cognitive load, emotional load acts as a cognitive interference (Cheng & McCarthy, 2018; Collins & Jackson, 2015; Collins et al., 2016). This aligns with the Modified Higher Order Theory of Consciousness which contends that conscious experiences of stimuli, whether emotional or not, are fundamentally processed by the same mechanisms in the brain (Ledoux & Brown, 2017). As a result, both kinds of stimuli may incur a cognitive cost. Despite nuanced effects for different emotions (Blanchette & Richards, 2010; Forgas, 2008), heightened affect impairs performance in logic and reasoning tasks (Blanchette & Nougarou, 2017; Blanchette & Richards, 2004; Oaksford et al., 1996; Palfai & Salovey, 1993). However, heightened negative affect may also increase attentional focus, cognitive effort and effective cognitive load capacity (Knörzer et al., 2016; Plass & Kalyuga, 2019), resulting in impaired efficiency but not necessarily in impaired performance. Considering these findings, it is important to consider affect as an important factor when studying decision-making processes.

Given workplaces today are managing high levels of uncertainty and high levels of exhaustion, both of which are associated with negative emotions (Anderson et al., 2019; Spagnoli & Molinaro, 2021), we examine the effect of negative emotional load by inducing anxiety and anger. Anxiety and anger are high arousal and negative affect emotions (Posner et al., 2005). These emotions can impair processing efficiency and the inhibition of impulsive responses (Derakshan & Eysenck, 2009; Moran, 2016; Sarason et al., 1986). However, both emotions can also lead to increased effort, particularly when there are clear task goals and the task is demanding (Eysenck & Derakshan, 2011; Eysenck et al., 2007; Nabi, 1999, 2002; Seo et al., 2004). We suggest that anxiety and anger may strengthen the relation between cognitive reflection and evidence collection. Although we expect individuals high on cognitive reflection to be more vulnerable to the cognitive interference associated with emotional load, we suggest they may also benefit more from an increased effort put into the decision-making task. Whereas individuals low on cognitive reflection may be unable to compensate for the detrimental effects of anxiety and anger on decision-making, those high on cognitive reflection may be more capable of mitigating the effect of emotional load on inhibition and heuristic processing, choosing instead to collect evidence prior to a decision (Fig. 2).

• H7: An anxiety-inducing context interacts with cognitive reflection in predicting evidence collection, such that induced anxiety strengthens the effect of cognitive reflection on evidence collection.

• H8: An anger-inducing context interacts with cognitive reflection in predicting evidence collection, such that induced anger strengthens the effect of cognitive reflection on evidence collection.

Fig. 2

Proposed hypotheses for active pathway to EBM

Overview of Studies

Hypotheses 1 to 5 are investigated in all three experiments. Study 1 examines the relation between active/passive pathways to EBM and cognitive reflection with decision-making accuracy through an online experiment. We also include an induction of cognitive load to examine how this may weaken the relation between cognitive reflection and evidence collection (hypothesis 6). Study 2 and study 3 test the effect of active and passive pathways to EBM as well as cognitive reflection on decision-making accuracy, and examine the moderation effect of emotional load (hypotheses 7 and 8). We induce a state of anxiety or anger to examine how these high arousal negative emotions may strengthen the effect of cognitive reflection on evidence collection.

Study 1

Method

Participants and Procedure

Participants were 332 adults recruited through Prolific. A-priori power analyses with an alpha criterion of 0.05 suggested that this would be more than adequate to detect medium effect size with a high level of power. Average age of participants was 40 years old (SD = 13.20); 43% were male and 57% were female. All participants were native English speakers who reported having managerial experience; this was important to ensure that our sample was representative of the population of interest. Participants were randomly assigned to one of two conditions (high cognitive load vs control), and completed an online survey followed by a decision-making task in which participants played the role of a manager in an organizational context. The decision-making task involved one practice and eight assessment blocks. In each block, participants were first presented with a problem statement. After reading it, participants were presented with eight boxes, labeled with the different sources of evidence they could consult with (e.g., consultants, senior managers, customers’ or employees’ concerns, scientific research) in random order. These labels represented the four sources of evidence portrayed in the EBM framework – managers’ expertise, organizational data, stakeholder concerns, and scientific research. Participants could choose to collect and follow the available evidence in relation to four decision alternatives provided, and select a solution within a limited timeframe. Half of the assessment blocks were presented as active trials of EBM, whereby the participant had to actively request the available information. The other half were presented as passive trials of EBM, whereby the participant was presented with the evidence without any prompt. (see Appendix 1 for problem statements and Appendix Fig. 6 for screenshots of the decision-making task).

The information provided by each source of evidence was framed in the same manner to ensure that it was clear, indicative of a response, and not definitive. For example, the information provided by consultants was framed as follows: “Our expertise suggests that [solution X] may be the best option and [solution Y] might result in negative outcomes”. Most of the evidence guided the participant towards the correct solution, however misleading evidence was also available. For example, four sources of evidence supported the correct solution, whereas one or two sources of evidence supported each of the incorrect solutions. See Appendix Fig. 6 for a complete example of a management problem and the available evidence. The experiment was programmed using o-Tree software (Fischbacher, 2007). Similar to other published tools that analyze information acquisition and decision-making such as Mouselab (Willemsen & Johnson, 2010), the decision-making task extracted two types of data: the number of sources of evidence the participant consulted with and the selected solutions. Participants were paid a flat-fee upon completion of the experiment.

Cognitive load was induced by presenting participants with mathematical equations in parallel to the decision-making task. Participants in the cognitive load condition were instructed to solve them as quickly as possible and to report back the solutions after each decision. The equations consisted of addition or subtraction of two terms, consisting of one or two-digit numbers. The manipulation was adapted from previous research (Oberauer et al., 2000). Participants in the control condition were presented with the decision-making task without having to complete calculations in parallel nor remember the results.

Measures

Manipulation Checks

The efficacy of the induced cognitive load was assessed after the decision-making task through a three-item scale (Kelly et al., 2016). A sample item is ‘My mental energy is running low’. The scale showed high reliability with a Cronbach’s alpha of 0.87.

Cognitive Reflection

The cognitive reflection test (Frederick, 2005) was used, which comprises of a 3-item test that the participant could answer in a maximum time of 2 min (e.g., If it takes 5 machines 5 min to make 5 widgets, how long would it take 100 machines to make 100 widgets?). The items have been designed to make an incorrect yet intuitive response salient, albeit a correct response identifiable upon reflection. Therefore, these items assess the degree of cognitive reflection by measuring whether System 2 overrides System 1 through more slow, analytic, and effortful thinking. Scores on cognitive reflection ranged from 0 to 3 depending on the number of correct responses.

Evidence Collection

Evidence collection refers to the purposeful choice to request information related to a future decision. Evidence collection was assessed through the decision-making task developed and validated by Criado-Perez et al. (2019). The objective measure consisted of four assessment blocks from the decision-making task¹ referred to as active trials of EBM, during which the total amount of evidence the participant collected prior to making a decision was measured. Each block consisted of a management-related problem and eight dissimilar sources of evidence with which they could consult. A timer was also displayed to indicate the time remaining. Consulting with sources of evidence involved a 20-second delay between the time they requested information until the evidence was displayed, during which the participant could not request any additional evidence. This delay in combination with the limited timeframe to make a decision was built into the experimental design to simulate the costs associated with collecting evidence in a work environment and the timebound nature of management decision-making. Measures of evidence collection ranged from zero to 32 (eight sources of evidence for each of the four active trials of EBM) depending on the total amount of evidence the participant collected prior to making a decision.

Decision-Making Accuracy

We measured the accurate synthesis and application of relevant evidence that is available to the decision-maker. The available evidence included conflicting advice, yet most of the evidence guided the participant towards the correct solution. Each source of evidence provided supportive or unsupportive evidence for two of the four possible solutions. Measures of decision-making accuracy resulted from the number of correct responses selected during the four active and four passive trials of EBM.

We also measured need for cognition—the tendency to engage in and enjoy effortful cognitive tasks—with a 5-item scale (Epstein et al., 1996), as well as the personality traits of the HEXACO personality inventory (Lee & Ashton, 2010). This 60-item scale measured six factors of personality named Honesty-Humility, Emotionality, Extraversion, Agreeableness, Conscientiousness, and Openness to Experience.

Results

Manipulation Check

Participants in the condition with high cognitive load indicated significantly higher levels of cognitive load (M = 2.91, SD = 0.93) compared to the participants in the control condition (M = 2.32, SD = 0.76); t(333) = 6.32 p < 0.001.

Hypotheses Testing

The means, standard deviations, and correlations of the focal variables are presented in Table 1. Need for cognition and HEXACO factors of personality were excluded for parsimony as they did not influence the conclusions about our hypotheses. They did not correlate with the examined constructs (with the exception of need for cognition correlating with cognitive reflection).

Table 1 Means, standard deviations, and correlations among study 1 variables

As expected, participants collected significantly less evidence in active trials of EBM compared to the evidence presented in passive trials of EBM, with a mean difference of 22.54, 95% CI [21.64, 23.44] (t (331) = 49.04, p < 0.001). In support of H1, a paired-samples t-test showed decision-making accuracy scores were on average significantly higher in passive trials of EBM than when they have to actively collect evidence with a mean difference of 0.15, 95% CI [0.01,0.30] (t (331) = 2.16, p = 0.02). Next we regressed decision-making accuracy on cognitive reflection. As per H2, cognitive reflection predicted decision-making accuracy (ß = 0.17, p < 0.01). In support of H3, cognitive reflection interacted with the pathways to EBM (b = 0.17, p < 0.01) such that cognitive reflection had a stronger effect on decision-making accuracy when individuals were provided with the relevant evidence² (Table 2).

Table 2 Regression results for all EBM trials in Study 1

Next, we examined whether evidence collection mediated the relationship between cognitive reflection and decision-making accuracy in active trials. Cognitive reflection showed a significant effect on evidence collection (ß = 0.18, p < 0.01) and in support of H4, evidence collection had a significant effect on decision making accuracy in active trials controlling for cognitive reflection (ß = 0.59, p < 0.01). We then tested H5 using model 4 from the SPSS macro PROCESS based on 5000 bootstrap samples. Decision-making accuracy in active trials was included as the dependent variable, evidence collection as the mediator, and cognitive reflection as the independent variable. In support of H5, cognitive reflection showed a significant indirect effect on decision-making accuracy through evidence collection (Indirect Effect = 0.10, SE = 0.03, 95% CI [0.04, 0.17]).

The moderated mediation resulting from H6 was tested with model 7 from the SPSS macro PROCESS based on 5000 bootstrap samples. Decision-making accuracy in active trials was included as the dependent variable; cognitive reflection was included as the independent variable; evidence collection as the mediator; and the condition as the moderator of the relation between cognitive reflection and evidence collection. As shown in Table 3, the indirect effect of cognitive reflection on decision-making accuracy via evidence collection was significant in the control condition (Indirect Effect = 0.17, SE = 0.05, 95% CI [0.07, 0.26]) but not when under cognitive load (Indirect Effect = 0.03, SE = 0.04, 95% CI [-0.04, 0.11]). In support of H6, the interaction of cognitive reflection and cognitive load had a significant effect on evidence collection, b = -1.74, t(332) = -2.20, p = 0.03. As represented in Fig. 3, as expected, the effect of cognitive reflection on evidence collection was significantly stronger in the control group, b = 2.16, t(160) = 3.85, p < 0.001; compared to the cognitive load condition, b = 0.42, t(174) = -0.75, p = 0.45.

Table 3 Indirect effect of cognitive reflection on decision-making accuracy in active trials via evidence collection in study 1

Fig. 3

Two way interaction predicting evidence collection in Study 1

Our results provide evidence that in active trials of EBM, cognitive load will moderate the effect of cognitive reflection on evidence collection.³

Discussion

The results show that decision-making accuracy was higher in the passive condition than in the active condition. Furthermore, it was demonstrated that cognitive reflection is an important antecedent for decision-making accuracy. Individuals high in cognitive reflection make more accurate decisions overall, and especially when operating under passive conditions. Yet the results demonstrate that this effect of cognitive reflection is less strong when operating under cognitive load. Given this finding with cognitive load, next we seek to understand if emotional load is a similar contextual constraint for antecedents of decision-making accuracy.

Study 2

Study 2 focuses on (a) replicating Study 1 (i.e., H1 to H5), and (b) extending the findings to investigate if emotional load also acts as an environmental constraint moderating the effect of cognitive reflection on decision-making accuracy (i.e., H7 and H8).

Method

Participants and Procedure

Participants were 180 students across three conditions enrolled in management courses at a major Australian university. Recruitment was restricted to individuals without a history or known medical condition of epilepsy or anxiety. Decision rules for participation exclusion were set a priori. Participants were excluded if they reported minimum levels of anxiety or anger (1 on a scale from 1 to 7) in the experimental conditions in which these emotions were respectively manipulated to be high. Participants were also excluded if they reported minimum levels of neutral emotions (1 on a scale from 1 to 7) as well as elevated levels of anxiety or anger in the control condition (7 on a scale from 1 to 7).⁴ This resulted in 14 participants being excluded from the analysis resulting in a sample size of 166.

All experimental sessions were conducted in a controlled environment where each participant was assigned a computer with a headset to complete the task. Participants were randomly assigned to one of the three conditions (anxiety, anger, or control). The experiment was programmed and conducted with the z-Tree software (Fischbacher, 2007). In each of the conditions, participants responded to a questionnaire including a cognitive reflection test. Following this, they were asked to take a few breaths and listen to a song they would evaluate at the experiment's end.

Music has been widely used to induce emotions in experiments. Given the length of the decision-making exercise, the duration of the induced emotion was of particular importance. Panksepp and Bernatzky (2002) found that emotions from listening to music peaked immediately after the music was played and the emotion diminished 10 min after the music stopped. We played the music in a loop throughout the exercise. In the anxiety-inducing condition, participants listened to a section of the soundtrack from the movie Psycho. In the anger-inducing condition, participants listened to a section of Refuse/Resist by Apocalyptica. Participants in the control group listened to Indecision by Yo-Yo Ma. These songs have been used successfully in previous studies to induce the corresponding emotions (Brooks & Schweitzer, 2011; Ford et al., 2010; Gino et al., 2012; Tamir & Ford, 2012).

After listening to the song for 90-seconds without distraction, participants completed a decision-making exercise that measured EBM use. The decision-making task in study 2 was identical to the task in study 1 except that five blocks were used instead of four for each of the two pathways to EBM.

Measures

Manipulation Checks

The efficacy of the emotion inductions was assessed through two separate tests. Participants were asked to report to what degree they experienced a number of emotions (i.e., anger, anxiety, relaxation, happiness, fear, neutral) based on a 7 point Likert scale (1 = not at all; 7 = an extreme amount). The three-item scale for anxiety and anger showed high reliability with a Cronbach’s alpha of 0.85 and 0.83 respectively. In addition, neutral emotions were measured through a two-item scale asking participants about their experience of ‘Neutral’ and ‘Indifferent’ emotions. The scale showed good reliability with a Cronbach’s alpha of 0.83. Further, as a final step in the experiment, participants briefly described their feelings when thinking about the song. Responses were analyzed through the Linguistic Inquiry and Word Count (LIWC) program, which allows for an objective assessment of emotional displays such as anxiety and anger. LIWC analyses text to calculate the percentage of words associated with 76 different linguistic categories, such as anxiety, and negative emotion.

Cognitive Reflection.

Same as study 1.

Evidence Collection

As per study 1, evidence collection was measured in the active trials of EBM. Measures ranged from zero to 40 (eight sources of evidence for each of the five active trials).

Decision Making Accuracy

As per study 1, decision-making accuracy was measured through the objective scale presented in Criado-Perez et al. (2019). We included five blocks of the decision-making task, compared to the four included in study 1. Measures ranged from zero to five in the active trials and from zero to five in the passive trials depending on the total number of correct responses according to the available evidence.

Results

Manipulation Check

A one way ANOVA confirmed that there were significant differences in anxiety between conditions F(2, 163) = 13.87, p < 0.001. Planned contrasts revealed that participants in the anxiety condition reported significantly higher levels of anxiety (M = 4.19, SE = 0.20) than participants in the control (M = 2.66, SE = 0.21) and in the anger condition (M = 3.41, SE = 0.20). Similarly, there were significant differences in anger between conditions F(2, 163) = 29.81, p < 0.001. Planned contrasts revealed significantly higher levels of anger for participants in the anger condition (M = 3.99, SE = 0.20) than in the control (M = 1.85, SE = 0.20) and in the anxiety condition (M = 2.59, SE = 0.20).

The manipulation was also assessed using the Linguistic Inquiry and Word Count (LIWC) software. to analyze participants’ responses to an open-ended question on how they would describe their feelings while listening to the music. A one-way ANOVA demonstrated that percentage of anxiety-related words differed significantly among the different conditions when participants described their feelings, F(2, 163) = 8.21, p < 0.001. Planned contrasts support a significant increase in the percentage of anxiety-related words in the anxiety condition compared to the control (p < 0.01) and to the anger condition (p < 0.01). Further, a one-way ANOVA demonstrated that the percentage of anger-related words also differed significantly among the different conditions, F(2, 163) = 6.66, p < 0.01; with planned contrasts supporting a significant increase in percentage of anger-related words in the anger condition compared to the control (p < 0.01) and the anxiety (p < 0.01) condition. Overall, the results support an effective manipulation of anxiety and anger in their respective conditions.

Hypotheses Testing

The means, standard deviations, and correlations of the focal variables are presented in Table 4.

Table 4 Means, standard deviations, and correlations among study 2 variables

Similar to study 1, we test all hypotheses except that related to cognitive load (H6) as we examined emotional load instead (H7 and H8). Individuals collected significantly less evidence in active trials of EBM compared to the evidence presented in passive trials of EBM with a mean difference of 22.86, 95% CI [20.96, 24.74] (t(165) = 24.00, p < 0.001). In support of H1, a paired-samples t-test showed that decision-making accuracy was significantly higher in passive trials than in active trials of EBM with a mean difference of 0.27, 95% CI [0.05,0.45] (t(165) = 2.39, p < 0.01). Next, we regressed decision-making accuracy on cognitive reflection. As shown in Table 5, 2 was supported as cognitive reflection was positively associated with decision-making accuracy (ß = 0.27 p < 0.01). However, the interaction of cognitive reflection with the active/passive trial of EBM was not significant in predicting decision-making accuracy (b = 0.08, p = 0.45).

Table 5 Regression results for all EBM trials in study 2

Next, we examined the predictors of decision-making accuracy in active trials of EBM. Cognitive reflection showed a significant effect on evidence collection (ß = 0.24, p < 0.01) and on decision-making accuracy in active trials (ß = 0.21, p < 0.01). In support of H4, evidence collection had a significant effect on decision-making accuracy when controlling for cognitive reflection (ß = 0.65, p < 0.01). We examined whether evidence collection mediated the relationship between cognitive reflection and decision-making accuracy using model 4 from the SPSS macro PROCESS based on 5000 bootstrap samples. In support of H5, cognitive reflection had a significant indirect effect on decision-making accuracy in active trials through evidence collection (Indirect Effect = 0.20, SE = 0.06, 95% CI [0.08, 0.33]).

To investigate emotional load as the moderator of cognitive reflection on evidence collection, we test H7 and H8 through model 7 in PROCESS based on 5000 bootstrap samples. Results are summarized in Table 6. In support of H7, the interaction term of cognitive reflection and induced anxiety was significant, b = 6.32, t(110) = 2.92, p < 0.01. In support of H8, the interaction term of cognitive reflection and induced anger was also significant b = 4.93, t(110) = 2.31, p = 0.02. The effect of cognitive reflection on evidence collection was significantly stronger under the conditions of anxiety, b = 5.39, t(56) = 3.55, p < 0.001 and anger, b = 3.99, t(56) = 2.65, p < 0.01; compared to the control condition b = -0.93, t(54) = -0.61, p = 0.54. The simple slopes are plotted in Fig. 4, indicating that induced anxiety and anger strengthen the relation between cognitive reflection and evidence collection.⁵

Table 6 Indirect effect of cognitive reflection on decision-making accuracy in active trials via evidence collection in study 2

Fig. 4

Two way interaction predicting evidence collection in study 2

Discussion

In study 2 we examined how cognitive reflection influenced decision-making accuracy when using EBM in emotion inducing contexts. The results replicate study 1 in that decision-making accuracy is lower in the active trials of EBM than in the passive trials; cognitive reflection is an important predictor of evidence collection and decision-making accuracy; and that situational context moderates the relation of cognitive reflection with evidence collection. Unlike in study 1, the effect of cognitive reflection on decision-making accuracy was not significantly different in active and passive trials of EBM.

Furthermore, as expected, the contextual restraints – cognitive and emotional load – only moderated the relation of cognitive reflection and the amount of evidence the individual collects. Recall in Study 1 that cognitive load weakened the relation between cognitive reflection and evidence collection. In Study 2 we replicated that, and also showed that emotional load strengthens instead of weakens the relation between cognitive reflection and evidence collection. The results suggest individuals high on cognitive reflection can mitigate the effects of the induced negative affect. Indeed, previous literature has suggested that rational processing styles may also help to make “affect-free” decisions (Pacini & Epstein, 1999) and that the associated larger working memory capacity helps regulate emotions (Schmeichel et al., 2008). Our findings provide empirical evidence for individuals high on cognitive reflection compensating for the induced emotional load, perhaps through attention regulation and emotion management. Given these novel findings with emotional load, we replicate and extend these in Study 3.

Study 3

Research on cognitive load theory has provided abundant evidence on the effects that extraneous load can influence decision-making. The influence of emotion and its incorporation into cognitive load theory has received much less attention, so we aim to replicate the findings related to emotional load. Our findings from study 2 suggest that the heightened affect and arousal may be driving individuals to exert more effort. In study 3 we investigate in more depth –with electrodermal activity and cardiovascular data – whether there is an increase in arousal during evidence-based decision-making in the conditions of induced emotional load. To further test our hypotheses—including the differing contingencies associated with passive and active evidence-based decision-making- we also aim to replicate study 2 by having the participants complete the decision-making task in a virtual environment.

Traditionally, decision-making research has focused on cognitive constructs, which lends itself to laboratory settings and computer based experiments (Cromwell & Panksepp, 2011). Yet a large body of research has emphasized the need to also consider affect and motivation to understand decision-making (Blanchette & Richards, 2010; Lerner et al., 2015), and understanding the influence of these constructs require more immersive simulations (Masmoudi et al., 2012). Virtual reality can enhance the affective experience and task engagement, providing better ecological validity without compromising the internal validity provided by an experimentally controlled environment (Parsons, 2015).

Method

Participants and Procedure

Participants were 106 students enrolled in a large Australian university. Following the same decision rules for participation exclusion as study 2, a total of 7 participants were excluded from the analysis resulting in a sample size of 99.⁶ This was smaller than the previous two studies because of limitations in collecting data using VR headsets. However, power analysis still indicates that we have adequate power. For example, a power analysis suggested that a sample of 100 results in a power of 0.92 to detect a medium sized correlation. Participants followed the same experimental procedure described in study 2, yet performed the decision-making task through a virtual reality headset and a 3D environment coded in Unity. The same management problems, evidence, and possible solutions were used in study 2. However, sources of evidence were now represented by individuals sitting around a meeting table with labels indicating the nature of evidence they provided (e.g., scientific research, consultant’s advice). See Appendix Fig. 7 for a sample screenshot.

Additionally, the arousal of participants was monitored through an Empatica E4 bracelet, which includes a galvanic skin response sensor and a heart rate sensor. The bracelet was placed on each participant’s non dominant wrist upon arrival. The collected data served as an additional validation of the emotional manipulation and subsequent arousal. Participants were requested to avoid caffeine and alcohol in the 6 h prior to the experiment, and to avoid intensive exercise 24 h prior to the session as these are known to interfere with the biometric measurements (Laborde et al., 2017; Nunan et al., 2010).

Measures

Manipulation Checks

The efficacy of the emotion inductions was assessed through separate tests. Participants were asked to report emotions based on a 7-point Likert scale (1 = not at all; 7 = an extreme amount) as per study 2. The three-item scale for anxiety showed a high reliability with a Cronbach’s alpha of 0.86. The three-item scale for anger also showed high reliability with a Cronbach’s alpha of 0.85. Neutral emotions were also measured through a two-item scale asking participants about their experience of ‘Neutral’ and ‘Indifferent’ emotions. The scale showed good reliability with a Cronbach’s alpha of 0.86. Participants also briefly described their feelings in their own words when reflecting about the song.

In addition to the manipulation checks described above, the effectiveness of the manipulation was assessed with measurements of the autonomic nervous system (ANS). The ANS is divided into the sympathetic and parasympathetic branches, generally associated with activation and relaxation, and influences involuntary body responses such as heart beats (Mauss & Robinson, 2009). The ANS plays a crucial role in emotion response and regulation, and are therefore helpful to validate the experimental manipulations of anxiety and anger (Cacioppo et al., 2000).⁷ Empatica E4 bracelets recorded heart rate variability, electrodermal activity, and motion to facilitate the detection of artefacts.

All other measures were the same as study 1 and 2, including: cognitive reflection, evidence collection, as well as decision-making accuracy.

Results

Manipulation Check

A one way ANOVA confirmed that there were significant differences in anxiety between conditions F(2, 95) = 8.27, p < 0.001. Planned contrasts revealed that participants in the anxiety condition reported significantly higher levels of anxiety (M = 4.04, SD = 1.54) than participants in the control condition (M = 2.69, SD = 1.35). Similarly, there were significant difference in anger between conditions F(2, 95) = 7.47, p < 0.001. Participants in the anger condition indicated significantly stronger levels of anger (M = 3.21, SD = 1.59) compared to the control condition (M = 1.77, SD = 1.15).

The participants’ heart rate variability was decomposed through autoregressive modelling for frequency domain analysis (Laborde et al., 2017; Tarvainen et al., 2018). The low frequency to high frequency ratio of heart rate variability (LF/HF) was measured through the software Kubious HRV 3.3.1 before and during the emotion induction. An increase in LF/HF is associated with feelings of stress, panic, and fight-or-flight behaviors (Kreibig, 2010; Murakami & Ohira, 2007; Shaffer & Ginsberg, 2017). Participants in the condition with induced anxiety indicated a significantly larger increase in LF/HF compared to participants without the induction (contrast estimate = 1.06, SE = 0.34, p < 0.01). Participants in the anger condition also indicated a significantly larger LF/HF compared to the control group (contrast estimate = 0.91, SE = 0.38, p = 0.02).

An electrodermal activity measure was used to assess the level of arousal of participants. Several studies have reported increased number of skin conductance response (SCR) under induced states of anxiety (see Kreibig, 2010 for a review). The collected data were inspected and decomposed using Continuous Decomposition Analysis (Benedek & Kaernbach, 2010) with the software Ledalab. As expected, the rate of SCR was higher in the anxiety condition in comparison with the neutral condition (contrast estimate = 44.01, SE = 18.31, p = 0.02); the rate of SCR was also higher in the anger condition in comparison with the control condition (contrast estimate = 55.24, SE = 21.27, p = 0.01).

Overall, the results suggest that the manipulation was successful. The physiological data support the induction of a negative valence emotion and a higher level of arousal in the conditions of induced anxiety and induced anger. Furthermore, the questionnaire supports a subjective experience of anxiety and anger in their respective conditions.

Hypotheses Testing

The means, standard deviations, and correlations of the focal variables are presented in Table 7.

Table 7 Means, standard deviations, and correlations among study 3 variables

A paired-samples t-test showed decision-making accuracy was significantly higher in passive trials of EBM with a mean difference of 0.58, 95% CI [0.30, 0.86] (t(98) = 4.09, p < 0.001), supporting H1. In support of H2, cognitive reflection predicted decision-making accuracy (ß = 0.24, p < 0.01). This effect was not moderated by active/passive trials of EBM (b = 0.09, p = 0.48), hence not supporting H3 (Table 8).

Table 8 Regression results for all EBM trials in Study 3

Evidence collection showed a significant effect on decision-making accuracy when controlling for cognitive reflection (ß = 0.54, p < 0.01), supporting H4. We tested whether evidence collection mediated the relationship between cognitive reflection and decision-making accuracy using model 4 from SPSS macro PROCESS based on 5000 bootstrap samples. Decision-making accuracy in active trials was included as the dependent variable, evidence collection as the mediator, and cognitive reflection as the independent variable. H5 was not supported (Indirect Effect = 0.04, SE = 0.06, 95% CI [-0.07, 0.14]).

Next we investigate emotional load as the moderator of cognitive reflection on evidence collection with model 7 from the macro PROCESS based on 5000 bootstrap samples. Decision-making accuracy in active trials was included as the dependent variable; cognitive reflection was included as the independent variable; evidence collection as the mediator; and the condition as the categorical moderator of the relation between cognitive reflection and evidence collection. As shown in Table 9, the indirect effect of cognitive reflection on decision-making accuracy via evidence collection was significant under induced anxiety (Indirect Effect = 0.23, SE = 0.08, 95% CI [0.08, 0.40]), but not in the anger condition (Indirect Effect = 0.01, SE = 0.10, 95% CI [-0.19, 0.19]) nor in the control condition (Indirect Effect = -0.07, SE = 0.09, 95% CI [-0.28, 0.10]). Further, the interaction of cognitive reflection and induced anxiety had a significant effect on evidence collection, b = 4.83, t(93) = 2.39, p < 0.05; whereas the interaction term of cognitive reflection and anger was not significant in predicting evidence collection b = 1.12, t(93) = 0.47, p = 0.64.⁸

Table 9 Indirect effect of cognitive reflection on decision-making accuracy in active trials via evidence collection in study 3

The simple slopes analysis supports that the effect of cognitive reflection on evidence collection was significantly stronger under the condition of anxiety, b = 3.61, t(92) = 2.12, p < 0.05; compared to the control condition b = -1.06, t(92) = -0.61, p = 0.51. The simple slopes are plotted in Fig. 5 and show that when placed in an anxiety-inducing context individuals high on cognitive reflection collected more evidence than those who scored low on cognitive reflection. In the anger condition the effect of cognitive reflection on evidence collection was not significant, b = 0.82, t(92) = -0.61, p = 0.51.

Fig. 5

Two way interaction predicting evidence collection in Study 3

Discussion

The core focus of Study 3 was on emotional load. Study 1 demonstrated the debilitating effects of cognitive load on the EBM process, and study 2 demonstrated that emotional load can strengthen the relation between cognitive reflection and evidence collection. Study 3 replicated these findings on emotional load with self-report and biometric data. The findings suggest that under a state of anxiety, those high on cognitive reflection will mitigate the debilitating effects of emotional load and result in more evidence collection. Interestingly, although anxiety interacted with cognitive reflection as predicted, the interaction of anger and cognitive reflection did not predict evidence collection.

Results provide additional confidence in the findings that decision-making accuracy is lower in active trials of EBM compared to passive trials (H1), that decision-making accuracy is predicted by cognitive reflection (H2), and that decision-making accuracy in active trials is predicted by evidence collection (H4). Surprisingly, cognitive reflection did not interact with active/passive trials of EBM to predict decision-making accuracy and cognitive reflection only predicted decision-making accuracy through evidence collection in the anxiety condition.

In comparing the predictors of decision-making accuracy in active and passive trials of EBM, we once again found that emotional load did not interact with cognitive reflection to predict decision-making accuracy in passive trials. This finding provides further support for the differing predictors and contingencies associated with passive and active trials of EBM as emotional load only moderates the relation between cognitive reflection and evidence collection.

General Discussion

The theoretical case for EBM as a timely and helpful practice to aid managers make better decisions has been presented compellingly by many scholars (e.g., Barends & Rousseau, 2018; Briner et al., 2009), and is supported by efforts to help managers rely on the multiple growing sources of evidence such as scientific research, organizational data, or big data (e.g., Burke et al., 2004; Gupta & George, 2016; Rousseau et al., 2008; Tonidandel et al., 2018). However, we are just beginning to scratch the surface of empirically examining the key enablers to facilitate more accurate decisions by using EBM. We aimed to address this important gap by investigating cognitive reflection as an enabler for more decision-making accuracy under differing pathways to EBM. We hypothesized and found the impact of cognitive reflection is contingent upon the evidence collection process and upon situational constraints (cognitive and emotional load).

Theoretical Implications

Our findings extend current research in two ways. First, for EBM we provide insights as to how the process of evidence collection (passive vs active) impacts decision accuracy. Specifically, we provide consistent evidence that decision-making accuracy is higher when decision-makers are provided with the relevant evidence instead of actively seeking for it, and that cognitive reflection is an important predictor of decision-making accuracy. The interaction of cognitive reflection with active/passive trials of EBM was inconsistent across studies. Thus, we aggregated the findings across the three studies and test the hypothesized interaction using multilevel modeling. We found support for a two-way interaction such that cognitive reflection had a stronger effect on decision-making accuracy when all the evidence was presented to the participants (b = 0.14, p < 0.01). This suggests individuals high on cognitive reflection will particularly benefit from situations in which evidence is presented.⁹ Our findings also suggest that it is important to study the different components involved in EBM as they will be influenced by different factors. Studying EBM use through a single overarching measure will likely fail to capture the important nuances involved in the different pathways to EBM and the respective steps it entails.

Second, we expand cognitive load theory by highlighting the importance of heightened negative emotion in influencing load and decision-making. We draw links between cognitive reflection and cognitive load theory in the prediction of decision-making accuracy, demonstrating the conditions in which this individual difference is most impactful in the context of EBM. Our findings suggest that individuals high on cognitive reflection will make better decisions in light of the evidence available, and will seek for more evidence, but that cognitive and emotional load are important contingency factors for the impact of cognitive reflection on decision-making accuracy through active (and not passive) trials of EBM. We show that cognitive load weakens the effect of cognitive reflection on evidence collection, but that negative emotional load, notably anxiety, strengthens this relation.

The seminal works on cognitive load theory (Sweller, 1994, 2011) conceptualize an individual’s capacity for cognitive load as stable, and do not consider the potential variance due to the exerted effort that an individual places on a task. Results from study 2 and 3 provide evidence that heightened negative affect may influence effective cognitive load capacity (Plass & Kalyuga, 2019). Our contribution is that negative emotional load – as opposed to cognitive load – can strengthen the relation of cognitive reflection and EBM through an increased effort in evidence collection.

In addition, the findings from studies 2 and 3 support slightly different implications related to emotional load. Study 2 suggests that emotional load should be minimized, and that individuals high on cognitive reflection will be able to cope with the debilitating effects of anxiety and anger on decision-making. Study 3 suggests that individuals high on cognitive reflection may perform better when anxious by exceeding their levels of evidence collection through compensatory efforts. This finding suggests that these individuals may benefit from a temporary state of anxiety if their goal is to increase EBM use. As suggested by Cheng and McCarthy (2018), a state of anxiety may interact with ability or motivation to increase episodic performance; however, dispositional anxiety will likely be associated with low performance due to emotional exhaustion. Further, an emotionally-induced increase in evidence collection can assist in evidence-based decision-making but may also be counterproductive for decision-makers if it facilitates anxiety-driven compulsive forms of information seeking in the workplace (Griffiths, 2010).

Practical Implications

Understanding what pathways, abilities, and contextual factors enable accurate evidence-based decisions can help educators, recruiters, and managers. Our findings support that passive pathways to EBM will lead to more accurate decision-making and highlight the benefits of investing in resources that can aid decision-makers in having the relevant evidence at hand. Our findings also suggest that extraneous cognitive load should be minimized in working environments where EBM use is particularly critical, and that some of the decision-making benefits of high cognitive reflection are lost under high cognitive load. However, our results suggest that negative emotions can have a different moderation effect, whereby individuals high on cognitive reflection may even benefit from moderate levels of induced anxiety. As a result, employers may want to apply the cognitive reflection test during screening and selection for roles where evidence-based accurate decisions are critical, particularly if they will be exposed to negative emotional stimuli in their professional roles (e.g., anxiety for healthcare workers such as psychologists and social workers as well as those in the military exposed to natural disasters and war).

Limitations and Future Research

Strengths of the three studies include the experimental manipulation of active and passive trials of EBM as well as cognitive and emotional load. This strengthens the internal validity in the examination of predictors of evidence collection and decision-making accuracy. However cognitive reflection was not a manipulated variable, hence the design doesn’t allow us to infer that its relation with decision-making accuracy is causal in nature. We also relied on objective measures of evidence collection and decision-making accuracy instead of self-reported measures which have been found to overestimate its use in other disciplines (Adams et al., 1999).

One important limitation is that the experimental design provides good internal validity but a weaker ecological validity due to limitations in simulating real management decision-making in a controlled environment. We aimed to mitigate this limitation by recruiting participants with management experience in study 1 and by designing a virtual environment for study 3. Future field research on EBM and decision accuracy would benefit from testing some of our findings in the work environment. In such field settings, it will be important to build on the EBM measures to achieve an increasingly comprehensive assessment of EBM that can also discern other components within the EBM framework beyond the dual pathway of passive and active decision-making. For example, the proposed measures of EBM could be modified to include evidence with varying degrees of external and internal validity, shedding light on the critical step of evidence appraisal.

This study focused on the proposed association between cognitive reflection and decision accuracy in the context of EBM. It would be beneficial for future research to consider additional individual differences relevant to EBM, both ability-based and motivational differences such as metacognition, curiosity, and other promising enablers. For example, intellectual humility – which refers to the degree to which an individual may recognize to hold beliefs that are wrong (Leary et al., 2017) – will likely influence how individuals will appraise evidence that contradicts their beliefs. A number of skills and domain knowledge will be required for the correct appraisal and application of evidence (Daouk-Öyry et al., 2020; Rousseau & Gunia, 2016). In addition to individual differences, dimensions of psychological distance of a decision – time, social distance, and probability – may also impact EBM use as they have shown to heavily influence preferences on evidence collection (see Halamish & Liberman, 2017). Similarly, the various forms of complexity of a decision (Haerem et al., 2015) may also impact evidence collection and use.

Further, future research could clarify some of the inconsistencies found by examining emotions beyond anxiety and anger. A promising line of research would also involve testing whether the effects on accuracy when engaging in EBM change for positive heightened emotions, and to discern the mechanisms through which emotional load in the form of dispositional and situational emotions influence EBM use. Previous research has reported conflicting findings on the effect of positive emotional states on cognitive load and learning (e.g., Hawthorne et al., 2019; Knörzer et al., 2016) which may be best understood when considering individual differences (see Chuang & Lin, 2007). We aimed to examine the effects of emotional load by inducing two negative emotions and found consistent results for anxiety yet differing results for anger between studies 2 and 3. Although both emotions successfully increased arousal, anxiety and anger differ in several aspects which could contribute to their differing effect. While some researchers contend anxiety is characterized by an appraisal of uncertainty and lack of control (Todd et al., 2015), anger scores high on the dimensions of certainty and perceived control (Lerner & Keltner, 2000). As such, anxiety and anger may have opposite effects on the perceived predictability of events and on action tendencies related to information seeking and risk taking (Lerner et al., 2015). It is possible the effect of arousal will depend on the action tendency or motivational system associated with the particular emotion (Carver & Harmon-Jones, 2009). Given that we did not find any difference between the effect of anxiety and anger in study 2, it is also plausible that the use of a VR headset and the subsequent immersion within the task might be influencing the strength of these action tendencies. More research is required regarding this relatively new research method to unfold these mechanisms. The use of FMRI or EEG data may shed light on the inconsistencies found and clarify questions regarding the level of arousal, motivational direction, and cognitive load experienced during the task.

Conclusion

Relevant data and evidence are ubiquitous in today’s work environment. Then why is effective EBM use so rare? Our results suggest that a passive pathway to EBM in which the evidence is collected for the decision-makers will result in more accurate decisions. Furthermore, cognitive reflection is an important predictor of decision-making accuracy, particularly when the relevant evidence is gathered separately and presented to the decision-maker. We show that cognitive load and emotional load, particularly anxiety, will also influence EBM. Importantly for theory and practice, the effect of cognitive reflection on evidence collection is weaker under cognitive load, but stronger under a state of anxiety. These findings highlight that for accurate evidenced-based decisions, decision-makers’ ability and context plays a vital role, and needs further investigation for EBM to become a reality.

Appendices

Appendix 1 – Examples of Management Problems

Trial question: Recently there have been several accidents at your workplace. An internal audit was performed and the resulting report indicates that some of your employees are not following the health and safety regulations. What initiative would you implement to make sure your employees follow these regulations?

• 1. Your finance department is growing and needs to move into a new office space. You decided to hire top architects and interior designers to create a workspace that helps their productivity and wellbeing. What kind of interior design would you ask for? You are presented with four options: Open space offices, closed offices, hot desking, or cubicles.

• 2. Your latest employee survey reflects a decrease in motivation in your team. You have a small budget to spend towards the end of the year. You wonder how you could use this budget to boost your team’s morale. You are asked to decide between organising a team building activity outside the office, setting goals to give a bonus to high performers, dividing the budget equally and distribute it at the end of the year, or distributing it equally right away.

• 3. You are interviewing candidates to hire as your new CEO for a subsidiary opening soon. It is critical that you choose the right person for the job as your company is making a heavy investment in this subsidiary. Your shortlisted candidates have completed several tests and gone through several interviews. You are deciding whether to hire Roger, who scored the highest in conscientiousness; Jessica, the highest in emotional intelligence; Ben, a candidate that has already been working in the company for 12 years; or Catherine who you preferred based on a long and unstructured interview.

• 4. Every year your organisation launches an expensive TV marketing campaign during Christmas to increase sales. This may be a good time to launch your new chocolate products but you wonder whether you should: Wait until after Christmas to reduce costs, focus your marketing efforts on social media, organise a large event in the city, or invest in an expensive TV marketing campaign.

• 5. Based on the latest forecasts, you expect your team’s workload to increase next year. Several of your team members spend a lot of their time commuting to work so you are considering implementing a new policy so people can decide to work from home or in the office. You are asked to decide who this policy should apply to: Apply it to everybody, apply it to those that work individually, apply it to those with children, do not implement the new policy at all.

• 6. Lately your sales department is underperforming. You are wondering how to boost their performance by setting annual goals. You can set annual goals with regular performance reviews, with no reviews during the year, with an emphasis on autonomy as to how to achieve them, or not set a goal and encourage employees to “do their best”.

• 7. As you set up the assembly line for your chocolate bars you hear about “Lean manufacturing” practices to increase quality and efficiency in the production lines. You are wondering who you would like prioritize to receive a training on “Lean manufacturing”: Top managers, middle managers, the operators in the assembly line, or a few individuals at each level in the hierarchy.

• 8. You are designing your new headquarters for the majority of your employees to work in. You have some extra budget and you want to spend it in a way that enables productivity and wellbeing. You are asked to decide whether you invest in: a state of the art office design, free corporate bus lines to transport employees, a sports centre for employees, or a childcare center.

Appendix 2

Figure 6

Fig. 6

Otree screenshots for study 1 with example evidence provided for a management problem

Available evidence:

• “Our expertise suggests that not implementing the policy at all may be the best option and applying it to employees with children might result in negative outcomes” [Consultant]

• “Our expertise suggests that not implementing the policy at all may be the best option and applying it to everybody might result in negative outcomes” [Consultant]

• A study of over 20 companies suggests applying the new policy to employees with children may be the best option, and that applying the policy to employees who work individually might result in negative outcomes

• A study of over 20 companies suggests applying the new policy to everybody may be the best option, and that applying the new policy to employees with children might result in negative outcomes

• A survey of some of your employees shows that not implementing the policy at all or applying the new policy employees who work individually may be the best option

• A survey of some of your employees shows that applying the new policy to employees with children may be the best option, and that applying the new policy to everybody might result in negative outcomes

• Consult senior manager: “I’ve worked in different departments where this policy was applied. It seems to me that not implementing the policy at all may be the best option and that applying the new policy to employees who work individually may result in negative outcomes” [Senior manager]

• Consult senior manager: “I’ve worked in different departments where this policy was applied. It seems to me that applying the policy to employees who work individually may be the best option, and that not implementing the policy at all may result in negative outcomes” [Senior manager]

Appendix 3

Figure 7

Fig. 7

Screenshot of virtual environment designed for study 3