Introduction

Recruiting a diverse workforce does not necessarily improve performance. Diversity may generate a performance bonus when a task is so complex that no individual expert can be expected to master all the relevant knowledge, experience, and perspectives needed to address the task (Page 2017). For less complex tasks (such as difficult mathematical problems), one expert can easily beat a crowd, regardless of how diverse it is.

The logic of generating a diversity bonus seems straightforward yet difficult to put into practice.Footnote 1 Organizations facing complex problems tend to put a misplaced belief in meritocracy—hiring the “best”. If a complex problem is like a jigsaw puzzle (Simonton 2004), the “best” may provide more pieces but unlikely all of them, because the “best” tend to be good in a homogeneous way (Hong and Page 2004). In addition, organizations tend to mistake identity diversity for cognitive diversity (Liu 2021). Recruiting a demographically diverse team does not necessarily generate non-overlapping cognitive repertoires, which are essential to generating a diversity bonus (Oliveira and Nisbett 2018; Page 2017). These biases imply that a performance bonus from recruiting a diverse workforce may exist as potential, but often be left unexploited.

I argue that random selectionFootnote 2 is an undervalued tool for overcoming several barriers to capturing the diversity bonus. Random selection cannot guarantee optimal hiring—locating additional team members who have exactly the missing pieces to the “puzzle”. But random selection can improve the baseline outcome of biased hiring decision, such as those polluted by the meritocracy bias, nepotism, homophily, or stereotypes. Randomness may help organizations uncover more missing pieces by revealing how self-confirming processes may have blinded them from finding these pieces (March 1991; Park and Puranam 2021). Because random selection means deciding based on no reason, blind luck can trump biased reasoning and generate a less-is-more effect: one has less control over outcomes but achieves more by saving time and resources, as well as detecting and sanitizing biased decisions.

Acknowledging the function of random selection has important implications for managers and organizational designers. Capitalizing on the diversity bonus offers the best hope of solving the increasingly complex tasks faced by organizations. While individual cognitive capacity and ability are unlikely to grow significantly, collective diversity can continue to increase if differences among individuals and teams are valued, developed, and mobilized. In contrast to ancient wisdom and practices in politics, which view random selection as a viable, alternative decision device (Pluchino et al. 2011; Stone 2011; Zeitoun et al. 2014), modern management ideology, channeled through business school education, leads too many managers to believe that every decision needs to be based on reasons (Augier and March 2011). Random selection is useful not because it can outperform decisions based on good reasonsFootnote 3 but because it relieves organizations from the fruitless pursuit of good reasons and attenuates the negative impact of hubris, illusion of control, and biased reasons (Berger et al. 2020; Liu and de Rond 2016; Stone 2011).

In what follows, I discuss how random selection can capture the diversity bonus by addressing: (1) the paradox of merit, by avoiding fruitless deliberation; (2) biased reasoning, by deciding on the basis of no reason; (3) learning traps, by uncovering self-confirming false beliefs. I conclude with a “GBU” decision flowchart to highlight when it is sensible to incorporate random selection in organizational design.

Random selection overcomes the paradox of merit by avoiding fruitless deliberation

Random selection cannot beat a decision based on good reasons. But is it always worthwhile to find a reason to justify why a hire is optimal? Suppose that several “good enough” candidates exist who have irreducible trade-offs rather than one optimal hire. Spending more time and resources to judge these trade-offs may not significantly enhance decision quality but entail substantial opportunity costs: the hiring committee could have spent their time on other recruitment opportunities that would make a bigger difference.

I argue that random selections can help organizations avoid fruitless deliberation, particularly in the later stages of recruitment. For example, shortlisted candidates are, by definition, not bad options. How much deliberation by the committee is needed to select the best one from the shortlisted? Research on the paradox of merit suggests that possible improvement from deliberation at later stage of selection is likely trivial (March and March 1977; Mauboussin 2012).

To illustrate, consider an organization that sorts its candidates by performance and removes the lowest-performing ones to enhance overall fitness. Let us assume each candidate i’s performance at period t is a combination of merit (Mi) and noise (Ei,t).Footnote 4 Merit is a time-invariant component and is drawn from a standardized normal distribution, i.e., N(0,1). Noise varies with time and is also drawn from a standardized normal distribution. This model assumes many candidates are present in the initial stage and that the selection operates for many rounds. The lowest-performing 5% of candidates at each round are eliminated, while the rest progress to the next round. We are interested in how the average merit (and its variance) of those selected in this recruitment process evolves over time (i.e., rounds).

Figure 1 illustrates the “paradox of merit”. Figure 1A shows the function of selection: the average merit of those selected increases over time because the least-skilled employees are weeded out. However, Fig. 1B highlights an important side effect of selection: the reduction of diversity (specific to the variance in meritFootnote 5) among those selected. Figure 1C shows the decreasing efficiency of selection: the merit-improvement rate over rounds drops very quickly. Such paradox of merit holds whenever the same selection criteria (e.g., having a college degree or not; publishing a certain number of academic papers; reaching a sales target) are applied to all candidates. The implication is that the “survivors”—those who passed multiple rounds of selections, such as the candidates in the final round—are very competent (Fig. 1A), but the differences among them are very small (Fig. 1B) and increasingly indistinguishable from those selected against (Fig. 1C).

Fig. 1
figure 1

How A average merit, B variance in merit, and C improvement ratio among survivors vary with selection rounds

Full size image

Random selection helps attenuate the challenges generated by the paradox of merit—that is, the decreasing effectiveness of deliberate selection over time. Random selection can save organizations from wasting time in endless meetings in the later stage of recruitment trying to choose the best from an already good-enough pool of candidates. Moreover, random selection may save firms from overspending on candidates who are not likely to provide more value to the firm than other candidates and allows resource allocation to potentially more productive activities, such as R&D, while also potentially attenuating inequality. This may be particularly relevant when the board determining executive pay is dominated by conservatives, who tend to over-attribute superior performance to the individual and often over-pay the chief executives they carefully chose (Gupta and Wowak 2017). Such excessive pay cannot be justified if those chosen are at best marginally better than the rest (March 1984).

Random selection sanitizes biased reasoning by deciding on the basis of no reason

Random selection can also help organizations address biases in the recruitment and evaluation process that go beyond gender and racial discriminations, such as the not invented here syndrome (Reitzig and Sorenson 2013), discounting of others’ ideas that are thematically close to one’s own work (Boudreau et al. 2016), the overvaluing of one’s own ideas (Keum and See 2017), the undervaluing of colleagues’ ideas owing to turf wars or competition for resources (Criscuolo et al. 2017), and to be fooled by exceptional luck (Denrell et al. 2017). These patterns are predictably irrational and random selections can help sanitize these biases.

Among the biases, the meritocracy bias is often overlooked, but in fact fuels the problems generated by the paradox of merit. Consider an organization that is hiring an additional member for its top management team to address a complex task. Whom should the organization recruit? According to the logic of generating a diversity bonus (Page 2017), the organization should first evaluate the nature of the task—that is, what types of knowledge, tools, or experiences are essential to address this task. Next, the organization should recruit additional team members with cognitive resources that match the task requirements and do not overlap with those of existing members.

However, there is a “no test exists” rule when assembling a diverse team: “no test applied to individuals will be guaranteed to produce the most creative groups” (Page 2017, p. 95). Complex tasks require a cognitively diverse team, but the team’s cognitive diversity cannot be recognized in isolation or ex ante (e.g., through a test with objective criteria); rather, it has to be identified along with the team composition and expansion.

Rather than appreciating the “no test exists” rule and hiring team members sequentially, organizations often believe that they can solve complex problems by recruiting the “best individuals” based on objective criteria (Thorngate et al. 2008). This belief not only wastes time and resources in finding a candidate who is at best marginally better than the rest (see Fig. 1C), but also a worse candidate under the logic of generating a diversity bonus in teams.

To capture the diversity bonus, organizations should avoid the best and instead engage the luck of the draw among the rest. That is, for later rounds of selection, the best candidates should be dismissed because they are decreasingly likely to provide additional diversity to an existing team (see Fig. 1B). The rest of the candidates may appear worse, but their inferiority to the “best” signals useful differences that may contribute to the team’s cognitive repertoire.

More generally, this approach—dismissing the best option and randomly selecting one among the worse—is not new (March 1991). To overcome premature convergence, organizations need to sample alternatives that appear (or are believed to be) worse from time to time. Random selection is a useful heuristic to overcome the exploration–exploitation dilemma, particularly in the domain of human resources, where belief in meritocracy creates strong biases in favor of exploiting reliably good candidates. Such hires eliminate the possibility of capturing diversity bonus because their superiority tends to correlate with homogeneity.

One caveat is that useful diversity may already be weeded out if the selection is highly efficient, suggesting all later-stage survivors fail to provide viable cognitive diversity to the existing team. Random selection at an earlier stage is perhaps desirable: once apparently destructive candidates are removed (soft screening or shortlisting), organizations should hire a randomly picked candidate from the pool. Note that the selected may be unqualified, according to objective criteria. But this is exactly the occasion where the diversity bonus may be created: The candidate is more likely to have perspectives or experience lacked by the “elites”—excellent human resources selected based on objective criteria.

Random selection creates competitive advantage by uncovering self-confirming learning traps

The idea that random selection can create serendipitous knowledge or competitive advantage has only been recognized recently (Biondo et al. 2013; Denrell et al. 2015; Larcom et al. 2017; Puranam 2021) and deserves more attention. In particular, some biases in organizations persist because they are protected by learning traps (Denrell et al. 2019): Actions based on these biased beliefs strengthen their illusory validity instead of revealing their flaws (Park and Puranam 2021). Competitive advantage can be created by discovering how rivals are fooled by these learning traps and exploiting their biased beliefs.

A generic approach is “contrarian experimentation”: using randomized experiments to test against conventional wisdom or common beliefs in an industry (e.g., adopting practice X will lead to high performance). Most of these experiments would fail, in the sense that common sense and best practices probably reflect the wisdom of the crowd, channeled by effective emulation and diffusion. Yet when the results show that the conventional wisdom is flawed (e.g., not following practice X in fact leads to higher performance), this creates an attractive opportunity. Firms that discover and strategize with such self-confirming learning traps are likely to enjoy competition-free growth, as their rivals will likely misattribute such unorthodox successes to luck instead of trying to eliminate the threat (Liu 2021).

This is how Richard Fairbank founded Capital One, which disrupted the credit-card-financing industry in the 1980s. Before 1980, credit-card financing was dominated by large banks, such as Citi, Bank of America, and Chase. The creditworthiness of new applicants was computed based on data on debt/income ratio, credit scores, and interview scores. Applicants whose overall scores were above banks’ cut-off point were offered credit cards with a unified annual percentage rate (APR) and annual fee.

While Fairbank was earning his MBA at Stanford in the early 1980s, a speaker from the credit-card-financing industry visited his class. Fairbank was puzzled by this dominant business model: “The fact that everyone had the same price (same APR and annual fee) for credit cards in a risk-based business was strange”, he later reflected. Higher-risk customers were subsidized by lower-risk ones, he realized. He believed he could improve efficiency by introducing “mass customization”, which is the key element of Capital One’s business model and led to exceptional success.

Capturing a performance bonus from engaging diversity also helped Capital One grow. Fairbank only hired people from outside the banking industry to design and run their contrarian experimentation. Experienced employees from the banking industry can be useful in the short run but they would likely have internalized all conventional wisdom. As such, experience in the banking industry could be seen as biased, particularly from Fairbank’s perspective who aimed at disrupting the industry with an uncommon belief. Instead, hiring was mainly targeted at those who were free from the possible constraints of conventional wisdom and norms, harnessing “the power of an objective ignorant view of the world from someone who really didn’t know anything about credit card business” as Fairbank highlights.

Fairbank also captured a diversity bonus from the customers overlooked by rivals. He had a hunch about possible inefficiencies in the evaluation of customers. In particular, he believed that interviews of customers were overrated. To examine this hypothesis, Capital One conducted experiments that tested whether randomly accepting applicants with good-enough credit scores, regardless of their interview scores. Capital One found interview scores had no predictable power for the loan repayment record in the next three years, thereby proving that interviews were neither necessary nor informative.

Interviewing as a “best practice” in this industry was not immune to stereotype bias: viable but atypical applicants during the era, such as female, minorities, and immigrants, were being improperly rejected due to their unwarranted low interview scores.

Capital One captured the diversity bonus by focusing on counter-stereotypical customers with decent debt/income ratios and credit scores who provided an excellent opportunity for market penetration. These customers have been not only profitable but loyal to Capital One, as many of them could not have gotten credit cards from major providers that implicitly discriminated against them.

Importantly, incumbent credit-card-financing firms cannot identify that they were wrongly rejecting some viable applicants because these errors are invisible to them. Instead, the importance of interviews is self-confirming: the incumbents’ profitable customers are increasingly like to be stereotypical ones, unless they decouple interviews from evaluations, like Capital One did.

Structuring incentives to encourage contrarian experimentation is crucial to prohibit learning traps from emerging in a contrarian strategist’s own backyard. For example, Capital One rewarded ideas rather than seniority by giving bonuses and promotions to employees whose proposed ideas were proven to work in randomized experiments. This policy enabled Capital One to sustain a contrarian culture, expressed in the hundreds of experiments the firm ran every year, while its competitors remained trapped in the illusory validity of outdated best practices. It took rivals at least 2 years to overcome these limits and emulate Capital One’s business model, a delay that allowed Capital One to gain a strong position in the credit-card-financing industry with almost no competition.

Determining when to incorporate random selection in organizational design

The above discussions suggest that random selection may help organizations capture the diversity bonus. But when should an organizational designer incorporate a luck-of-the-draw approach? Fig. 2 provides a decision flowchart that highlights the GBUFootnote 6 considerations: Good (can a randomly selected candidate be sufficiently Good?), Bad (can a deliberately selected candidate be sufficiently Bad?), and Ugly (can an organizational designer defend the decision to go random when things turn Ugly?).Footnote 7

Fig. 2
figure 2

A decision flowchart of when one should apply random selection to overcome diversity biases

Full size image

The first question (the “Good”) is whether the designer has minimized the chance of falsely omitting good reasons, e.g., by using effective selection criteria to filter out inferior candidates. It is absurd to apply random selection when there are still good reasons available that can be applied to decision-making. That is, randomly selected candidates are not yet sufficiently good.

Only when you have exhausted viable reasons should you move on to the second question (the “Bad”): Are there poor reasons that could creep into the process and have dire consequences, such as high opportunity costs (e.g., wasting time on fruitless deliberation) or strong bias or learning traps that blind the organization from viable but atypical opportunities? If the answer is no, random selection may not be necessary even when the randomly selected can be good enough. This is because the luck of the draw may face other challenges, such as an identity threat or lack of psychological safety because the way they are selected may appear illegitimate.

If the answer to the second question is positive, random selection can lead to a potentially better outcome than a decision based on biased reasons. However, even when random selection is likely to trump biased decisions, this does not necessarily imply that an organizational designer should apply it when career concern is considered. Question 3 (the “Ugly”) suggests that the organizational designer should evaluate whether important stakeholders understand the reasons for deciding on the basis of no reason. If important internal and external stakeholders do not appreciate the logic of random selection, a diversity bonus may not be realized (due to failure of inclusion) or be discounted even when realized (due to lack of legitimacy). Moreover, the designer will likely be held accountable for any low performance resulting from the luck-of-the-draw approach, even when the failure may simply be a matter of bad luck.

Thus, applying random selection is only feasible when one’s stakeholders appreciate your strategy or when you are insensitive to others’ evaluations (Benner and Zenger 2016; Liu 2021; Zuckerman 2012). Otherwise, the organizational designer should abandon this random approach and prepare for the worst-case scenarios that may result from biased decisions.

This last consideration perhaps suggests why random selection is rarely observed in modern management.Footnote 8 Since the Enlightenment, many have been convinced that human reasoning can overcome all challenges. Decision by lottery has been degraded to an irrational belief in luck or to giving up control to divinity, like our ancestors did. This belief can create a self-confirming false belief: true believers that insist on reason-driven decisions can never discover reason-absent decisions can sometimes be more effective.

But we live in a world characterized by uncertainty, limited foresight, and controllability. Insisting on deciding by reasons can lead to an illusion of control when many factors are actually beyond our control (Gimpl and Dakin 1984; Langer 1975). Random selection can have a less-is-more effect—deciding by luck of the draw is to exercise less control over outcomes but to achieve more by detecting and sanitizing biased reasons.