The ability to discriminate self from non-self is fundamental to the evolution and function of biological systems ranging from multicellular organisms to colonies of social insects (Grosberg 1988; Hölldobler and Wilson 1990; Bourke and Franks 1995; Crozier and Pamilo 1996; Tsutsui 2004). In the social insects, it is believed that a worker discriminates nestmates from non-nestmates by comparing the olfactory cues on the surface of individuals they encounter with a cognitive representation of the suite of acceptable odors, known as a template (Getz 1982; Lacy and Sherman 1983; Stuart 1988; Breed et al. 1988; Reeve 1989; Tsutsui 2004). In many cases, young workers form their template by imprinting on the odors of nestmates (Lacy and Sherman 1983). The odor profile is a composite of various chemicals, which may be of genetic or environmental origin, with cuticular hydrocarbons, which have a genetic basis, playing a prominent role (Vander Meer and Morel 1998; Soroker et al. 1998; Lah av et al. 1999; Howard and Blomquist 2005; Foitzik et al. 2007; Martin et al. 2008, 2009, D’Ettorre and Lenoir 2010; Guerrieri et al. 2009; D’Ettore and Lenoir 2009).

A widely accepted and influential model for nestmate recognition has explored the forces that drive the evolution of optimal acceptance thresholds in individual workers (Lacy and Sherman 1983; Reeve 1989; Downs and Ratnieks 2000; Couvillon et al. 2008). In this approach, individuals are unable to distinguish nestmates from non-nestmates with perfect precision because there may be overlap in the labels (odor cues) possessed by individuals in separate colonies and because sensory systems may be insufficiently sensitive to detect small differences in labels. The resulting recognition errors can take two forms. On one hand, extremely stringent recognition systems will effectively exclude non-colony members, but may also inadvertently lead to the inappropriate rejection of nestmates (referred to as false rejections). On the other hand, extremely permissive recognition systems should minimize false rejections, but open the door to the inappropriate acceptance of non-nestmates (acceptance errors). Although the costs of recognition errors vary among species and through time, the trade-off between them is thought to lead to an optimal acceptance threshold that dictates whether an individual accepts or rejects in a given encounter (Reeve 1989).

In contrast to theory, which assumes an overlap between the cues of different colonies such that members of different colonies can have identical cues, work on the chemistry of nestmate recognition shows that olfactory cues are variable between colonies and uniform within colonies (reviewed in Howard and Blomquist 2005; Martin et al. 2009, in press). The diversity of recognition cues allows an experimenter, for example, to reliably distinguish one colony’s members from another’s (Vander Meer et al. 1989; Martin et al. 2008). Research has also shown that even in polygynous colonies, which have the greatest genetic diversity underlying cues, mechanisms for sharing chemical cues ensure that each member of a colony has the same overall cue profile (Martin et al. 2009). In spite of these results, however, there is still likely a functional overlap between the cue distributions of different colonies because workers may not be able to perceive relatively small differences in composite odor profiles. Further, the complex environment in which nestmate recognition occurs ensures numerous errors. For one thing, the nest can be crowded and workers may not get a good chance to inspect another individual as it pushes its way through the bustle of workers. Returning workers may also have been exposed to food stuffs with strong and/or similar odors to their own, for example (Liang and Silverman 2000; Chen and Nonacs 2000). Likewise, a honey bee that has partially acquired the odor cues of a foreign colony while robbing must still be accepted by its own colony (Breed et al. 1988; Couvillon and Ratnieks 2008). Hence, there is likely some limitation on the part of a worker to quickly distinguish between similar odors that create an error window around the threshold for acceptance, which itself is modified by the current environment. This context is illustrated in Fig. 1. If the acceptance window is moved to the left to exclude all non-nestmates, it would result in the rejection of nestmates; thus, illustrating the basic trade-off between true and false rejections.

Fig. 1
figure 1

Conceptual model for the trade-off between the prevalence of true and false rejections based on the limited olfactory acuity of individual workers. Any perceived worker with a template cue dissimilarity value to the right of the acceptance threshed is rejected, while those to the left are accepted. Imperfect discrimination ability on the part of the worker is proposed to generate an error window (within which odor differences cannot be distinguished). Depending on the width of this window, and the placement of the acceptance threshold, this window can generate a trade-off between acceptance and rejection errors. In order to account for studies showing no false rejections, the acceptance level must be set far to the right of the nestmate distribution, necessitating a large number of false acceptances of non-nestmates

Here, we explore nestmate recognition as a collective process. We test the hypothesis that colonies, unlike individuals, can achieve near-perfect nestmate discrimination (in spite of the above-mentioned physiological trade-offs) because a colony, unlike a single worker, gets multiple chances to make the correct decision. We first review the literature to determine whether data on recognition errors supports the traditional individual-level model or a collective one. We then develop two agent-based models to explore nestmate discrimination when it is treated as a collective process. The first focuses on the context of when a colony posts no dedicated guards. The second model explores the more complex case of active nest defense, in which the colony posts guards specialized for the detection of intruders.

Materials and methods

Review of social insect recognition studies

We conducted a search, using Web of Science, with nestmate recognition as the search phrase. We evaluated for suitability all papers on nestmate recognition in ants, bees, and wasps during the past 20 years. Studies were included if they presented clear data on nestmate and non-nestmate rejection rates. Although the studies focused on numerous topics, such as the role of genetic or environmental variation on nestmate recognition, many studies used unmanipulated controls for comparison with their treatment results. We report the results of the controls, as they indicate typical rates of worker success and error in the course of nestmate recognition. Codes for aggression scoring methods are as follows: a = aggression index, b = presence of aggression, c = percentage rejected by guards, d = percentage rejected, e = aggression towards dead workers, f = % aggression in 1 to 1 assays, g = # attacks per trial, h = time based aggression scale, I = aggressive vs. non aggressive interactions, m = number of aggressive ants.

Model 1: within-nest nestmate recognition

A spatially explicit agent-based model was developed using the Netlogo programming language (Wilensky 1999; code in the supplemental information). The model is heuristic in nature and should be applicable to most species. The model was developed to test the hypothesis that increasing worker density within a nest leads to an increase in worker encounter rate which, in turn, should decrease the time required to recognize and reject individuals from a different colony. The nest contains 528 patches, over which workers move in a continuous manner (they do not jump from patch to patch). Patches thus do not govern movement behavior, but are instead used as reference for the area over which workers can perceive others. Patch number is arbitrary and does not affect the results of the model. Workers split their time evenly between activity and inactivity in 20 minute bouts (based on studies of within-nest honey bees, Johnson 2008). Inactive workers remain motionless and are unresponsive to non-nestmates. Active workers that encounter non-nestmates engage in nestmate recognition. Workers have a 10% probability of encountering a non-nestmate that is on their patch. Based on Fig. 1, the threshold for acceptance is set such that workers have no probability of rejecting nestmates, but a high probability of false acceptances. The mean per-encounter probability of rejecting non-nestmates is 10% (each non-nestmate has an individually distinct odor profile represented as a probability of rejection that varies normally with mean, 10%). Nestmates move at random while working (rate based on data from Johnson 2008), while non-nestmates perform a continuous correlated random walk with the same step length. Hence, non-nestmates are assumed to be walking in more of a straight line as they search the nest for resources to steal, while nestmates travel less as they work. Each non-nestmate begins at the entrance of the nest and moves about until detected.

The results of the model are robust with respect to variation in all the parameter settings (probability of encountering a non-nestmate on a patch, movement rates, per-encounter rejection rate, and so forth). Setting the probability of detection per patch higher, for example, leads to more rapid detection, as does increasing the probability of rejection per encounter. The effect of varying each parameter is thus intuitive, which is not surprising given the heuristic nature of the model.

Model 2: nest guarding

An agent-based model was developed to simulate guarding at the entrance of honey bee colonies (code in the supplemental information). The model was inspired by studies conducted to test the optimal acceptance trade-off model (Reeve 1989; Downs and Ratnieks 2000; Couvillon et al. 2008). Two contexts were modeled. First, fixed numbers of participants were simulated with no recruitment of new guards, and second, a feedback process was modeled that regulates guard number with respect to invader success, which correlates with strength of attack against the colony.

Nest entrance

The entrance is 3 by 5 patches. Each guard can only perceive workers attempting to pass through the patch they currently occupy. Thus, the entrance is a narrow one of the sort favored by honey bees nesting in tree hollows. Variation in the length of the entrance does not affect the qualitative behavior of the model, but longer entrances are more difficult to defend. For simulations shown in the text, bees (both invaders and nestmates) took 20 s to walk completely through the entrance.

Guard behavior

Guard behavior is patterned after studies of guarding (Moore et al. 1987; Breed et al. 2002). Each time step, a guard randomly contacts either a nestmate or non-nestmate attempting to move through its patch. Guards do not fight for extended periods of time with bees they reject; they simply eject them from the nest. Adding fighting to the model does not change the results, other than to necessitate a higher number of guards to ensure an equal probability of keeping out invaders (because a guard occupied with fighting is not available for nestmate recognition). Guards do not reject one another, but can encounter one another instead of a returning worker. Guards remain in one location, as per real guards who remain motionless in a characteristic stance when not inspecting workers (Winston 1987). Guards turn over rapidly, quitting their task after 45 min (Moore et al. 1987; Breed et al. 2002).

Returning foragers

Returning foragers attempt to run directly (in a straight line) into the nest. They move at a rate of 0.2 patches per second. When rejected, returning foragers exit the nest and wait outside for 60 s, before attempting to re-enter the nest. Foragers continue to attempt to re-enter until they are successful, which translates into 5–15 min of effort in most runs of the model. Each returning foraging has an individually distinct odor profile represented as a fixed normally distributed probability of rejection per encounter. Mean rejection probability was 20% per encounter, an empirical result from Downs and Ratnieks (2000).

Invading non-nestmates

Invader (robber) behavior is the same as for returning foragers with a few exceptions. First, non-nestmates do not continue to attempt to re-enter the nest indefinitely. Instead, they have a quitting time that is the result of two factors: the need to conserve energy in order to fly back to their own nest (often over 5 km distant for bees living in the wild; Seeley 1985) and the cost of continued attempts, which is higher than for nestmates because the robber is assumed to differ more from the guard’s template and to trigger greater aggression. Persistence in attempting to enter was varied from 1.875 to 30 min. As for returning foragers, robbers have a distinct odor profile that results in an individually distinct probability of rejection. Robbers, however, have an overall mean probability of rejection of 80%, an empirical result from Downs and Ratnieks (2000).

Context 1: fixed guard model

To demonstrate the basic relationship between guard number and invader/nestmate success in entering the nest, a model with fixed numbers of each was constructed. During each run of the model, ten nestmate foragers and five non-nestmate invaders attempt to pass a fixed number of guards. The model was run 100 times. Thus, 1,000 nestmates and 500 invaders attempted to enter the nest for each guard setting.

Context 2: feedback guard model

An expansion of the fixed guard model was constructed such that the number of guards was allowed to fluctuate according to a simple feedback process. The model is based on a full-size Apis mellifera colony (population size ≈ 20,000) and all the parameters are within biologically realistic levels. However, the model is best seen as a heuristic that demonstrates the necessary relationship between weak discrimination ability at the individual level and strong discrimination at the colony level and the ease with which colony-level discrimination can be modulated in accordance with changes in the environment. In other words, the mechanism we chose to ensure that colonies can regulate guard number with robber pressure is arbitrary. It was chosen because it leads to the net result we wish to explore. The mechanism is thus a black box, as in a mathematical model with guard number being a function of robber number multiplied by some parameter.

Six hours of behavioral interactions were simulated. Simulations began with 2 initial guards and 60 nestmate foragers and 20 non-nestmate invaders arriving per minute. At hour 2, the invader arrival rate was increased to 80 per min for 2 h. At hour 4, the arrival rates returned to 20 for invaders. We present the mean results from 50 runs of the model.

Recruitment of new guards

The release of alarm pheromone by disturbed bees triggers defensive behavior (Winston 1987; Moore et al. 1987; Breed et al. 2002). This signal thus provides a simple mechanism for the recruitment of new guards. The short task performance durations of guards provides a mechanism by which guard numbers can decrease in the absence of a threat (Moore et al. 1987). A feedback process was modeled based on these observations. Every time an invader successfully enters the nest, a number of guards are recruited after a variable period of time. For simulations shown in the text, three guards were recruited each time a robber successfully invaded the nest. Guards arrived at the nest entrance after a random normally distributed period of time (3 ± 1.5 min). The quitting time for each guard was set to a randomly chosen value of mean 45 min and sd 15.


Review of empirical studies on nestmate recognition

The fundamental prediction of the optimal threshold individual-level approach to nestmate recognition is a correlation between true and false rejections, because increasing the probability of correct rejection of non-nestmates increases the chance of false rejection of nestmates. Table 1 shows that false rejection of nestmates is exceedingly rare (almost absent overall and zero for most species), whereas the probability of correct (true) rejection of non-nestmates is quite variable. Although the varieties of methods used for quantifying aggression make a meta-analysis inappropriate, the fact that nearly all the studies show no false rejections (no variation in this variable), means there could be no correlation between false rejections and true rejections. Moreover, these data show that the only context in which significant false rejections occur is in guarding the nest entrance in bees and wasps (a context we explore in detail later). These data suggest that individual workers nearly always err on the side of erroneously accepting non-nestmates. Thus, the basic tenet of our hypothesis is supported: workers in diverse species use a permissive per-encounter recognition strategy that does not cause rejection of nestmates, but does lead to a high rate of per-encounter errors of acceptance.

Table 1 Nestmate and non-nestmate rejection rates for species of ants, wasps, and bees

Within-nest model of nestmate recognition

We combined a simple mathematical and conceptual approach with agent-based modeling to explore the behavior of social insects involved in nestmate recognition. Overall, the probability of rejection increases with increasing encounters among workers, taking the probability of rejection per encounter raised to the number of encounters as the overall probability of rejection (Fig. 2). At a rejection probability of 20% per encounter, for example, the overall probability of acceptance falls to less than 1% after 21 encounters. Given the high interaction rates that characterize social insect nests (Gordon and Mehdiabadi 1999; Greene and Gordon 2003); this number of encounters will occur rapidly for most species, meaning a moderate per-encounter probability of rejection could suffice for strong nestmate recognition. Figure 3 shows that worker density, which should be causally related to encounter rate, strongly determines the ultimate rate of rejection in an agent-based model.

Fig. 2
figure 2

Ultimate acceptance probability for non-nestmates having encountered variable numbers of nestmates with a fixed probability of rejection per encounter. The acceptance probability for a given number of encounters is simply the per-encounter rejection rate raised to the number of interactions. For low probabilities of acceptance per encounter, the overall acceptance rate quickly falls to nearly zero, while a much slower rate of decay is exhibited by higher per-encounter probabilities

Fig. 3
figure 3

Results of an agent-based model simulating the time to rejection for invading non-nestmates in a colony without explicit guards. Density of workers within the nest strongly determines encounter rate, which in turn underlies the ultimate probability of rejection. Simulated workers have a 10% probability (mean with individual variation amongst workers) of rejecting non-nestmates per encounter

The high probability of ultimate rejection that arises from relatively low per-encounter rejection probabilities also has important implications for within-nest interactions among nestmates. Because interactions among individuals occur at high frequency in populous social insect colonies (perhaps thousands of interactions per individual per day), very low rates of inappropriate rejection (e.g., 0.05-3.0% per encounter) would translate into appreciable levels of within-nest fighting (Fig. 4). Such fighting is not typically observed (except in the context of guarding the entrance), so a requirement of colonial life may be to set the threshold of acceptance such that no nestmates are rejected (as per Fig. 1).

Fig. 4
figure 4

Ultimate acceptance rate for varying numbers of interactions with a fixed probability of rejection per encounter. Even the slightest probabilities of rejecting nestmates (0.5%) leads to rejection after 100–200 encounters. Thus, in large colonies, even slight probabilities of rejection would lead to continual fighting. A precondition of sociality may therefore be setting the acceptance threshold such that nestmate rejection does not occur

Guarding model

In some social insects, nest defense is not passive, but instead, guards are posted at the colony entrance. Honey bees, for example, stockpile honey in their nests, and during times of dearth, raid one another for these resources (Winston 1987; Downs and Ratnieks 2000). During periods of robbing, highly conspicuous guards patrol the colony entrance inspecting each worker as it attempts to enter the nest. Empirical studies have shown that posting guards decreases the rate of acceptance for both nestmates and non-nestmates, in support of the optimal threshold individual-level approach to nestmate recognition (Downs and Ratnieks 2000; Couvillon et al. 2008). For example, in one study, per-encounter acceptance rates fell from 100% during a time of favorable foraging to approximately 80% for nestmates and 20% for non-nestmates during a period of robbing during a dearth (Downs and Ratnieks 2000). Here, we show that these results are also interpretable within the context of near-perfect collective nestmate recognition. When non-nestmates have an 80% chance of rejection per encounter, nearly no non-nestmates gain entry when they encounter at least three guards (Fig. 2). Further, at this number of encounters with guards, nestmates, who have a 20% chance of rejection per encounter, should only be rejected about 50% of the time. Hence, if rejected individuals try repeatedly to gain entry, then nestmates should be able to gain entrance reasonably quickly, while non-nestmates (who have nearly no probability of acceptance per attempt) should not be able to gain entry at all. This result should be general as long as there is a strong difference between the rate of acceptance for nestmates and robbers and as long as the number of guards can be modulated. We constructed a model to test this hypothesis.

The results of an agent-based model using fixed numbers of guards, returning foragers, and non-nestmates supports the hypothesis that guards create a selectively permeable barrier at the entrance that restricts non-nestmates, but allows in nestmates so long as there is a strong difference in acceptance rates between nestmates and robbers (Fig. 5). This barrier comes at a cost, however, because as the number of guards increases, returning foragers (nestmates) are increasingly delayed before entering the colony (Fig. 5).

Fig. 5
figure 5

Probability of ultimate acceptance and time to entry for 10 nestmates and 5 non-nestmates attempting to enter a honey bee nest with guards. When non-nestmate and nestmate per-encounter rejection rates are strongly dissimilar (80% and 20% in present case), it is possible for a colony to ensure next to no admittance of non-nestmates by regulating the number of guards at the entrance. This comes at a price, however, as increasing numbers of guards increase the time to admittance for nestmates

In many species of social insects, guarding behavior may be inducible, as when workers emit an alarm pheromone that activates a systemic colony defense response. Incorporation of this behavior into our model results in a dynamic process of nest defense that optimally allocates guards in accordance to the threat level. In particular, when the number of invaders increases, so does the number of guards, such that the colony maintains a low rate of invader success (Figs. 6 and 7). Further, as Fig. 6 shows, some invaders always penetrate the nest, by chance alone, thus leading to a background rate of robbing. In addition, when a colony is seriously attacked, its guards may be killed by the robbers, thus allowing the robbing of a colony. Hence, near-perfect nestmate recognition ability does not necessarily translate into the ability to defend oneself.

Fig. 6
figure 6

Simulation of colony defense during a period of robbing. A feedback mechanism, based on the recruitment of guards via alarm pheromone, is shown to be capable of collectively regulating nestmate recognition. Simulations start with two guards; 20 invaders and 60 nestmates arrive per minute. At hour 2, the invader arrival rate is increased to 80 per minute for 2 h. Mean per minute acceptance rates are shown. The acceptance rate for invaders is relatively high for a few minutes before additional guards have been recruited, at which time it falls drastically. When the invader arrival rate is increased, acceptance rate increases briefly before returning to below 1 per minute

Fig. 7
figure 7

Mean (±sd) numbers of guards, foragers, and invaders at the entrance over time for the same simulations shown in Fig. 6. The number of guards quickly rises, before reaching a stable level. When the number of invaders increases, the number of guards rises weakly. However, the number of foragers waiting to get into the nest strongly rises when the number of guards is increased


This study advances our understanding in two ways. First, it shows that physiological trade-offs, which can create severe barriers to adaptation at the individual level (Reeve 1989), can be easily overcome by collective decision making (Marshall and Franks 2009). Although individual workers cannot exclude all non-nestmates without also excluding nestmates (as suggested by previous studies), we show that colonies can because they have multiple opportunities to make the assessment. Thus, a colony can accomplish a task that an individual cannot, a central tenet of collective decision making (Seeley et al. 2006; Marshall and Franks 2009). Further, although much of this study deals with honey bees, the factors modeled are common to all social insects, so the results should be broadly applicable. In addition, Table 1 reviews mainly work on ants, showing that this hypothesis may be equally applicable to ants, as well as bees and wasps. Finally, this collective decision-making result has consequences for the physiological study of nestmate recognition because it suggests that for many species there may only be weak selection pressure for sophisticated individual-level recognition ability.

Although our study is primarily focused on the general theoretical underpinnings of nestmate recognition, we found theoretical support for some interesting hypotheses related to the natural history of nestmate recognition in ants and bees. First, we found that a worker trying to invade another colony faces a daunting task, as it must evade detection almost continuously as it moves through the nest (Fig. 3). This is perhaps why some species do not post guards. They are unnecessary, as penetrating a nest involves numerous encounters between an invader and the host workers. Second, posting dedicated guards at the nest entrance is likely a costly trait. This is because, although guarding allows for stringent nest defense, it carries the cost of disrupting the colony’s own foraging, as guards inspect and falsely reject nestmates (Fig. 5). Therefore, it might not be cost effective to post guards in many contexts in which the cost in lost foraging outweighs the benefits of preventing robbing. This may be why colonies, such as honey bees, only intermittently post guards (Downs and Ratnieks 2000).

Previous research predicts a correlation between true and false rejections due to the similarity of the odor profiles of nestmates and non-nestmates (Reeve 1989). That work assumed each individual-level error carried with it a cost, however, and as we show here, when this assumption is relaxed, as the biological context suggests it must, the prediction changes. When a colony gets many chances to make the correct decision, it does better than an individual who gets only one chance. Table 1 supports this by showing the permissive individual-level acceptance thresholds of species of ants, wasps, and bees. Although the basic prediction of the individual-level and collective approach differ, there is some overlap, however. Studies of bees and wasps have shown significant nestmate rejection in some contexts (guarding) predicted by the individual-level approach (Downs and Ratnieks 2000; Couvillon et al. 2008). When we modeled this guarding behavior as a collective process, we found that colonies are nevertheless still making near-perfect decisions. Thus, when individuals lower their acceptance thresholds (causing the false rejection of nestmates at the per-encounter level), it is best interpreted as a colony-level adaptation to create a selectively permeable barrier at the nest entrance, rather than as an individual-level adaptation focused on optimizing the costs and benefits of accepting nestmates versus non-nestmates. In general, the present work suggests that colonies should typically be capable of near-perfect nestmate recognition, in spite of the physiological constraints at the individual level. However, strong nestmate recognition may not always be cost effective, as it slows the colony’s foraging rate. Thus, nestmate recognition, like most social insect problems, is a task allocation problem (Seeley 1995). As for every task, nestmate recognition has a rich proximate basis, but the individual-level characteristics of the workers must always be considered from a colony-level perspective when questions of adaptation are considered.

It has previously been argued that context-dependent nestmate recognition is a solution to the problem modeled by our within-nest nestmate recognition model (Errard et al. 2006; Lenoir et al. 2009; Ozaki et al. 2005). From this perspective, they argue that the problem of multiple interactions potentially leading to large numbers of false rejections in the nest, and hence within-nest fighting, is solved by only performing nestmate recognition outside the nest and never within it. This is an intriguing alternative to our solution to this problem, which is nevertheless not mutually exclusive with the main point of this study. This is because the studies reviewed in Table 1 were not conducted within nests. Thus, low rejection of nestmates is something that occurs both outside and inside the nest. Hence, our result that multiple interactions can lead to near-perfect nestmate recognition at the colony entrance is independent of the notion of context-dependent nestmate recognition.

Because a colony of weak discriminators is more effective than an individual with even maximal recognition ability (given physiological constraints), there may be less selective pressure at the individual level for increased recognition ability than previously thought. However, a greater ability, at the individual level, for accurate nestmate discrimination is adaptive, as it decreases the time (number of encounters) to rejection for non-nestmates (Fig. 2). Hence, we might expect a range of individual-level recognition abilities resulting from the need of each species at the colony level to solve their unique nestmate and parasite recognition problems (Martin et al. 2010). Studies show a broad range of recognition abilities (reviewed in Table 1) which, although known to correlate with the complexity of the genetic and environmental cues being used, also supports this prediction. This range of individual-level discrimination abilities between species may also magnify the difficulty of determining the neural mechanisms by which nestmate recognition occurs, as mechanisms sufficient for producing weak recognition ability could be different from those that are considerably stronger. It is also likely that neuronal mechanisms may strongly vary between species, as there are likely many ways to generate the weak recognition abilities social insects possess and because nestmate recognition systems are not likely to have been inherited intact from a common ancestor.