Introduction

Mating between close relatives usually results in a reduction in fitness, known as inbreeding depression (Darwin 1877). Inbreeding depression is the result of an increase in genetic homozygosity which allows for deleterious recessive alleles to be expressed and reduces heterozygote advantage (Charlesworth and Willis 2009). The consequences of inbreeding depression include reduced survival, growth and reproductive success, and higher susceptibility to diseases (Wells et al. 2018). Consequently, many species that are at risk of substantial inbreeding depression have evolved mechanisms by which to avoid inbreeding (Pusey and Wolf 1996) (although exceptions occur; see Kokko and Ots 2006; Puurtinen 2011; Wang and Lu 2011; Szulkin et al. 2013; Duthie and Reid 2016).

The dispersal of one or both sexes away from the natal site prior to reproductive maturity occurs in the majority of vertebrates and reduces the probability of inbreeding by minimising encounters between close relatives (Waser and DeWoody 2006; Waser et al. 2012). However, in some species, populations are viscous and dispersal is not sufficient to prevent contact between relatives. For example, in cooperatively breeding species, sexually mature offspring (known as “helpers”) often remain with their parents and assist in rearing subsequent young (Stacey and Koenig 1990; Koenig et al. 1992). Under such circumstances, the pool of mates available to an individual (from within and/or outside of the social group) is likely to vary in relatedness. Inbreeding avoidance therefore requires some form of kin discrimination (Pusey and Wolf 1996; Cornwallis et al. 2009), whereby behavioural responses of an individual differ towards kin versus non-kin, based on cues correlated with kinship (Waldman et al. 1988; Tang-Martinez 2001; Nichols 2017).

A cue that usually correlates well with kinship is familiarity, particularly during early life (Fadao et al. 2000). Animals can learn the unique phenotypic traits of individuals they frequently encounter during early development such as parents, helpers, or siblings (Halpin 1991; Berger et al. 1997). These individuals can then be recognised as likely kin during subsequent encounters (Halpin 1991; Berger et al. 1997). As early-life encounter rates and relatedness usually covary, this mechanism has been shown to be effective in avoiding incestuous matings in both laboratory studies (Blaustein and O’hara 1982; Bateson 1983; Holmes and Sherman 1983; Frommen et al. 2007) and in the wild (Berger and Cunningham 1987). For example, cross-fostering experiments in wild Seychelles warblers (Acrocephalus sechellensis) have shown that young birds treat individuals that tended them at the nest as close relatives, regardless of actual genetic relatedness (Komdeur et al. 2004). Similarly, long-tailed tit (Aegithalos caudatus) nestlings learn and emulate the calls of their parents and helpers that provision them prior to fledging, and use these calls to identify kin later in life (Sharp et al. 2005; Leedale et al. 2020).

Under some circumstances, familiarity may not be a good indicator of relatedness, such as when individuals have a high likelihood of encountering unfamiliar relatives or when individuals that substantially vary in relatedness are likely to be equally familiar (Le Vin et al. 2010; Leclaire et al. 2013). Here, alternative kin recognition mechanisms are likely to be beneficial (Nichols 2017). One possibility is that animals can use behavioural rules to avoid inbreeding, such as avoiding mating in locations that are frequented by close relatives. This may explain why individuals often refrain from breeding in their natal territory (Harrison et al. 2013) or seek extra-group mates when they do breed in their natal territory (Brouwer et al. 2011). For example, in red-winged fairy wrens, a female inheriting a territory may come into contact with her father or brothers, including those unfamiliar to her due to extra-group paternity. Such females are more likely to seek extra-group paternity or travel further when pursuing extra-group paternity compared to females that disperse prior to breeding (Brouwer et al. 2011). The resultant offspring from these extra-pair matings are less likely to be inbred than within-pair offspring (Hajduk et al. 2018). Another behavioural rule is shown in acorn woodpeckers (Melanerpes formicivorus) where females mate with immigrant males that have arrived after the female was born (Koenig and Pitelka 1979). It is also conceivable that females bias mating against certain age groups that are more likely to contain close relatives, but this possibility has rarely been investigated (Nichols 2017).

Alternatively, individuals could identify unfamiliar relatives via phenotype matching. Phenotype matching occurs when individuals use knowledge of their own phenotypic traits or the traits of their familiar close relatives (e.g. their mother or littermates) to build a “kin template”. Comparison of newly encountered individuals against this template permits the identification of likely relatives even when they have not encountered them previously (Holmes and Sherman 1982; Schausberger 2007). This mechanism relies on a close correlation between phenotypic and genetic similarity and has been proposed to occur primarily based on odour cues (Mateo and Johnston 2000). Evidence for phenotype matching in the absence of familiarity has been found in numerous species including rainbow trout (Oncorhynchus mykiss) (Brown et al. 1993), golden hamsters (Mesocricetus auratus) (Mateo and Johnston 2000), and among cooperatively breeding cichlids (Neolamprologus pulcher) (Le Vin et al. 2010) and meerkats (Suricata suricatta) (Leclaire et al. 2013).

Debate continues over the exact circumstances under which animals use particular inbreeding avoidance mechanisms over others (Nichols 2017). In general, it is expected that this is dependent on the cost–benefit ratios of different mechanisms, including the extent to which different mechanisms introduce errors into the estimation of relatedness, their reliability (Duncan et al. 2019). For example, where cues based on familiarity or spatial distribution are either unavailable or may lead to inaccurate assessments of relatedness, phenotype matching may be favoured over other mechanisms (Holmes and Sherman 1982). In some cases, multiple mechanisms may be expected to evolve where one alone cannot sufficiently reduce the level of inbreeding. This is shown in western bluebirds (Sialia mexicana) where inbreeding is avoided both through kin recognition within winter groups and through sex-biased dispersal (Dickinson et al. 2016). On the other hand, in cases where submaximal mechanisms are able to reduce the level of inbreeding depression to a tolerable level, stronger inbreeding avoidance mechanisms may not necessarily be expected to evolve (Szulkin et al. 2013).

Banded mongooses (Mungos mungo) present an excellent opportunity to investigate mate choice rules in a species where identifying relatives through familiarity is likely to be challenging. This species lives in tight-knit stable family groups normally consisting of 10–30 individuals (Cant 2000). New groups form when a cohort of males from one group fuses with a cohort of females from another group, so group founders have unrelated breeding partners available (Nichols et al. 2012c). However, once groups are formed, dispersal is rare and both sexes breed in their natal group despite the presence of close relatives including parents and siblings (Gilchrist et al. 2004; Nichols et al. 2014). This leads to high levels of inbreeding; 66.4% of individuals have non-zero inbreeding coefficients, 12.9% are the result of moderate inbreeding, and 7.1% are the product of first-degree inbreeding (Wells et al. 2018). While rates of moderate inbreeding are analogous to other cooperative mammals, namely black-tailed prairie dogs (Cynomys ludovicianus; 26%, Hoogland 1992) and meerkats (Suricata suricatta; 15%, Nielson et al. 2012), first-order inbreeding is unusually common in banded mongooses compared to other group-living species (Koenig and Haydock 2004; Nichols 2017). Inbreeding depression occurs in yearling weight and male reproductive success, with banded mongoose males whose parents are first degree relatives fathering on average 79% fewer offspring than outbred males (Wells et al. 2018). Furthermore, moderate inbreeding is also costly as yearling weight decreases linearly as the level of inbreeding increases (Sanderson et al. 2015; Wells et al. 2018). Inbreeding also depresses the survival of pups that receive below-average levels of care and reduces contributions to cooperative care in adults; the severity of this again appears to vary linearly according to the level of inbreeding (Wells et al. 2020). Therefore, even moderate levels of inbreeding are costly in banded mongooses so inbreeding depression is expected to exert a substantial selective pressure on individuals to avoid inbreeding.

There are several mechanisms that banded mongooses could use to avoid inbreeding. First, they could disperse to form new groups, and while this would greatly reduce the relatedness between males and females (mean opposite-sex relatedness in new groups =  − 0.021 (SE ± 0.029) (Nichols et al. 2012c)), it would be highly costly to do so, as members of new groups suffer significantly higher mortality rates (Cant et al. 2013). Secondly, they could selectively mate with unrelated immigrants, as is common in other social species such as meerkats (O’Riain et al. 2000) and pied babblers (Nielson et al. 2012). However, this strategy is unlikely to be effective in banded mongooses as immigration into existing groups is almost absent (Cant et al. 2013) so mating with unrelated immigrants is rarely if ever possible. Second, they could mate with extra-group individuals as close relatives are rarely found in other social groups (Nichols et al. 2012c). Indeed, extra-group breeding does occur, with around 18% of litters containing extra-pair young, and it successfully prevents inbreeding, as the resulting pups have an average inbreeding coefficient of zero (Wells et al. 2020). However, extra-group breeding is also associated with high costs, as 15% of deaths from known causes are the result of aggressive interactions between groups, which may limit its effectiveness as an inbreeding avoidance strategy (Nichols et al. 2015).

While extra-group mating may be an important mechanism of inbreeding avoidance in banded mongooses, inbreeding avoidance has also been shown to occur within social groups, even after accounting for extra-group breeding (Sanderson et al. 2015); the relatedness between within-group breeding pairs was significantly lower (0.15) than if they paired randomly (0.18). This is possible because the level of relatedness between group members varies considerably (from zero to 0.5 or higher), which provides the potential for individuals to both recognise and avoid mating with close relatives while remaining in their natal group. This suggests that within-group inbreeding avoidance mechanisms do exist in this species, but we do not yet know what these mechanisms are, and identifying them is the focus of our study.

The use of familiarity as a cue to avoid mating with relatives may pose a particular challenge for banded mongooses as typical cues for familiarity-based kin recognition are likely to be “scrambled” by the mating system and provision of cooperative care. Specifically, the majority of reproductively mature females within the group come into oestrus within a few days of each other and often mate promiscuously (Cant 2000; Gilchrist 2001; Hodge et al. 2011). Subsequently, they give birth in tight synchrony (usually on the same night) in an underground den, and immediately after birth their offspring are combined into a large communal litter of mixed parentage (Gilchrist 2006). The communal litter is raised by the whole group (Cant 2003), including both parents and non-parents (Gilchrist and Russell 2007). Parents do not direct help towards their own pups, and individual pups have been observed to suckle from multiple females (Cant 2003; Gilchrist et al. 2004; Gilchrist 2006; Vitikainen et al. 2017). In this species, it is therefore possible that familiarity resulting from the early-life provision of care or associations between communal littermates may only be weakly correlated with relatedness, if at all. This unusual care system therefore leads to a potential disconnect between those groupmates that an individual becomes more familiar with in early life and those to which it is more closely related. We are thus presented with an excellent opportunity to study the mechanisms of inbreeding avoidance in a system where not only do we have extensive knowledge on the costs of inbreeding but also where cues that are important in many species are likely to be unreliable.

Here, we investigated mechanisms of inbreeding avoidance within banded mongoose groups using 24 years of behavioural and life history data combined with a genetic pedigree. Specifically, we compared realised within-group breeding patterns to those expected under different mate choice rules (Table 1), following the approach of Sanderson et al. (2015). First, we evaluated the magnitude of inbreeding avoidance within banded mongoose social group reproducing analysis conducted by Sanderson et al. (2015). We then extended this analysis by testing the hypotheses that inbreeding is avoided by (i) not breeding with individuals that are familiar from early life or during adulthood, namely communal littermates, maternal littermates (males that they shared a womb with), carers and other individuals born in the same group, and potential sons (males born into communal litters the dam gave birth into); and (ii) through age-assortative breeding patterns that have previously been identified in banded mongoose groups, such as females breeding either with older males or with partners of similar age (Cant 2000; Nichols et al. 2010). In some species, patterns of relatedness change with age (Johnstone and Cant 2010; Nichols et al. 2020) and so age-related rules for mating could help reduce inbreeding. In banded mongooses, the relatedness of males to the rest of the group declines with age (S. Ellis, pers. comm.) so females are potentially less related to older males. It is also possible that inbreeding is avoided using multiple mechanisms; we then tested whether (iii) a combination of the two best mechanisms reduced inbreeding to the levels observed.

Table 1 Summary of the inbreeding avoidance hypotheses tested for in this study, including the mechanism and justification for how each mechanism could reduce levels of inbreeding
Full size table

Methods

Study site and data collection

Behavioural, genetic, and life history data were collected from a population of wild mongooses (Mungos mungo) residing in Mweya, Queen Elizabeth National Park, Uganda (0°12ʹS, 27°54ʹE) between May 1997 and October 2018. This population consisted of approximately 200 mongooses at any one time, split into 10–12 cohesive social groups (Cant et al. 2013). Individuals were identifiable in the field due to the application of unique patterns of hair dye (L’Oreal, UK) or a unique shave pattern in their fur. Some fully grown individuals were fitted with colour-coded plastic collars for identification. To maintain dye markings, shave patterns, and collars, all individuals were trapped and anaesthetised every 3–6 months as described previously (Cant 2000; Hodge 2007; Jordan et al. 2010). Upon first capture, individuals were given either a unique tattoo or a subcutaneous pit tag (TAG-P-122IJ, Wyre Micro Design Ltd., UK) to allow permanent identification. For genetic analysis, a 2-mm tissue sample was taken from the tip of the tail using surgical scissors, which was stored refrigerated in 70% or 96% ethanol, and a topical application of dilute solution of potassium permanganate was applied to minimise infection risk.

All individuals in the population were habituated to human observers at < 10 m. Social groups could be reliably located using 27 g radio collars (< 2% of body mass, Sirtrack Ltd., New Zealand) attached to 1 or 2 individuals per group. Each social group was visited every 1–3 days to collect behavioural and life history data. Groups have an age-based dominance hierarchy, with the oldest individuals in the group (between 1–5 dominant males and 3–7 dominant females) forming a breeding “core” that reproduce approximately 3–4 times annually. Groups also contain a variable number of younger subordinate individuals that breed alongside dominants when environmental conditions are favourable (Cant et al. 2010; Nichols et al. 2010, 2012b, c). Multiple females within each group give birth synchronously, resulting in communal litters of mixed parentage (Gilchrist 2006). Dates of birth were inferred from changes in female weight and body shape as well as from the onset of pup care behaviour. It was not possible to record data blind because our study involved focal animals in the field.

Two main forms of pup care exist: babysitting and escorting (Cant et al. 2013). Babysitting occurs when one of more adults remain at the natal den while the rest of the group forages, where they guard the young pups from predators and rival groups (Cant et al. 2013). We recorded an individual as being a babysitter of a litter if they were observed at the den with the pups while the main group was more than 100 m away, or if they were recorded as being absent from the foraging group. When the pups emerge from the den at approximately 30 days of age, a one-to-one caring relationship is established between a pup and a specific helper known as an escort (Cant et al. 2013). These escorts protect, feed, carry, and groom the pup until it reaches approximately 90 days of age (Cant et al. 2013). Escorting is a conspicuous behaviour that is easily identified. We considered an individual to be escorting a pup if it spent more than 50% of a focal observation session within 30 cm of a particular pup. Observation sessions of babysitting and escorting were at least 20 min in duration. Breeding and non-breeding individuals of both sexes contribute to escorting, but parents are more likely to escort than non-parents (Gilchrist and Russell 2007). However, parents do not direct care towards their own pups, instead escorting is disproportionately directed towards members of the escort’s own sex (Vitikainen et al. 2017).

Relatedness analysis

To quantify relatedness, we used a pedigree of 1940 individuals constructed by Wells et al. (2018) based on genotypes at 35–43 microsatellite loci. The pedigree was 9 generations deep and contained 1725 maternities and 1625 paternities. Genotyping methods are described in full in Sanderson et al. (2015) and Wells et al. (2018). Briefly, DNA was extracted from tissue samples using Qiagen® DNeasy blood and tissue kits following the manufacturer’s instructions or (pre-2010) by lysis in proteinase K followed by a phenol–chloroform extraction (Sambrook et al. 1989). Up until 2010, the loci were amplified individually and visualised through radioactive incorporation (Nichols et al. 2012c). More recent samples were genotyped using a Type It kit (Qiagen®) according to the manufacturer’s protocol with an annealing temperature of 57 °C and a reaction volume of 12 µl. PCR products were resolved by electrophoresis on a capillary sequencer. Parentage analysis was conducted using MASTERBAYES (Hadfield et al. 2006) and COLONY (Jones and Wang 2010) as described in detail in Sanderson et al. (2015) and Wells et al. (2018).

Statistical analyses

It was rarely possible to observe matings directly as they often occurred out of sight of observers, for example in dense undergrowth. We therefore used realised breeding patterns obtained from the pedigree to investigate potential inbreeding avoidance mechanisms within groups. To investigate inbreeding avoidance strategies in the banded mongoose, we compared the mean relatedness of males and females within observed breeding pairs in banded mongoose social groups to (1) expected patterns of relatedness if females mated randomly within their groups and (2) patterns of relatedness if females bred according to a particular mate choice rule. A randomisation approach was used to obtain the distributions of relatedness expected under within-group random mating as well as under different mate choice rules (described below) using R 3.3.1 (R Studio Team 2016).

To calculate the observed mean level of relatedness within banded mongoose pairs, we first created a list of mother–father pairs by extracting all pups with both parents assigned from the pedigree. Multiple pups are often produced by the same parents within a litter (full siblings). These are likely to be the product of the same mating (or group of matings) and are therefore non-independent. To account for this, if a dam had multiple pups sired by the same sire within a litter, we counted this as a single male–female breeding pair. If a female had pups assigned to more than one male within a litter, she must have mated with multiple males, so we included each sire–dam dyad as a breeding pair. Similarly, if the same breeding pair was identified in different breeding attempts (producing full siblings over multiple litters), these litters resulted from different matings and therefore the dyad was included in the list once for each breeding attempt in which they were identified as a breeding pair. Data were available from a total of 959 breeding pairs.

We subsequently reduced this dataset to 351 breeding pairs comprising 104 males and 107 females across 10 social groups based on a set of criteria previously applied by Sanderson et al. (2015). Specifically, (1) the sire had to belong to the same social group as the dam at the time of conception (773/959 breeding pairs); (2) both of the dam’s parents had to be assigned at ≥ 80% confidence (613/959 breeding pairs); (3) at least 80% of the potential sires (including the sire of the pup and other males present in the same group as the dam that could be candidate fathers) had both parents assigned at ≥ 80% confidence (530/959 breeding pairs). Here, the confidence level of the assignment represents the marginal posterior probability of the parentage assignment in a Bayesian framework. As we were interested in within-group inbreeding avoidance, the first criterion allowed us to exclude any effects of extra-group mating. Exclusion criteria two and three minimised noise arising from using relatedness coefficients from individuals with obscure parentage. All three criteria were met by 351/959 breeding pairs, which were used in our randomisation analyses.

In order to compare observed breeding patterns to those expected under different mate choice rules, we conducted randomisations. A single randomisation involved selecting a potential sire for each of the dams within the breeding pairs (up to 351 pairs, but for some scenarios a subset of pairs was excluded due to missing data; explanations for these exclusions are given in the relevant sections below). The sire was selected from a list of males present in the same social group as the female and that were at least one year old on the date she likely conceived (60 days prior to the birth of the litter). The relatedness value between the dam and the potential sire was extracted and a mean relatedness value across all breeding pairs was calculated. This procedure was repeated 10,000 times to generate a distribution of mean relatedness values. For each mate choice rule that we tested, we compared the empirical mean relatedness values between observed breeding pairs to (i) a null distribution of 10,000 relatedness values generated under random within-group mating, whereby each male present in the list of potential sires had an equal chance of being selected and to (ii) a distribution of 10,000 relatedness values generated by the mate choice rule being evaluated (whereby some males were more or less likely to be selected than others). P-values were derived from the proportion of randomisations that were at least as extreme as the observed value. This method is somewhat analogous to a two-tailed one-sample t-test but uses the empirical distribution of randomised relatedness values for the null distribution instead of a theoretical distribution to calculate whether the observed value is significantly different from this distribution of relatedness values. Where the null distribution and the distribution generated by a mate choice rule showed substantial overlap, we used Kolmogorov–Smirnov tests to determine whether the two distributions were significantly different.

The randomisations used to simulate mate choice rules are described below. If inbreeding avoidance in banded mongooses was based on one of the rules tested, we would expect the empirical mean relatedness of breeding pairs to fall within the distribution of mean relatedness values produced by the randomisation.

To what extent is inbreeding avoided within banded mongoose social groups?

We quantified the magnitude of inbreeding avoidance within banded mongoose groups by comparing the empirical mean relatedness values of 351 breeding pairs to (i) the null distribution of relatedness values that would be expected from random within group mating, (ii) the distribution of relatedness values that would be expected if females were less likely to mate with related males, and (iii) the mean relatedness value that would be obtained if all females optimally avoided inbreeding by always breeding with the least-related available within-group male. For distribution (ii), we simulated submaximal inbreeding avoidance by weighing the probability of a sire being selected in inverse proportion to his relatedness to the female such that as relatedness increases, the potential sire is less likely to be successful in mating with the female. For some breeding pairs, the relatedness value of the dam and the potential sire was zero. Therefore, in order to avoid assigning a weighing of zero, which would prevent that pairing from entering the randomisations, we added an arbitrary relatedness value of 0.25 to the weighing of relatedness values for all pairs (i.e. creating a zero offset). Note that the value of the offset does not affect the results as the distribution of differences in relatedness between the dam and potential sires remains identical. Randomisations were then carried out using all 351 breeding pairs.

Is inbreeding avoidance based on familiarity?

We tested six familiarity-based hypotheses based on our prior knowledge of the banded mongoose social system (Table 1). During infancy, banded mongooses associate with their communal littermates and carers to a greater degree than other group members, and hence these categories of individuals may be avoided as mates in later life. Furthermore, inbreeding could potentially be minimised by avoiding breeding with animals born into the same social group (birth pack) or by avoiding males that could potentially be their sons. We test these possibilities as described below.

Do females reduce inbreeding by avoiding communal littermates?

To determine whether females avoid breeding with their male communal littermates (i.e. males born and raised in the same mixed-parentage litter as the female, that likely includes brothers), we conducted randomisations as described above but removed males from the potential sire list that were born into the same communal litter as the female. Each female was therefore assigned a random sire from their social group that was not a member of her communal litter. The litter for one dam was unknown and so this breeding pair was removed (n = 350 breeding pairs).

Do females reduce inbreeding by avoiding maternal littermates?

It is possible that cues to relatedness could be learnt in very early life (Sharp et al. 2005). For example it may be possible to recognise littermates that shared a womb using olfactory cues (Robinson and Smotherman 1991). To test whether females may avoid breeding with males that they shared a womb with, we removed males from the potential sires list that were born into the same maternal litter as the female (i.e. those that were born to the same mother on the same day). The litter for one dam was unknown and so this breeding pair was removed (n = 350 breeding pairs).

Do females reduce inbreeding by avoiding potential sons?

To determine whether females avoid breeding with potential sons, we removed males from the potential sire list that were recorded as being born into the same communal litters that the female gave birth into. This resulted in there being no potential sires remaining for two dams and so these breeding pairs were removed (n = 349 breeding pairs).

Do females reduce inbreeding by avoiding babysitters?

To determine whether females avoid breeding with their babysitters, we removed males from the potential sire list that were recorded as having babysat the dam’s litter of birth. Babysitting data were available for the dams of 258 of the 351 breeding pairs (these data were not collected until February 2000 so were not available for all females). Removing all males that were recorded as babysitters of the dam’s litter resulted in there being no potential sires remaining for 9 dams. We therefore removed these breeding pairs from the dataset (n = 249 breeding pairs).

Do females reduce inbreeding by avoiding escorts?

To determine whether females avoid breeding with their male escorts, we removed males from the potential sire list if they were recorded as having escorted the dam. Escorting data were available for 147 of the 351 breeding pairs (these data were not collected before June 2000 so were not available for all pups). Although each pup usually has only one escort, some pups were observed moving between several escorts (Sheppard et al. 2018). Consequently, in some cases we removed multiple potential sires from the simulations, which led to one dam with no more potential sires remaining, which was excluded (n = 146 breeding pairs).

Do females reduce inbreeding by avoiding males born into the same group as themselves?

To investigate whether females avoid breeding with males born in the same natal group, we removed all males from the potential sire list that were born in the same group as the female. Data on the group of birth were not available for one dam so this breeding pair was excluded. After removing potential sires from the same birth pack, 267/350 (76.3%) of dams did not have any potential sires remaining. This strongly suggests that this strategy is not a feasible inbreeding avoidance strategy in banded mongooses as it would leave many breeding females without any mating options. We therefore did not consider it worthwhile to carry on investigating this inbreeding avoidance strategy.

Is inbreeding avoidance based on age-related rules?

While familiarity appears to be one of the most common methods of kin discrimination in cooperative breeders (Komdeur et al. 2008; Jamieson et al. 2009), it is possible that banded mongooses use rules that are not based on familiarity. For example, females might select males of a particular age group that are less likely to be close relatives (Nichols 2017). In some species, patterns of relatedness change with age (Johnstone and Cant 2010; Nichols et al. 2020) and so age-related rules for mating could help reduce inbreeding. We tested two possible age-based rules (Table 1).

Do females reduce inbreeding by preferentially breeding with older males?

Male banded mongooses show moderately high reproductive skew with an average of 85% of pups sired by the oldest three males in the group, and lower ranking males having low reproductive success (Nichols et al. 2010), although skew likely varies between social groups and breeding attempts. Furthermore, on average, an individual’s relatedness to the rest of the group declines with age (S. Ellis, pers. comm.). To determine whether this pattern could result in inbreeding avoidance, all potential sires for each dam were ranked in order of age (a male’s rank was the number of older males in the social group + 1), and all males with a rank below 3 were removed from the list of potential sires. As it was possible to identify the top-ranking males for each dam, all 351 breeding pairs were used for this randomisation.

Do females reduce inbreeding by preferentially breeding with males of a similar age?

Despite older males having greater reproductive success, there is some evidence that male and female banded mongooses mate with individuals that are similar in age to themselves. Specifically, older and more dominant males tend to breed with older females, which in turn produce more pups (Nichols et al. 2010). To investigate whether inbreeding avoidance could be mediated by this pattern, we implemented randomisations sampling potential sires in reverse proportion to their absolute difference in age from the female (i.e. biasing the randomisations towards males of a similar age to the female). One breeding pair where the female did not have a birth date recorded was removed from this analysis (n = 350 breeding pairs).

Is inbreeding avoidance based on multiple mechanisms?

It is possible that females may use multiple mechanisms to avoid inbreeding. In order to test this hypothesis, we combined the two potential mechanisms that resulted in relatedness distributions that best fitted the observed value (females preferentially breeding with older males and females avoiding mating with potential sons). Consequently, we ran a randomisation only using the top three ranking males and excluded any males that the female could have potentially sired. This resulted in two breeding pairs with no potential sires, which were removed from this analysis (n = 349).

Results

To what extent is inbreeding avoided within banded mongoose social groups?

Levels of inbreeding in banded mongoose social groups are lower than would be expected under random within-group mating, as the observed mean relatedness of breeding pairs was significantly below that of randomised pairs (observed mean r = 0.144, within-group random mating mean r = 0.176, 95% CI: 0.160–0.191, P < 0.0001, Fig. 1). However, observed inbreeding levels were considerably higher than those expected under maximum inbreeding avoidance, defined as when females always breed with the least-related available within-group male (observed mean r = 0.144, maximum inbreeding avoidance r = 0.021, P < 0.0001, Fig. 1). Interestingly, the observed mean pairwise relatedness value for breeding pairs fell within the expected distribution of our randomisation in which males were selected as sires in inverse proportion to their relatedness to the female (observed mean r = 0.144, randomisation mean r = 0.136, 95% CI: 0.123–0.150, P = 0.252, Fig. 1). This suggests that females may reduce inbreeding within their social groups by using one or more proxies of genetic relatedness, although inbreeding is not completely avoided.

Fig. 1
figure 1

Banded mongooses show reduced inbreeding in comparison to within-group random mating, but they still inbreed frequently. The observed mean relatedness of breeding pairs (red arrow) is significantly below that expected under within-group random mating (grey histogram) and falls within the 95% CI of the randomised distribution if females preferentially breed with males less related to the females (yellow histogram). Mean relatedness under maximum inbreeding avoidance (blue arrow), in which females always bred with the least related available male groupmate, was significantly lower than both randomised distributions

Full size image

Is inbreeding avoidance based on familiarity?

Our randomisation based on females avoiding breeding with communal litter mates resulted in a projected marginal decrease in relatedness between the breeding pairs in comparison to within group random mating (Kolmogorov–Smirnov test; D(10,000) = 0.1459, P < 0.0001; see the grey and yellow shaded histograms in Fig. 2a). However, this was not sufficient to explain inbreeding avoidance as the observed mean relatedness value was significantly lower than the distribution of values expected when females avoid breeding with their communal littermates (observed mean r = 0.144, mean r based on the randomisation avoiding communal littermates = 0.173, 95% CI: 0.157–0.188, P < 0.0001; Fig. 2a).

Fig. 2
figure 2

The results of simulation-based tests of familiarity-based inbreeding avoidance hypotheses. Yellow histograms represent distributions of mean relatedness between males and females generated from randomisations whereby a females avoided breeding with communal littermates, b females avoid breeding with maternal littermates, c females avoided breeding with males born into communal litters they had given birth into, d females avoided breeding with males that babysat them when they were pups, and e females avoided breeding with males that escorted them. Grey histograms represent within-group random mating and red arrows show the empirical mean relatedness value for breeding pairs

Full size image

Similarly, avoiding maternal littermates resulted in a slight decrease in relatedness between breeding pairs compared to random within-group mating (Kolmogorov–Smirnov test; D(10,000) = 0.3193, P < 0.0001; Fig. 2b). However, this was not enough to explain the observed relatedness value (observed mean relatedness r = 0.144, mean r based on avoiding maternal littermates = 0.179, 95% CI: 0.164–0.194, P < 0.0001; Fig. 2b). Avoiding potential sons also reduced the level of relatedness between breeding pairs (Kolmogorov–Smirnov test; D(10,000) = 0.7395, P < 0.0001; Fig. 2c). Again, this was not sufficient to explain the observed relatedness value (observed mean relatedness r = 0.143, mean r based on avoiding potential sons = 0.158, 95% CI: 0.145–0.172, P = 0.0248).

There was also no evidence that females reduced inbreeding by avoiding breeding with their babysitters; the mean relatedness value between observed breeding pairs was not within the distribution of values generated from randomisations avoiding babysitters (observed mean r = 0.160, mean r based on the randomisation avoiding babysitters = 0.201, 95% CI: 0.183–0.219, P < 0.0001; Fig. 2d). In fact, our randomisations where females avoided babysitters predicted a small increase in the mean relatedness between breeding pairs compared to random within-group mating (Kolmogorov–Smirnov test avoiding babysitters; D(10,000) = 0.6408, P < 0.0001; Fig. 2d).

Our randomisation of breeding patterns assuming that females avoided mating with their escorts did result in mean relatedness values consistent with the observed mean relatedness value (observed mean r = 0.165, mean r based on the randomisation avoiding escorts = 0.181, 95% CI: 0.159–0.204, P = 0.157; Fig. 2e). However, the observed mean relatedness value was also consistent with random within-group mating (observed mean r = 0.165, random within group mating mean r = 0.179, 95% CI: 0.156–0.203, P = 0.231, Fig. 2e). This is because the distributions of expected relatedness values under random within-group mating and when females avoid escorts were strongly overlapping for the subset of 146 breeding pairs with escorting data. In fact, our randomisation of females avoiding breeding with escorts resulted in a slight increase in average relatedness compared to random within group mating (Kolmogorov–Smirnov test; D(10,000) = 0.0965, P < 0.0001; Fig. 2e). Avoiding escorts as mates is therefore unlikely to account for inbreeding avoidance within banded mongoose groups.

Is inbreeding avoidance based on age-related rules?

Our randomisation found that breeding with one of the three oldest-ranking males in the group would reduce the probability of inbreeding (Kolmogorov–Smirnov test; D(10,000) = 0.7475, P < 0.0001; Fig. 3a). However, this was not sufficient to explain inbreeding avoidance in our dataset as the observed mean relatedness value was significantly lower than the 95% CI of the randomisation distribution (observed mean r = 0.144, randomisation mean r based on breeding with one of the three oldest ranking males = 0.159, 95% CI: 0.145–0.172, P = 0.038; Fig. 3a). Breeding with males of a comparable age made little difference to the probability of inbreeding (Kolmogorov–Smirnov test; D(10,000) = 0.0453, P < 0.0001; Fig. 3b) and failed to explain inbreeding avoidance in banded mongooses (observed mean r = 0.144, randomisation mean r based on mating with males of a similar age to the female = 0.175, 95% CI: 0.160–0.1897, P < 0.0001; Fig. 3b).

Fig. 3
figure 3

The results of simulation-based tests of age-based inbreeding avoidance hypotheses. Yellow histograms represent the distributions of mean relatedness between males and females if a females preferentially breed with older males and b females preferentially breed with males of comparable age to themselves. Grey histograms represent within-group random mating and the red arrow shows the empirical mean relatedness of breeding pairs

Full size image

Is inbreeding avoidance based on multiple mechanisms?

Our randomisation combining the two best mechanisms found that preferentially breeding with older males, unless they were born into communal litters the female had previously given birth into (i.e. were potential sons), would reduce the probability of inbreeding (Kolmogorov–Smirnov test; D(10,000) = 0.7774, P < 0.0001; Fig. 4). This was not, however, sufficient to explain the observed relatedness value (observed mean r = 0.143, randomisation mean r based on mating with older males unless they were born into a communal litter the female gave birth into = 0.157, 95% CI: 0.144–0.170, P = 0.0324; Fig. 4).

Fig. 4
figure 4

The results of simulation-based tests of a combination of the two best mechanisms tested for. The yellow histogram represents the distribution of mean relatedness between males and females if females preferentially breed with older males unless the males are potential sons (were born into communal litters the females gave birth into). The grey histograms represent within-group random mating and the red arrow shows the empirical mean relatedness of breeding pairs

Full size image

Discussion

We found that the overall magnitude of inbreeding within banded mongoose social groups was below that expected under random within-group mating, indicating that some degree of inbreeding avoidance occurs. This is consistent with a previous study using a smaller dataset (Sanderson et al. 2015). However, inbreeding avoidance appears to be far from maximised, and females often did not breed with the least-related available male. Instead, our data are consistent with females preferentially breeding with males in inverse proportion to their degree of relatedness.

We found no evidence that this inbreeding avoidance was the result of familiarity-based kin recognition as our randomisation analyses of relatedness patterns based on females avoiding breeding with their communal littermates, maternal littermates, potential sons, or carers (babysitters and escorts) were not consistent with the observed mean relatedness of breeding pairs. Furthermore, inbreeding avoidance based on avoiding natal group members was deemed unfeasible since it would leave most (76.3%) of the females with no potential mates. Females could mate with extra group males; however, only ~ 20% of offspring are born to extra-group males (Nichols et al. 2015), and none of the females included in our analysis bred extra group. Similar to familiarity-based mating rules, age-based breeding patterns in banded mongooses also fail to explain the observed level of inbreeding avoidance. A combination of the two best mechanisms (preferentially breeding with older males and avoiding potential sons) came closest to explaining the observed level of inbreeding avoidance; however, again these two rules combined were not sufficient to explain observed inbreeding avoidance. Our results therefore suggest that inbreeding avoidance may be based on an as yet unknown proxy of genetic relatedness or is avoided through post-copulatory mechanisms. Because the degree of inbreeding avoidance is modest, such proxies (or post-copulatory mechanisms) could either be fairly error prone or decisions based on them must be subject to other constraints that reduce their effectiveness. For example, there might be costs associated with avoiding mating with close relatives within groups, such as females losing foraging time by refusing male mating attempts.

The lack of familiarity-based mate choice rules in banded mongooses is highly unusual for a cooperative breeder (Komdeur et al. 2008; Jamieson et al. 2009; Nichols 2017). Familiarity is generally considered to be a reliable indicator of relatedness in highly stable social groups and is widely used to inform mate choice (Pusey and Wolf 1996). However, in banded mongooses, familiarity may be an unreliable proxy for relatedness due to their high level of promiscuity and communal care (Sanderson et al. 2015). For example, in the majority of cooperative breeders, litters or broods contain full siblings and potentially also half-siblings (where extra-group males have fathered young) but are unlikely to contain lesser relatives. But in banded mongooses, the large communal litters that pups are raised in are composed of full siblings, half siblings, cousins, lesser relatives, and non-relatives, so litter membership is unlikely to be a sufficiently reliable proxy of relatedness to inform inbreeding avoidance.

Similarly, the paucity of unrelated immigrants in banded mongoose groups (Nichols et al. 2012c; Cant et al. 2013) may explain why we found no evidence that inbreeding could be avoided by avoiding natal group mates. This is in contrast to other cooperative breeding mammals such as meerkats (Griffin et al. 2003; Nichols 2017) where females breed with immigrant but not natal males. Furthermore, in banded mongooses, relatedness does not explain which adults provide escorting care for which pups (Vitikainen et al. 2017), making the provision of early life care a poor proxy of kinship, and explaining why inbreeding is not reduced by avoiding breeding with babysitters or escorts. This is in contrast to studies of facultative cooperative breeders such as long-tailed tits and Seychelles warblers, where helpers direct care towards relatives (Russell and Hatchwell 2001; Komdeur et al. 2004). However, a similar situation may occur in meerkats and other obligate cooperative breeders, as non-relatives often help at similar rates to relatives (Duncan et al. 2019). In meerkats, unlike banded mongooses, inbreeding is avoided through other familiarity-based mechanisms, as females only breed with immigrant or extra-group males (Nielson et al. 2012).

Banded mongooses are known to show age-related mating patterns; the highest ranking (oldest) dominant males gain an average of 85% of paternities, and these males preferentially mate with the oldest females in the group, leaving younger females to breed with younger males (Nichols et al. 2010). We therefore tested whether these patterns could result in the observed level of inbreeding avoidance. We found that breeding with males of a similar age to themselves did not result in inbreeding avoidance. In contrast, breeding with the oldest males in the group appears to reduce within-group inbreeding levels in comparison to random within group mating. However, this behaviour alone was not sufficient to explain the level of inbreeding avoidance found within banded mongoose groups. It is therefore possible that the high reproductive success of older males could contribute to inbreeding avoidance in banded mongoose groups, but other mechanisms are also likely to come into play (see below). Cases where animals avoid inbreeding using behavioural rules not based on familiarity are relatively uncommon in cooperative breeding species (Nichols 2017). One possible reason for this could be that potential proxies of relatedness tend to be unreliable in comparison to familiarity or phenotype matching. Alternatively, rules not based on familiarity might apply regularly but are rarely tested for.

The mechanism that banded mongooses use to avoid inbreeding is currently unclear. We were able to confidently exclude the possibilities that banded mongooses avoid inbreeding by avoiding familiar individuals from early life or via age-related breeding patterns (Table 1). This leaves two possibilities. First, inbreeding avoidance might invoke post-copulatory mechanisms. For example, where females mate multiply, as is often the case in banded mongooses (Nichols et al. 2010; Wells et al. 2018), paternity could be biased towards less-related males. A similar situation occurs in house mice, whereby paternity is biased towards non-relatives when females are experimentally mated to their brothers and unrelated males (Firman and Simmons 2008). This appears to arise through egg-driven sperm selection, whereby eggs are more likely to be fertilised by sperm from non-relatives (Firman and Simmons 2015), but it is also possible that further paternity skew may be generated if inbred embryos are more likely to die prior to birth. Such mechanisms are possible in banded mongooses, where skews in paternity could potentially arise from sperm selection or inbred offspring being less likely to survive in utero or in the first few days after birth. While inbreeding depression in banded mongooses has been shown to impact on weight and reproductive success, there is relatively little evidence that inbreeding influences survival beyond 30 days after birth (the earliest that it is possible for us to measure survival) (Sanderson et al. 2015; Wells et al. 2018). However, Wells et al. (2020) found that inbred pups that received little escorting care were less likely to survive to nutritional independence (90 days) than outbred pups that received little escorting (there was no inbreeding-associated difference in survival for pups that received high levels of care). It is therefore possible that survival before pups reach 30 days old may be negatively impacted by inbreeding, especially when they encounter poor conditions in utero or immediately after birth, and this may be sufficient to explain the observed level of inbreeding avoidance.

Second, banded mongooses may assess relatedness using a proxy that we have been unable to measure in the current study. Supporting this possibility, banded mongoose females have been shown to discriminate kinship in the context of reproductive conflict, whereby dominant females expel closely related subordinates from the group to prevent them from breeding (Thompson et al. 2017). Here, phenotype matching could be used, based on a cue such as odour similarity, as has been shown for several other mammals (Mateo and Johnston 2000; Leclaire et al. 2013). Banded mongooses are prolific scent markers (Jordan et al. 2010), have individually distinct and stable scent profiles (Jordan et al. 2011), and have been shown to react differently to the scent of group members depending on their relatedness (Mitchell et al. 2018). However, there is currently no evidence that they are able to distinguish between related and unrelated unfamiliar individuals based on scent (Mitchell et al. 2018). Consequently, it is unclear whether they might obtain information on relatedness directly from odours or whether they could simply recognise familiar individuals from their odour and use a different cue to inform them about relatedness. Future studies will further explore the role of odour in communicating relatedness in this species. Alternatively, vocal calls could be another source of information on relatedness as they incorporate individual distinguishable vocal signatures (Jansen et al. 2012). Finally, direct genetic kin recognition is potentially possible but is extremely rare and is considered unstable as a kin recognition mechanism; one reason being that if mating success is based on sharing a genetic marker, common genetic markers will increase in frequency until they are no longer a viable proxy of relatedness (Gardner and West 2007).

It is also possible that female banded mongooses exploit a combination of behavioural rules (based on familiarity or other traits) that we have not tested for in this study. For example, inbreeding could be avoided in different ways for females of different ages, that may differ in their degree of behavioural dominance and that are also likely to have different categories of male relatives present within their group. Inbreeding avoidance strategies might also differ for group founders versus females that remain in their natal group. This has been shown for extra-group breeding patterns, whereby female group founders are likely to have access to unrelated males within the group and hence are less likely to mate extra-group than natal females (Nichols et al. 2014). However, both founders and natal females have been shown to avoid inbreeding within the social group to a similar degree, so within-group inbreeding avoidance cannot be explained solely by founders having access to unrelated mates (Sanderson et al. 2015).

Although inbreeding levels were lower than expected from random mating within the social group, inbreeding avoidance in banded mongooses appears to be far from maximised. If females always bred with the least-related male within the group, they would almost completely avoid inbreeding; the average relatedness between breeders would be 0.021 compared to the realised value of 0.144. One possible explanation for this is that the cues available to banded mongooses may be relatively poor indicators of relatedness. Given that the familiarity-based proxies of relatedness used by the majority of cooperative breeders are ineffective in banded mongooses, inbreeding avoidance may be particularly challenging. Phenotype matching may be a better proxy of relatedness, although traits such as odour may have a strong environmental component (Stoffel et al. 2015) that could lead to errors in the identification of relatives (Ferkin et al. 1997; Fisher and Rosenthal 2006; Havlicek and Lenochova 2008). Consistent with this, studies have found substantial individual variation in responses to scent (e.g. Krause et al. 2012; Leclaire et al. 2013), even when the overall distribution of responses suggests that kin discrimination occurs. Such errors in identifying kin may also explain why altruistic help is provided indiscriminately in regards to relatedness in some cooperative breeders; in fact, where average relatedness within groups is high and errors in kin discrimination exist, a decision to always help may be stable (Duncan et al. 2019).

An alternative explanation for the observed deviation from optimal inbreeding avoidance in banded mongoose groups is that females may be unable to always mate with the least-related member of the opposite sex due to constraints, for instance via reproductive competition. During oestrus, dominant males attempt to guard females from the advances of other males (Cant et al. 2010). Although males appear to direct mate guarding towards lesser relatives (Sanderson et al. 2015), they are likely to base guarding investment decisions on other factors in addition to relatedness. For example, older females produce more offspring and so are preferentially guarded by dominant males (Nichols et al. 2010). Similarly, only a single male can guard a female at one time (but males can guard multiple females sequentially), so male–male competition will also restrict mate choice. Females can (and frequently do) refuse mating attempts (Cant 2000) and “sneak” matings with males other than their guard (Sanderson et al. 2015). In fact, only 50% of females produce offspring with a male observed to guard them, with the majority of the remainder mating with a different within-group male (Sanderson et al. 2015). Where females sneak matings, extra-pair males are on average less related to the female than the guard, suggesting that females can reduce their probability of inbreeding if they evade their guard (Sanderson et al. 2015). However, mate choice in both sexes is still likely to be constrained by mate guarding, and so may reduce the effectiveness of attempts to avoid inbreeding. Unfortunately, the difficulty in observing mating, incomplete guarding data, and the relatively weak association between guarding and paternity meant that it was not possible for our randomisations to take into account potential differences in the degree to which different within-group males could access each female. Nevertheless, the potential for constraints on mate choice should be considered when interpreting our results.

Finally, a certain level of inbreeding might also be adaptive, providing inclusive fitness benefits through an increased representation of shared genes (Kokko and Ots 2006; Puurtinen 2011; Szulkin et al. 2013). It is therefore possible that the level of inbreeding observed could reduce inbreeding depression to a tolerable level while allowing banded mongooses to increase their inclusive fitness. However, negative fitness consequences (reduced pup survival and yearling weight) have been shown to be associated with even moderate levels of inbreeding in this system (Sanderson et al. 2015; Wells et al. 2020). This lends further support to the argument that some kind of constraint on kin recognition and/or mate choice is the most parsimonious explanation for our results than “optimal inbreeding”.

Conclusion

We show that banded mongooses inbreed at a lower rate than would be expected under random mating. However, inbreeding avoidance is far from maximised, and inbreeding occurs more often than would be expected if females always mated with the least-related group mate. Inbreeding avoidance does not appear to occur via avoiding individuals that are familiar in early life such as communal littermates, maternal littermates, potential sons, carers, and natal group members. This is in contrast to the situation that has been reported in other cooperative breeders (Komdeur et al. 2008; Jamieson et al. 2009; Nichols 2017) and is likely to be explained by the unusual breeding system of this species, which may render such proxies of relatedness ineffective (Vitikainen et al. 2017), combined with the lack of unrelated immigrants into social groups (Cant et al. 2013, 2016). We therefore suggest that alternative methods of inbreeding avoidance, such as phenotype matching or post-copulatory mechanisms, are likely to occur when familiarity is of limited use in detecting relatives. Furthermore, mating decisions in banded mongooses are likely to be constrained by mate guarding from dominant males, which may reduce the ability of both sexes to select minimally related partners. The constraints imposed on mate choice and kin recognition in this system may explain the unusually high levels of inbreeding found, whereby 7.1% of pups are the product of first-order inbreeding (Wells et al. 2018). Our results support the idea that high levels of inbreeding in natural populations can be driven by constraints against prominently proposed methods of inbreeding avoidance.