Introduction

Terrestrial mammals rely heavily on chemosensory information to investigate, interpret, and navigate their surroundings. Perception of exogenous chemical cues is mediated by highly specialised peripheral sensory organs that are exposed to the external environment. The two major chemosensory systems in mammals are olfaction and gustation, which detect odorants and tastants, respectively. Both types of cues are important as a form of chemical communication that directs natural behaviour. For example, learning the smell of a mother by her newborn is critical for guiding suckling interactions in mice and rats (Blass and Teicher 1980; Logan et al. 2012). Once the olfactory bond is established, the rewarding taste of milk further reinforces the drive to suckle. Associative learning of chemosensory cues is widely exploited in experimental investigation into mammalian behaviour and cognition: for example, simple conditioned odour preference tests can be used to investigate memory retention (Schellinck et al. 2001). For these behavioural responses, conditioning to a recognizable odour is paramount, but the precise nature of the odour is less important. We and others have experimentally manipulated rodent mother/pup suckling interactions so that the young respond to an array of artificial odours, including garlic, vanilla, lemon, and almond (Logan et al. 2012; Pedersen et al. 1982). In the case of olfactory regulation of suckling behaviour, it appears that almost any smell will do so long as it is appropriately conditioned. Like the natural signature odours that pups learn, all four artificial odorants are detected by sensory neurons of the main olfactory epithelium (MOE), the largest olfactory subsystem in mammals.

In contrast, some behaviours are released only by very specific odorants and in a manner that is independent of prior experience. These so-called innate or instinctive responses to defined chemical cues tend to be highly stereotyped between individuals of the same sex and age. While it is now clear that some of these specialized cues are also MOE-mediated (Kobayakawa et al. 2007; Schaal et al. 2003), the olfactory subsystem largely (though by no means exclusively) tasked to detect them is the vomeronasal organ (VNO). In this review we focus exclusively on genes regulating VNO-mediated behaviour. The emerging roles of other olfactory organs in innate behaviour are discussed elsewhere (Ma 2010; Stowers and Logan 2010a).

The olfactory cues that elicit specific innate behaviours are classified according to both the source of the signal and the nature of its influence. Pheromones are social cues that are transmitted between two members of the same species, such as a chemical signal emitted by a sexually receptive female that is innately attractive to a male. Kairomones are chemicals transmitted between species that benefit the receiver on detection and allomones are interspecific signals that benefit the emitter (Wyatt 2003). These categorisations were influenced by ground-breaking work on insects prior to significant research into chemical communication in mammals (Sbarbati and Osculati 2006). Current scientific opinion differs on if and when it is appropriate to describe mammalian semiochemical cues in these terms, given the strong confounding influence of environment, experience, and emotional state on behavioural responses (Petrulis 2013). At one extreme, chemicals that are emitted in a sexually dimorphic manner but with little or no support for innate bioactivity are often misleadingly described as “putative pheromones” (see Wysocki and Preti 2004 for a critical analysis); at the other end, it has been argued that mammalian pheromones simply do not exist and all the examples described to date are flawed in their methodology or interpretation (Doty 2010). In this review we use these terms only when a chemical is demonstrated to be emitted, then synthesized or purified and shown to elicit a stereotypical behaviour, independent of prior experience, in a controlled, quantitative bioassay. When these conditions are not met, we instead describe the proposed chemical signals as semiochemicals or olfactory cues.

The stereotypy of innate responses to defined chemical signals and the importance of the behaviours in social interaction, reproduction, and survival imply a substantial genetic investment underpinning the process. Here we present an overview of the many hundreds of genes involved in mediating innate VNO-mediated behaviour. In the emitting mammal, a growing number of gene families have been implicated in either encoding olfactory cues or are involved in the presentation or stabilization of the cues. In parallel, there are large gene families that encode the receptors that detect the cues in the receiving animal and pattern the neural circuits that transmit the signals to the brain. Other individual genes encode proteins that are involved in signal transduction processes specific to the VNO that influence a range of behaviours. Both sets of gene families share genomic characteristics that make them technically challenging to study. Therefore, the genes described here are among the least understood in the genomes of mammals.

The vomeronasal organ

The VNO, also known as Jacobson’s organ, is a bilateral blind-ended tubular structure divided by the nasal septum, typically located directly above the roof of the mouth at the base of the nasal cavity (Doving and Trotier 1998). Each half of the organ contains a crescent-shaped sensory epithelium, medial to a fluid-filled lumen, and a nonsensory epithelium and blood vessel located laterally (Fig. 1). There is a rostral opening to the nasal cavity, allowing semiochemicals access to the lumen. A pumping mechanism that involves the constriction and dilation of the cavernous organ walls actively forces fluid in and out of the lumen, delivering stimuli to the vomeronasal sensory neurons (VSNs). The neurons project axons caudally to the accessory olfactory bulb (AOB) where they condense into neuropil called glomeruli. A VNO is found in many but not all mammalian taxa. It is missing in cetaceans, some bats, and some primates (Mucignat-Caretta 2010). There are conflicting anatomical reports on the existence of a VNO in humans. However, a consensus has emerged that some adult humans may have vomeronasal cavities, but these do not contain neural tissue. Consistent with this, humans lack an AOB and many of the genes with VNO-specific functions are pseudogenised. Nevertheless, there is growing evidence that some Old World monkeys, including humans, may use semiochemical cues to mediate behaviours that are analogous to VNO-mediated behaviours in rodents (Gelstein et al. 2011; Roberts et al. 2012a). The sensory mechanisms that underpin these responses remain elusive.

Fig. 1
figure 1

The mouse vomeronasal organ. A coronal section through half of the VNO of adult mouse (left) with a cartoon of the corresponding tissue morphology (right). S nasal septum, C cavernous tissue, G glandular tissue, B blood vessel, V vomer, N nonsensory epithelium, L lumen, E sensory epithelium with apical (right) and basal (left) layers of vomeronasal sensory neurons

Full size image

Signal transduction

Semiochemicals that reach the VSNs in the VNO need to be recognised and their identity must be transmitted to the AOB. Three families of receptor genes (VRs) have been identified in the mouse VNO—two families of vomeronasal receptors (Vmn1rs and Vmn2rs) and a group of formyl peptide receptors (Fprs)—and some evidence exists to support their role in binding olfactory cues. This leads to the activation of a signal transduction pathway that results in the generation of an action potential in the stimulated VSNs. Initial efforts to characterise the signalling cascade focused on the genes involved in the same process in the MOE; none of these could be detected in the VNO (Berghard et al. 1996). A search for analogous components led to the identification of the G-protein α subunits Gαi2 and Gαo. These are highly expressed in VNO neurons in two mutually exclusive populations (Fig. 2); VSNs that express Gαi2 are located in the apical region of the neuroepithelium while the ones expressing Gαo sit in the basal portion (Berghard and Buck 1996). For both cellular populations, expression is localized to the microvilli of the neurons, where ligand detection occurs. The functional importance of both subunits in mediating behavioural responses was established by ablating the genes in mice. Gαi2-mutant males display a diminished aggressive response in a classical “resident-intruder test”, where an intruder male is introduced to the cage of a territorial resident. Likewise, mutant lactating females are also less aggressive, but sexual behaviours appear unaltered (Norlin et al. 2003). However, Gαi2 is expressed in other tissues and the mutant animals have other debilitating phenotypes (Rudolph et al. 1995); therefore, it remains possible that the aberrant behaviour is not a direct consequence of VNO-mediated signalling. With this caveat in mind, Chamero et al. (2011) generated a mutant line with Gαo ablated only in vomeronasal neurons. These animals display strikingly similar behaviour to that of Gαi2-deficient mice in that both sexes are less aggressive. Thus, both classes of VSN appear to transduce chemosensory-mediated aggressive behaviour: a subset of apical Vmn1r- and Gαi2-expressing neurons via uncharacterised small molecule cues in male urine, and some basal Vmn2r- and Gαo-expressing neurons via major urinary proteins (MUPs) (Chamero et al. 2007).

Fig. 2
figure 2

Signal transduction proteins in vomeronasal sensory neurons. There are two subclasses of mammalian vomeronasal sensory neuron (VSN). In apical VSNs (a), a V1R receptor associated with the Gαi2 G-protein subunit is activated by a small, volatile chemical ligand. In basal VSNs (b), a V2R receptor from subfamily C is coexpressed with one from subfamily A, B, or D. These are associated with the Gαo G-protein subunit and are individually or collectively activated by a peptide or protein ligand. One or more of nine major histocompatibility complex (MHC) class 1b proteins and β2-microglobulin (B2M) are also expressed in a subset of these neurons. Both types of neuron additionally express a transient receptor potential ion channel (TRPC2) and calcium-activated chloride channels (CACCs), which together depolarise the cell

Full size image

In 1999, Liman et al. (1999) identified another key player in eliciting VNO signal transduction: a member of the transient receptor potential (TRP) family of ion channels, TRPC2. The rat Trpc2 gene was shown to be abundantly expressed in the VNO and absent in the MOE. Detailed analysis showed that the protein is found in the microvilli of the sensory neurons and colocalises with expression of both Gαi2 and Gαo (Menco et al. 2001). The dramatic role of Trpc2 in vomeronasal-mediated behaviour was made evident when the gene was knocked out in mice. Two groups independently showed that VSNs from these animals are either nonresponsive or have a significantly reduced response to urinary semiochemicals (Leypold et al. 2002; Stowers et al. 2002). Behavioural analyses of the mutant males revealed a diminished aggressive response in the resident-intruder paradigm. Instead of initiating an attack, Trpc2 −/− males displayed sexual behaviour towards the intruder, just as a Trpc2 +/+ mouse does when presented with a female. Additionally, when presented with both a male and a female, Trpc2 −/− males did not discriminate (Leypold et al. 2002; Stowers et al. 2002). These led to the conclusion that these mice are unable to determine the sex of the conspecifics they encounter due to the lack of signal transduction of olfactory cues through VSNs.

However, residual electrophysiological activity could still be detected in the VNO of Trpc2 −/− animals, suggesting at least one other signalling pathway is operative in VSNs (Zhang et al. 2010). Consistent with this, VNO-mediated pregnancy block still occurs in Trpc2-mutant females but surgical lesion of the organ disrupts the response (Kelliher et al. 2006). Another class of ion channel was subsequently identified in VSNs: calcium-activated chloride channels (CACCs) (Yang and Delay 2010). Although activity of these channels is both necessary and sufficient for activation of the neurons (Kim et al. 2011), to date it is Trpc2 −/− mice that have proven most useful for revealing additional VNO-mediated behaviours. Like males, Trpc2-mutant lactating females are not aggressive toward intruder males (Hasen and Gammie 2009; Leypold et al. 2002). They display male-like sexual behaviours towards intruders, such as mounting and pelvic thrusts, and are deficient in maternal behaviours (Hasen and Gammie 2011; Kimchi et al. 2007). As with male residents, when mutants are presented with both male and female intruders, they show no preference towards mounting one sex. Thus, TRPC2 appears necessary for VSNs to effectively transduce a range of chemosensory cues that are transmitted between mice to initiate social behaviours. More recently, Trpc2 −/− mice were used to demonstrate that VSNs also detect olfactory cues from other species (Ben-Shaul et al. 2010). The mutant mice do not display innate defensive and avoidance behaviours, or a stress response, when exposed to predator cues from snakes, cats, and rats (Papes et al. 2010).

The specialization of TRPC2 in VNO function has made it a useful genomic proxy for assessing which mammalian species display VNO-mediated behaviour. Most terrestrial mammals have an ostensibly functional Trpc2 gene, though it became pseudogenised in the common ancestor of Old World monkeys and apes (including humans) (Liman and Innan 2003). It has been proposed that Trpc2 loss is concordant with the emergence of male trichromatic vision in this lineage and indicates a move toward social communication by visual cues at the expense of pheromones (Zhang and Webb 2003). However, New World howler monkeys have both trichromatic vision and a functional Trpc2 gene, indicating that trichromacy does not necessarily lead to a loss of VNO function in primates (Webb et al. 2004). Fully aquatic cetaceans, including dolphin and fin whale, have a pseudogenised Trpc2, as do the semiaquatic harbour seal and river otter (Young et al. 2010). However, another amphibious species, the California sea lion, appears to have a functional gene (Yu et al. 2010). Some, but not all, species of bat also have a pseudogenised Trpc2 gene (Young et al. 2010; Zhao et al. 2011), suggesting mammalian adaptation to both flight and water are factors that can reduce the reliance on chemosensory-mediated behaviour.

The vomeronasal receptor genes

VR genes are encoded by two multigene families of G-protein-coupled receptors (GPCRs). These are not closely related to the olfactory receptors (ORs) expressed in the MOE, nor are they particularly similar to each other within the GPCR phylogeny. The characterisation of ORs in the MOE revealed an unusual epigenetic control mechanism: only one allele of one receptor gene is expressed per neuron, which means that only a subset of the neurons express any given OR (Chess et al. 1994). Under the assumption that a similar mechanism might be operating in the VSNs, Dulac and Axel (1995) devised a clever differential hybridisation strategy that allowed them to find coding sequences expressed specifically in one neuron but not others. This yielded a coding sequence for a gene encoding a putative seven trans-membrane domain, characteristic of GPCRs, that was expressed in a subpopulation of VNO neurons. Additional related genes were then identified and it was confirmed that they were part of a multigene family. Each of the receptors tested by in situ hybridisation was expressed in a subset of neurons, and a combination of probes for several genes resulted in a number of labelled VSNs equivalent to the sum of the cells recognised by the individual probes. Interestingly, expression could be detected only in the apical, Gαi2-positive region of the neuroepithelium. All the above suggested that these genes were putative receptors and that each VSN likely expressed a single receptor gene (Dulac and Axel 1995). This receptor family comprises the V1Rs. A couple of years later, three different groups reported the expression of an unrelated multigene family of receptors expressed in the basal, Gαo-positive portion of the VNO (Fig. 2). Each was found to be similarly expressed in a small sub-population of VSNs (Herrada and Dulac 1997; Matsunami and Buck 1997; Ryba and Tirindelli 1997). Correspondingly, these receptors were termed V2Rs.

V1Rs

With the availability of a good mouse reference genome, it has been possible to identify the complete receptor repertoire. The mouse genome contains 392 V1R genes (named Vmn1r in mice), 239 of which have an intact open reading frame (ORF) (Young et al. 2010). A phylogenetic tree, constructed with 137 of the intact genes, groups them into 12 distinct subfamilies (Vmn1ra-j). Receptors from the same subfamily share at least 40 % identity at the amino acid level, but the diversity between different families is large, and identities can be as low as 15 % (Rodriguez et al. 2002). Vmn1r genes of the same subfamily tend to be found together in the genome, arranged in tight clusters of genes that are dispersed across several chromosomes (Zhang et al. 2004).

V1Rs have been shown to respond to low-molecular-weight organic molecules with great sensitivity. Screening of VSNs with six different chemicals with putative pheromonal activity showed that each activated a small subset of neurons (Leinders-Zufall et al. 2000), and at least one of them was able to generate responses in neurons expressing different V1Rs (Boschat et al. 2002). Exposure of VSNs to sulphated steroids, which are present in female urine and are proposed to account for most of its vomeronasal bioactivity, results in the firing of both male and female Vmn1r-expressing VSNs; while some receptors respond to specific steroids, others recognise several compounds that are chemically related (Isogai et al. 2011; Nodari et al. 2008). To characterise the behavioural role of Vmn1r-expressing VSNs, a group of 16 intact receptor genes belonging to the families Vmn1ra and Vmn1rb were deleted in the mouse genome by chromosome engineering. Mutant female animals showed deficits in maternal aggression towards intruders and mutant males had lower mating rates (Del Punta et al. 2002). Therefore, at least some of these receptors are necessary for the normal display of innate behaviour.

The number of V1Rs encoded in the genomes of different mammalian species varies greatly, as does the proportion of the receptors that are pseudogenised (Fig. 3). To date the V1R gene repertoire has been studied in 37 mammalian species with available genomic sequences of relatively high quality. Rodents, in which they were first characterised, are among the species with the highest number of genes, along with the mouse-lemur and the rabbit. Around half of these receptors contain intact ORFs (Young et al. 2010). Along with these, the semiaquatic platypus has the largest repertoire known to date, with 283 intact V1R genes and many more pseudogenes (Grus et al. 2007). A large number of other mammals have a medium-sized V1R family, but the dolphin, the little brown bat, and the flying fox have no intact genes. In the case of humans, other apes, and Old World monkeys, the number of intact V1R genes is very low, but they contain a large number of pseudogenes (Young et al. 2010). A positive correlation has been observed between the V1R repertoire size and the complexity of the vomeronasal system and AOB size (Grus et al. 2005; Young et al. 2010).

Fig. 3
figure 3

The vomeronasal receptor gene repertoires of mammals. The species represented are limited to those in which the full repertoire of V1R and V2R genes and pseudogenes are reported. The data are collated from Grus et al. (2007), Young and Trask (2007) and Young et al. (2010)

Full size image

Evolutionary analyses of the V1Rs in diverse mammalian species reveal that receptor sequences were present in the common ancestor of placental and marsupial mammals; however, it is common to find species-specific expansions and deletions of certain subfamilies, even in lineages as close as the mouse and rat. Interestingly, the pseudogenes found in dogs and primates fall into all the major clades of the V1R phylogenetic tree, indicating that the common ancestor had a diverse receptor repertoire that was then lost and degenerated in these species while maintained and further expanded in animals with an active vomeronasal system (Grus et al. 2005; Young et al. 2005). Therefore, the V1R repertoire of mammals has likely been shaped by each species’ ecological niche. In support of this hypothesis, species that live in nests (where the utility of vision and hearing are limited) have larger repertoires than those living in open areas. Similarly, nocturnal animals have a larger collection of V1Rs than diurnal species (Wang et al. 2010).

V2Rs

The mouse reference genome contains 279 V2R genes (termed Vmn2r in mice), 158 of which are characterised as pseudogenised (Young and Trask 2007). The predicted intact sequences can be grouped into four different subfamilies (A–D). Most of the genes (85 %) belong to the A subfamily, which is further subdivided into nine clades. As with Vmn1rs, closely related genes tend to be clustered in the mouse genome (Yang et al. 2005). Vmn2r genes, however, are distinct in their expression logic. Each VSN of the basal VNO expresses a member of the subfamily C (composed by seven genes in mouse), along with an additional Vmn2r gene from subfamily A, B, or D in a nonrandom fashion (Ishii and Mombaerts 2011; Martini et al. 2001; Silvotti et al. 2007). In addition to this, some basal VSNs have been shown to express genes of the major histocompatibility complex (MHC) class 1b and β2-microglobulin (B2M, which is essential for the proper expression of MHC class Ib molecules at the cell surface). These proteins localise to the dendritic tips of VSNs, as do TRPC2 and Gαo. Each of the nine genes in this family (M1, M9, M11, and six members of the M10 family) is expressed in a subset of neurons positive for Gαo; even though most of the neurons express a single gene, some can express two or three. The expression of specific members of this family appears to pattern the basal Vmn2r-expressing VSNs into two sublayers: the middle VSN layer is MHC class Ib negative, while the most basal layer is MHC class Ib positive (Ishii and Mombaerts 2008). Along with B2M, they have been proposed to form a protein complex necessary for the transport of the receptor to the plasma membrane (Ishii et al. 2003; Loconto et al. 2003).

V2Rs have been found to respond to water-soluble peptides and proteins that can be found in urine and other bodily secretions of conspecific mice, as well as from other species. The first evidence for this came from the finding that peptide ligands of the MHC class I molecules activate around 1 % of the VSNs, all situated in the basal neuroepithelium (Leinders-Zufall et al. 2004). The presentation of different peptides leads to activation of different neural populations, which overlap to some extent. It has been shown, for example, that those VSNs that express Vmn2r26 (also known as V2R1b) recognise some of these peptides, but neurons expressing other receptors are also responsive to the same stimuli. The different peptides that activate the same neurons share key residues at anchor positions, and these are necessary and sufficient to induce the response (Leinders-Zufall et al. 2004, 2009). These peptide cues also induce the Bruce effect in female mice [a selective chemical cue-induced pregnancy failure (Bruce 1959)] when spiked into otherwise familiar male urine (Leinders-Zufall et al. 2004), thus establishing them as a “signature mixture” of odours (Wyatt 2010). Subsequently, further protein ligands that activate Vmn2r-expressing neurons have been identified. These include products of the Mup and Esp gene families that either encode identity or variously initiate sexual, attractive, aggressive, and avoidance behaviours (Chamero et al. 2007; Haga et al. 2010; Hurst et al. 2001; Papes et al. 2010; Roberts et al. 2010). With the exception of some ESPs (detailed below), the V2R receptors that bind these cues and mediate their behavioural effects have remained elusive.

V2Rs are multiexonic genes, making their identification through bioinformatic analyses more difficult than that for V1Rs (which tend to have their coding sequence spanning a single exon). Nevertheless, the repertoires of several mammalian species have been studied in detail (Fig. 3). The mouse and rat, along with the opossum, have the largest number of V2Rs. The platypus also has an expanded repertoire, but most are pseudogenised. At the other extreme, dog, cow, human, chimpanzee, and macaque have few V2Rs, and none of these are functional. In an interesting difference to V1Rs, those species with a functional V2R gene set show expansions after the lineages diverged; for example, only four orthologous V2R pairs can be found between the mouse and rat (Yang et al. 2005; Young and Trask 2007). In addition to interspecific variation, V2R repertoires are also likely to show high levels of functional variation between individuals of the same species. A study of the vomeronasal receptor repertoires of inbred mouse strains found that the Vmn2r subfamily A clades A1, A5, and A8 are particularly variable while subfamilies B, C, and D are highly conserved (Wynn et al. 2012). Thus, differential selective pressures are acting on the Vmn2r subfamilies, presumably in a manner consistent with the pheromones they detect and the behaviours they mediate (Keller 2012).

Formyl peptide receptors

In order to determine if additional chemosensory receptors are expressed in the VNO, two groups independently prepared cDNA from mouse VSNs and amplified GPCRs that had not previously been implicated in chemodetection (Liberles et al. 2009; Riviere et al. 2009). Five of the seven members of the formyl peptide receptor (FPR) family were recovered. In situ hybridization revealed that each receptor is expressed in a subset of VSNs, in a manner similar to that observed with Vmn1rs. Similarly, no single neuron was patterned by two different Fpr genes. The VSNs that express four of the five FPRs were also positive for Gαi2, while expression of a single receptor (Fpr-rs1) was restricted to Gαo-positive neurons (Liberles et al. 2009; Riviere et al. 2009). No coexpression of VRs and FPRs could be detected. All these findings suggest that the VNO contains a third population of VSNs that express a different type of receptor gene.

N-formylated peptides are found in prokaryotes and mitochondria; accordingly, the other FPRs are expressed in the immune system and play a role in the host response. Thus, it has been proposed that the VNO-expressed FPRs may be pathogen chemosensors that elicit avoidance behaviours to resist infection. While this has yet to be demonstrated behaviourally, a number of studies have identified FPR ligands by calcium imaging of VSNs. These include bacterial N-formylmethionine-leucine-phenylalanine, the antimicrobial CRAMP, and the mitochondrially encoded peptides NDI-6T and NDI-6I (Chamero et al. 2011; Riviere et al. 2009). More recently, FPR-RS1 was found to display stereoselection for peptides with a D-amino acid in the C-terminal position, further supporting a role in detecting pathogenic chemosignals (Bufe et al. 2012). FPRs are also expressed in the VNOs of rats and gerbils (Riviere et al. 2009), but it is possible that the expansion of the Fpr gene family to encompass an olfactory function is rodent-specific, as only the genes expressed in the immune system are found in the genome of primates (Yang and Shi 2010).

Genetically encoded semiochemicals

It is often presumed that a necessary characteristic of a semiochemical cue is volatility in air, as the chemical must reach the sensory neurons via the intake of breath through the nose. Accordingly, the search for many candidate pheromone cues historically has focused on small, low-molecular-weight chemicals in the secretions or excretions of mammals. However, the fluid-filled lumen of the VNO can facilitate the transfer of larger, nonvolatile peptides and proteins to the epithelial surface. Moreover, nasopalatine ducts connecting the VNO and oral cavity permit the delivery of cues in solution directly to the organ, a process that occurs when animals display the Flehmen response. Recently, significant progress has been made in identifying two gene families that likely encode such signals in rodents. These display strikingly similar genomic, transcriptional, and evolutionary characteristics that are consistent with those expected of species-specific protein pheromones (Karn and Laukaitis 2009), though only a few of over 50 proteins in these families have been convincingly demonstrated to elicit an innate behaviour. There are other gene families with similar characteristics that likely encode semiochemicals, most notably the salivary androgen-binding proteins (Emes et al. 2004), but these have yet to be shown to directly elicit a behavioural response in a bioassay.

Major urinary proteins

The MUPs, also called α2u-globulins in rat, are a class of small hydrophobic lipocalins. As their name suggests, they are abundantly expressed in the urine of some rodents. However, MUPs are also found in other secretions, including saliva and milk. Over 30 years ago it was recognized that rodent MUPs were encoded by a multigene family but that the rat and mouse families were relatively dissimilar (Hastie et al. 1979). More recently, the Mup locus was analysed in the mouse reference genome by two independent studies revealing a cluster of homologous, highly identical genes and pseudogenes, many of which are arranged in tandem arrays (Logan et al. 2008; Mudge et al. 2008). The locus has 42 known genes, of which 21 are pseudogenised in the reference. However, the structurally repetitive nature of the arrayed Mup genes, some with sequence identity greater than 97 %, means that the locus is one of the few regions of the mouse genome that is incomplete. Therefore, it is likely that additional genes are present at the locus. The unusual levels of gene homology combined with the ongoing annotation efforts have resulted in multiple nomenclatures being independently proposed for the Mup gene family. Each has gained traction in the literature to varying degrees, making it difficult to easily compare genes between studies. In an effort to minimize confusion, in Table 1 we list the two attempts at providing a systematic nomenclature, alongside the more idiosyncratic official nomenclature adopted by the Mouse Genome Database (MGD) (Bult et al. 2010).

Table 1 Major urinary protein genes in the GRCm38 mouse reference genome
Full size table

The first hint that MUPs may act as semiochemical signals came from their sex- and strain-dimorphic expression in mice. Male urine contains significantly higher concentrations of MUPs than female, while males from different inbred strains express alternative combinations of the genes (Cheetham et al. 2009; Clissold et al. 1984). This variability is even more pronounced in the MUP content of individual wild mice, with the exception of one unusual protein that is almost uniformly present (Robertson et al. 1997). Subsequent experiments implicated MUPs as potential individuality signals that mice use to avoid inbreeding and assess heterozygosity (Hurst et al. 2001; Sherborne et al. 2007; Thom et al. 2008). The MUP content of C57BL6/J mice has been purified from male urine (Table 1) and produced in vitro. Both forms of MUP were shown to be sufficient to provoke aggressive behaviour in a resident male when daubed on the back of a castrated intruder (Chamero et al. 2007). Importantly, the pool of MUPs directly activated a proportion of VSNs and the behavioural response was dependent on the resident having a functional VNO (Chamero et al. 2007, 2011).

When the response of females to the combination of male MUPs was assessed, a strong experience-independent attraction was observed. However, only the atypically invariant MUP elicits this behaviour individually (Roberts et al. 2010). Officially named MUP20 but widely referred to as Darcin, this particular MUP appears to meet the classical definition of a sex pheromone. However, consistent with an enhanced role for olfactory learning in mammalian behaviour, it can also condition other odours with behavioural salience and can even induce spatial learning (Roberts et al. 2010, 2012b). These two Roberts et al. studies suggest that the Mup gene family encodes one pheromone that attracts females and at least one cue that can provoke aggression in males; however, it remains to be determined whether the same MUP is uniquely responsible for both.

A family of genetically encoded cues must be sufficiently divergent between sympatric species to avoid crossing wires during chemical communication, although eavesdropping on the semiochemicals from another species can provide the receiver with a selective advantage, especially if the signal reliably indicates the presence of a predator. Intriguingly, both mice and rats have large Mup gene expansions relative to other mammals, but their loci are structurally different, suggesting that they expanded independently in each lineage (Logan et al. 2008). Most other mammals have only one or a few Mup genes and humans have only one pseudogene (Meslin et al. 2011). While studying how mice respond to predators, Papes et al. (2010) demonstrated that the orthologous MUP proteins secreted by cats and rats are sufficient to provoke avoidance and defensive behaviours. Just like the aggressive response to mouse MUPs, these behaviours are both innate and VNO dependent. Thus, in addition to encoding at least one intraspecific pheromone, Mup genes from predator species have a credible claim to be the first genetically encoded kairomones to be characterised in mammals (Rodriguez 2010).

Exocrine-secreted peptides

The exocrine-secreted peptide (Esp) gene family shares many similar genomic features with Mups (Stowers and Logan 2010b). They are expanded into a poorly annotated cluster in rodents (the mouse genome has 38 and the rat has 10) but are missing in the human genome. Putatively functional mouse genes are interspersed with pseudogenes, and there are some species-specific expansions in the rodents (Kimoto et al. 2007).

Like MUPs, ESPs are also expressed in secretory tissues, including Harderian, lacrimal, parotid, sublingual, and submaxillary glands, in sex- and age-dimorphic patterns. A few mouse ESPs have been characterised in detail. ESP36 (also known as Gm20408) is produced in the lacrimal glands of juveniles and females but not adult males. Like all ESPs, it stimulates basal VSNs but the behavioural consequence of this is not currently known (Kimoto et al. 2007). In contrast, ESP1 (also known as Gm6084) is produced in the lacrimal glands of adult males and secreted into their tears (Kimoto et al. 2005). In a series of elegant experiments, it was shown that when female mice chemoinvestigate the faces of males, ESP1 activates a specific class of basal VSNs resulting in lordosis, i.e., the adoption of a stereotypic mating stance (Haga et al. 2010). By combining VNO immunohistochemistry for the immediate-early gene c-fos with in situ hybridization using specific Vmn2r probes, Haga et al. were able to identify the first pheromone-receptor-behaviour association in the mouse. ESP1 interacts with the VMN2R116 receptor (also called V2rp5) at its large, extracellular N-terminal domain, probably involving electrostatic charge–charge interactions (Yoshinaga et al. 2013). Consistent with the sexually dimorphic behaviour, increased neural activation patterns are observed in some female higher brain centres after ESP1 exposure, including the posteromedial cortical amygdaloid nucleus and ventromedial hypothalamus (Haga et al. 2010).

A subsequent in vitro study provided further insight into specificity of ESP binding (Dey and Matsunami 2011). Of five ESPs tested, only one, ESP5, activated VMN2R111 (also known as V2Rp2), while both ESP5 and ESP6 activated VMN2R112 (V2Rp1). These three receptors are all members of the Vmn2r A3 clade (spanning genes Vmn2r111 to Vmn2r117 on Chromosome 17), suggesting that this receptor cluster may have coevolved with the expansion of the Esp genes in rodents.

Future perspectives

In the post-genomic era, where numerous mouse genomes are available and large-scale efforts are underway to assign function to most mammalian genes (van der Weyden et al. 2011), those involved in VNO-mediated behaviour remain frustratingly obscure and under-resourced. Only a handful of the many hundreds of genes described here currently have mutant mice lines available. This is partly because clusters of highly identical, homologous genes can be technically problematic for gene targeting, and partly because genes with single exon coding sequences (like Vmn1rs) were not included in large-scale conditional knockout production programmes (Skarnes et al. 2011), but also because a lack of human orthologues makes these genes a low priority in the eyes of many. However, even though the system is nonfunctional in humans, a better understanding of the genes underpinning VNO-mediated behaviour would likely benefit behavioural research in general. For example, the use of single chemical signals to elicit stereotyped responses offers huge advantages for behavioural analysis of mouse models of psychiatric, neurodevelopmental, and neurodegenerative disease (Huckins et al. 2013).

VR gene families are also particularly poorly suited to identification by massive parallel sequencing. A recent analysis of the vomeronasal receptor repertoire in the sequenced genomes of 17 inbred mouse strains found that only 54 % of the C57BL/6J reference gene set could be identified (Wynn et al. 2012). To some extent this is due to the high levels of interstrain variation in Vmn1rs and Vmn2rs, but a significant number of genes are simply too similar to permit the unique mapping of short sequence reads to a reference genome. Other technical challenges continue to hinder research into these gene families. Many of the receptor genes are so similar that neither antibodies nor nucleic acid in situ hybridization probes can distinguish between them (Silvotti et al. 2007). The molecular mechanism underpinning the expression logic of the receptor genes hinders their expression in heterologous systems (Matsunami et al. 2009), though recent progress into identifying a molecular chaperone of V2Rs may have resolved this (Dey and Matsunami 2011). Nevertheless, a better understanding of the genes patterning the VNO is an essential step towards understanding how behaviour is wired in the mouse brain. For example, the finding that ESP1 promotes lordosis via VMN2R116-expressing neurons (Haga et al. 2010), when combined with emerging technologies for precise single-neuron visualisation and activation, should now permit the mapping of a full sexual behavioural circuit for the first time. This paradigm can be applied to additional behaviours once the cognate VRs for the other pheromones and kairomones described here are identified.

However, it is important to remember that, in a natural context, VNO-mediated cues are unlikely to be detected in isolation from other odour signals. It is now clear that some innate behavioural responses are mediated by specific neurons in the mouse MOE and Grueneberg ganglion, including aversion to certain volatile compounds emitted by predators (Brechbuhl et al. 2013; Dewan et al. 2013; Ferrero et al. 2011), while a range of aberrant social behaviours have been reported in MOE-deficient (or ablated) mammals. Therefore, to fully understand how olfaction instructs behaviour, integration of VNO-mediated signalling with other olfactory subsystems requires further investigation, both at the periphery and in the brain.