Introduction

Understanding the mechanisms of speciation is one of the most important research areas in evolutionary biology. In particular, speciation driven by ecological adaptation (ecological speciation) has attracted the attention of many researchers because it can cause rapid speciation even with gene flow, and many studies have reported empirical examples of and theoretical possibilities for ecological speciation (Dieckmann and Doebeli 1999; Coyne and Orr 2004; Doebeli and Dieckmann 2005; Rundle and Nosil 2005; Gavrilets and Losos 2009; Schluter 2009).

“Sensory drive” speciation is one example of ecological speciation in which sensory traits that are directly adapted to specific habitats induce the establishment of assortative mating (Boughman 2002). Under sensory drive, a female is more likely to choose a male whose mating signature can be more easily detected by their sensory apparatus. For example, visual systems adapted to different light environments recognize different coloration patterns as signals for choosing mating partners. Such visually mediated sensory drive is well documented in fishes such as guppies (Endler and Houde 1995), sticklebacks (Boughman 2001) and cichlids (Terai et al. 2006; Seehausen et al. 2008).

Among these, the cichlids of Lake Victoria in East Africa, which consists of approximately 500 species (Seehausen 1996; Turner et al. 2001), are of particular interest. These species are monophyletic (Meyer et al. 1990; Meyer 1993; Verheyen et al. 2003), and they are thought to have undergone speciation within the last 10,000–15,000 years based on the geological history of the lake (Johnson et al. 1996, 2000; Beuning et al. 1997; Scholz et al. 1998; Seehausen 2002). Sensory drive has been hypothesized to be one of the important mechanisms of this rapid adaptive radiation (Terai et al. 2006; Seehausen et al. 2008).

Kawata et al. (2007) have shown that ecological speciation caused by sensory drive can theoretically occur rapidly if the selection pressure on the sensory trait is neither too strong nor too weak. Their results were supported empirically by Seehausen et al. (2008), who investigated cichlids in several regions of Lake Victoria, and found that divergent adaptation of opsins (or visual perception) to the different light environments in regions with intermediate light gradients across depths induced divergence of nuptial coloration in males, which caused reproductive isolation. In contrast, no such divergence was found in regions with steep or shallow light gradients. These studies shed light on the mechanism of speciation by visually mediated sensory drive.

In this study, we theoretically examine two questions not investigated by Kawata et al. (2007). First, although Kawata et al. (2007) found that a single species/type can rapidly diverge into two descent species/types, divergence into several types would be necessary to explain the adaptive radiation of the Lake Victoria cichlids. In fact, many more than two cichlid species have been found in continuous habitats (Seehausen and Bouton 1997), but it remains unknown whether they could have diverged by adaptive radiation on a short time scale. One might think that multiple locally adapted phenotypes could emerge by repeated bifurcation of lineages, but the increased number of phenotypes would increase the migration load through the immigration of non-adapted phenotypes and make it more difficult for the adaptation to occur (Butlin et al. 2009, pp 81–85). Therefore, we hypothesized that the adaptive evolution of more than two phenotypes was much more difficult than that of two phenotypes, and we examined whether this can be caused by sensory drive. Second, we monitored the patterns of variation at selectively neutral genes in addition to the evolution of the genes involved in adaptation. Although Gavrilets et al. (2007) also examined variation at selectively neutral genes during ecological speciation theoretically, they considered a model in which the traits determining the viability and mating success were completely different. Therefore, their model is not considered to be that of sensory drive. In addition, we investigated the variation at selectively neutral loci in more detail by examining its correlation with model parameters such as past selection pressures or levels of migration. If the patterns of variation at selectively neutral genes change depending on past selection pressures or levels of migration, we may be able to infer these parameters by comparing the observed variation at selectively neutral genes with the expected variation obtained theoretically. Thus, observing the patterns of neutral variation would provide valuable information for understanding the process of adaptive radiation by sensory drive in nature.

To answer these two questions, we conducted individual-based simulations. While the previous theoretical studies assumed models with patches arranged in two circles laying one on top of another (Gavrilets et al. 2007) or two-dimensional structure (Kawata et al. 2007), we assumed a three-dimensional habitat representing one part of a lake, where individuals tend to migrate to their favorable patches with fitness dependent migration. Furthermore, we assumed that one sensory trait determines the viability and habitat preference of each individual and that a male whose mating trait is detectable by the sensory trait of females is more likely to mate with those females. We considered three factors related to fitness—viability selection, habitat preference and mating preference—and examined their effects and the migration effects on adaptive radiation and the pattern of variation at neutral genes.

Model

Habitat

We considered a cube-shaped habitat corresponding to a part of a lake’s water column and divided it into 5 × 5 × 5 patches. Each individual lives in one of these 125 patches. We assumed that the light environment changes gradually with depth. Consequently, the optimal phenotype for the sensory trait (the absorption spectrum of opsin in cichlids) also changes gradually. A detailed description of the fitness function for the sensory trait is given in the section on viability selection.

Individuals

Each individual is diploid and dioecious and is assumed to have L S loci controlling an equal number of sensory traits and L M loci controlling a mating trait (nuptial color in cichlids). Each locus has an infinite number of alleles, and the phenotypic contribution of each allele is represented by an integer (…, −2, −1, 0, 1, 2, …). We assumed a stepwise mutation model (Ohta and Kimura 1973) with mutation rates u S and u M at the sensory and mating trait loci, respectively. When a mutation occurs, alleles change their value by +1 or −1 with a probability of one half. For the sensory traits, each individual has 2 × L S sensory trait values because each of the 2 × L S alleles is assumed to express an integer-valued phenotype independently. This modeling was chosen because of the biology of the opsin genes of East African cichlids and animal olfactory receptor genes (Carleton and Kocher 2001; Buck and Axel 1991; Ota et al. 2012, see Discussion). In contrast, a mating trait such as the body color of East African cichlids is determined by one phenotypic value. For example, female cichlids prefer reddish color in deep areas but bluish color in shallow areas (Seehausen et al. 2008). Thus, we assumed that the mating trait of an individual is determined by the sum of the integer-valued phenotypic contributions of its 2 × L M alleles. In addition, each individual has 50 selectively neutral loci. Each neutral gene/allele consists of 100 nucleotides, which evolve according to the Jukes-Cantor mutation model (Jukes and Cantor 1969, pp 21–132) with a per-site mutation rate of u N. Finally, we assume free recombination between any pair of loci.

Generations are discrete and non-overlapping, and allele frequencies change during a given generation as a consequence of viability selection, habitat choice and mating choice, in this order. These three factors are explained in detail below.

Viability selection

The viability of each individual is determined by its sensory trait values and the population density in the patch. As explained above, we assume five different depths. Let θ d be the optimal sensory trait value at depth d, and assume θ d  = d. We define d T for an individual living at depth d as

$$d_{T} = \sum\limits_{k}^{{2L_{s} }} {\left( {\frac{{|d_{k} - \theta_{d} |}}{{5 \times 2L_{s} }}} \right),}$$
(1)

where d k is the sensory trait value of the kth gene. Thus, d T is a standardized sum of the absolute differences of the trait values from the optimal value. Using d T , the viability of the individual is assumed to be proportional to

$$W = \exp \left( {\frac{{ - \left( {d_{T} } \right)^{2} }}{{\sigma_{s}^{2} }}} \right),$$
(2)

where σ 2S quantifies the strength of viability selection for the sensory traits. W is used in the Beverton-Holt equation (Kot 2001; Gavrilets et al. 2007) to incorporate the effect of population density, and the viability, V, is given by

$$V = \frac{1}{{1 + \left( {b - 1} \right)\left( \frac{N}{KW} \right),}}$$
(3)

where N is the population size in the patch, K is the carrying capacity of the patch and b is the number of offspring per female.

Habitat preference

After viability selection, individuals can emigrate to one of the neighboring patches. In the present model, emigration occurs in three steps, which are defined as movement along the x-axis, y-axis and z-axis within the habitable space. In each step, an individual emigrates with a probability m (the emigration rate). The emigrating individual may change its patch coordinate by one along the given axis depending on its habitat preference. We assumed that each individual prefers the patch where it has a higher viability [W in Eq. (2)]. Thus, we calculated its preference for either of the two neighboring patches along a given axis and its current patch, and we used that preference to determine the actual emigration probability of the individual as explained next. Emigration along the x- or y-axes does not change viability. Thus, the patch coordinate of the individual increases or decreases by one with a probability of m/3, and the patch coordinate does not change when the individual does not move to another patch with a probability of (1−m) or when it moves but then returns to the current patch with a probability of m/3. On the other hand, for the z-axis, each individual has a preference because the light environment differs at various depths. We defined the preference for depth as

$$P_{H} = \exp \left( {\frac{{ - \left( {d_{T} } \right)^{2} }}{{\sigma_{H}^{2} }}} \right),$$
(4)

where d T is defined as in Eq. (1) and σ 2H determines the strength of habitat preference. We calculated P H for the upper neighboring depth (P Hu), the lower neighboring depth (P Hl) and the current depth (P Hc). Their values of each determine the probabilities with which the individual emigrates to a given depth: the patch coordinate of the individual along the z-axis increases by one with a probability of [mP Hu/(P Hu + P Hl + P Hc)] and decreases by one with a probability of [mP Hl/(P Hu + P Hl + P Hc)]; it does not change when the individual does not move (probability of 1-m) or when it moves but then returns to the current depth with a probability of [mP Hc/(P Hu + P Hl + P Hc)].

Mating preference

After migration, females mate with males in the same patch. In the present model, we assumed that each female can mate only once and chooses its mating partner depending on the total difference, D T , between her sensory trait values, d k , and the male mating trait values j,

$$D_{T} = \sum\limits_{k}^{{2L_{o} }} {\left( {\frac{{|d_{k} - j|}}{{5 \times 2L_{o} }}} \right)}$$
(6)

Then, the mating probability is given by

$$P_{M} = \exp \left( {\frac{{ - \left( {D_{T} } \right)^{2} }}{{\sigma_{M}^{2} }}} \right).$$
(7)

The males in the patch are presented to the female in a random order without replacement until she chooses one as a mate [with the probability given by Eq. (7)]. If P M < 1 for all males in the patch, the female may not be able to find a mate.

Next generation

After mating, each fertilized female produces b offspring in the patch. The genotype of each offspring is determined by a random sampling of gametes from its female and male parents. The sex of the offspring is determined randomly.

Initial conditions

The initial population is founded in the center patch of the habitat space (x = 3, y = 3, z = 3) and consists of 10 males and 10 females perfectly adapted to that patch (having allele 3 at all sensory and mating trait loci). All neutral loci are monomorphic with a sequence of nucleotide A.

Parameters

We investigated the effects of viability selection, habitat preference, mating preference and emigration rate by varying the parameters σ 2S , σ 2H , σ 2M and m, respectively. For σ 2S , σ 2H and σ 2M , we examined cases with four values (1, 0.1, 0.025 and 0.01). For m, we examined cases with three values (0.1, 0.05 and 0.01). We also examined the effects of the carrying capacity, K, with three values (500, 250 and 125). All combinations of these parameter values were investigated assuming three different genetic models: (1) 1 locus (L S = L M = 1), (2) 4 L M (L S = 1, L M = 4) and (3) 4 L S (L S = 4, L M = 1). Additionally, we briefly considered cases with more loci. For illustrative purposes, Table 1 shows viability selection, habitat preference, mating preference and mating probabilities when L S = 1, L S = 4 and the sensory trait value deviates from the optimal value by one mutation. For the other parameters, we used the fixed values of b = 10, u S = u M = 10−5 and u N = 10−6. All parameters used in the model are listed in Table 2.

Table 1 Fitness of the phenotype that deviates from the optimal by one mutation
Full size table
Table 2 List of the parameters used in the model
Full size table

Results

We observed the population state in the simulations after 10,000 generations because the rapid adaptive radiation of the cichlids in Lake Victoria is believed to have occurred within approximately 10,000 years and because the generation time for Lake Victoria cichlids is 0.5–1 year/generation (Terai, personal communication). After 10,000 generations, the number of phenotypes adapted to different habitats seemed to be stable and did not change even after a total of 20,000 generations. We defined adaptation of an individual to the current patch using two criteria: (1) more than 75 % of the 2 × L S sensory trait values are optimal in the patch and (2) the difference between the male mating trait and the optimal sensory trait values in the patch is less than 0.25 (for example, at z = 3, 2.75 ≤ the mating trait value ≤ 3.25). Note that we can also determine adaptation in the females based on their mating trait genes even though the genes are not expressed. If these criteria are satisfied by more than 50 % of the individuals in the patch, we considered adaptation to have occurred at that depth for that patch. Because we considered five different depths, there are at most five different adapted phenotypic clusters. We used the number of clusters as the number of adaptations in the following models.

Adaptive radiation in the three genetic models

We first examined whether adaptive radiation into more than two phenotypes occurred in the three genetic models. We replicated the simulation five times for each parameter set and calculated the average number of phenotypic clusters.

Figure 1 (for K = 500) and Supplementary Fig. 1 (for K = 250 and K = 125) show the results for the 1-locus model. Evolution of more than two adapted phenotypes occurred but did so under more restricted conditions than evolution of only two adapted phenotypes. We observed that divergent adaptation occurred by successive branching rather than simultaneously. Because the state did not change after 20,000 generations, this result suggests that simple repetitions of lineage splitting from 1 to 2 phenotypes did not always result in the evolution of a large number of phenotypes and that more stringent conditions were required for adaptive radiation to occur, in particular, with high emigration rates and intermediate strengths of viability selection and mating preference. On the other hand, the effect of habitat preferences seems weak. The carrying capacity also had a strong effect, with an increase in K leading to an increase in the number of adapted phenotypes. However, K did not qualitatively alter the effects of the other parameters (Supplementary Fig. 1).

Fig. 1
figure 1

The average number of adapted phenotypes for various levels of viability selection (σ 2S ), habitat preference (σ 2H ), sexual selection (σ 2M ) and emigration rates in the 1-locus model. The case for K = 500 is shown. We ran the simulation five times for each parameter set and calculated the average number of adaptations from these five replicates. The average number of adapted phenotypes is shown in gray scale

Full size image

Figure 2 (K = 500) and Supplementary Fig. 2 (K = 250, 125) show the results for the 4-L M model. Although the number of adapted phenotypes was smaller than in the 1-locus model, this difference was very small and the effects of the three factors related to fitness, the emigration rate and the carrying capacity were similar to those in the 1-locus model. These results suggest that small differences in the number of mating trait loci do not affect adaptation to new habitats.

Fig. 2
figure 2

The average number of adapted phenotypes for various levels of viability selection (σ 2S ), habitat preference (σ 2H ), sexual selection (σ 2M ) and emigration rates in the 4-L M model. The case for K = 500 is shown. As in Fig. 1, we calculated the average number of adaptations from five replicates and represented the average in gray scale

Full size image

Figure 3 (K = 500) and Supplementary Fig. 3 (K = 250, 125) show the results for the 4-L S model. In contrast to the mating-trait model, an increase in the number of the sensory trait loci greatly decreases the number of adaptive phenotypes. Intermediate strengths of viability selection and mating preference were more important for allowing adaptation to occur compared to the cases shown in Figs. 1 and 2. These results suggest strong effects of the number of sensory trait loci on the probability of adaptation. The sensory traits might have a stronger effect than the mating trait because they affect both natural and sexual selection. As for the two genetic models mentioned above, decreasing the carrying capacity made adaptation/divergence more difficult, and when K = 125, adaptation never occurred (Supplementary Fig. 3b).

Fig. 3
figure 3

The average number of adapted phenotypes for various levels of viability selection (σ 2S ), habitat preference (σ 2H ), sexual selection (σ 2M ) and emigration rates in the 4-L S model. The case for K = 500 is shown. As in Fig. 1, we calculated the average number of adaptations from five replicates and represented the average in gray scale

Full size image

Finally, the effect of isolation by distance seemed weak because, in almost all cases, adaptive phenotypes spread to all patches at the same depth, and there was no phenotypic differentiation along the horizontal axis (data not shown).

FST under different selection pressures and emigration rates in the 1-locus model

We calculated F ST values (Hudson et al. 1992) at the neutral loci to determine the extent of gene flow between divergent phenotypes. First, we calculated F ST assuming the 1-locus model in 276 out of the 576 parameter sets in which the average number of adapted phenotypes exceeded 1. We used a generalized linear model (Nelder and Robert 1972) with the strength of viability selection (σ 2S ), habitat preference (σ 2H ), mating preference (σ 2M ) and emigration rate (m) as the explanatory variables, and we assumed that the response variable, F ST, followed a Gamma distribution based on the Akaike Information Criterion (Akaike 1974). Table 3 shows the estimates of the coefficients of the explanatory variables in the linear model. Among the four factors, habitat preference (σ 2H ), mating preference (σ 2M ) and the emigration rate (m) had significant negative correlations with F ST, which shows that strong habitat and mating preferences and a low emigration rate effectively prevented gene flow between phenotypes. In contrast, however, the strength of viability selection (σ 2S ) did not have a significant effect on F ST. This observation was somewhat counterintuitive and will be discussed later.

Table 3 The results of the generalized linear model analysis between the F ST values and the strengths of viability selection, habitat and mating preference and emigration rate
Full size table

Differences in FST values between the three genetic models

Next, we examined how differences between the three genetic models affected F ST. We chose 30 parameter sets for which adaptation occurred in all three models and compared the F ST values between them (Fig. 4). Because the parameter range in which adaptation occurred in the 4-L S model was narrower than and mostly included in the other two models (Figs. 1, 2, 3), the 30 parameter sets correspond to the parameter sets of divergent adaptation in the 4-L S model. As shown in Fig. 4, the effects of the number of loci seemed to be small. In fact, the sign of the correlation between F ST and σ 2S , σ 2H , σ 2M or m was the same in all three genetic models (data not shown). However, the F ST values themselves were significantly different between the models in some parameter sets. While F ST differed significantly between the 1-locus and 4-L M models in only 3 parameter sets, it differed significantly between the 1-locus and 4- L S models and between the 4-L M and 4-L S models in about half of the 30 parameter sets. These results suggest that the number of sensory trait loci affects F ST more than the number of mating trait loci.

Fig. 4
figure 4

The average and standard error for the F ST values for the three genetic models. We show the results for 30 parameter sets where more than one phenotype appeared in all three of the genetic models: a K = 500, m = 0.1; b K = 500, m = 0.05; c K = 500, m = 0.01 and d K = 250, m = 0.1 and 0.05. *X–Y indicates that the F ST value was significantly different (p ≤ 0.05) between the genetic models X and Y

Full size image

More loci

We also investigated cases with more loci and describe the results briefly here. We considered two models—(1) 8 L M (L S  = 1, L M  = 8) and (2) 8 L S (L S  = 8, L M  = 1)—and calculated the average number of adapted phenotypes as described in the previous models (Fig. 5). In accordance with our previous results, an increase in the number of loci controlling the mating trait did not affect the results very much. In contrast, increasing the number of loci controlling the sensory trait greatly decreased the number of adapted phenotypes. We also found that the number of loci controlling the mating trait did not greatly affect F ST as in Fig. 4 (data not shown).

Fig. 5
figure 5

The average number of adapted phenotypes for various levels of viability selection (σ 2S ), habitat preference (σ 2H ), sexual selection (σ 2M ) and emigration rates in (a) the 8-L M model and (b) the 8-L S model. The cases for K = 500 are shown. As in Fig. 1, we calculated the average number of adaptations from five replicates and represented the average in gray scale

Full size image

Phylogenetic relationships among phenotypes

Finally, we examined the phylogenetic relationships among individuals of the adapted phenotypes. We chose four parameter sets that showed different F ST values and reconstructed Neighbor-Joining trees using MEGA version 4 (Tamura et al. 2007) for the sequences of the 50 neutral loci (Supplementary Fig. 4). Individuals with each phenotype were never monophyletic, which suggests continued gene flow. In particular, when F ST was low, individuals of all phenotypes seemed completely intermingled.

Discussion

Previously Kocher (2004) proposed that the speciation process of East African cichlids can be divided into three phases: (1) adaptation to rocky or sandy habitat, (2) adaptation to different feeding and (3) diversification of color patterns. Sensory drive accelerates the third phase through which populations in similar habitats can speciate even under gene flow; therefore, this mechanism would make a large contribution to the adaptive radiation of East African cichlids (Terai et al. 2006; Seehausen et al. 2008).

In this study, we developed a model of sensory drive, which allowed us to study the subsequent divergent adaptation of a mating trait and the establishment of assortative mating. Such sensory-driven speciation has been reported in fishes living in different light environments (Endler and Houde 1995; Boughman 2001; 2002; Seehausen et al. 2008). In a lake, the light environment varies gradually by depth (Seehausen et al. 2008), and so conditions such as those simulated in our model are likely to be found in nature. Using this model, we investigated two questions that have not been considered in previous studies: (1) the possibility of adaptive radiation (i.e., the emergence of more than two adapted phenotypic clusters) caused by sensory drive in the presence of gene flow and (2) the effects of viability selection, habitat preference, mating preference and emigration rate on the patterns of variation at selectively neutral loci.

We found that adaptive radiation was possible for specific values of the parameters representing the number of loci regulating the sensory trait, the selection pressures, the habitat preference, the emigration rate and the carrying capacity (Figs. 1, 2, 3 and Supplementary Figs 1, 2, 3). Because we did not observe differentiation along the horizontal axis, the radiation along the vertical axis under these parameter sets seemed to be caused by adaptive evolution and not by isolation by distance. The number of adapted phenotypes was five in these parameter sets, which was the maximum set by the number of ecological niches in the simulation. In the other sets, however, the number was less than five. Some of the possible reasons why the maximum was not achieved, including the increase of the migration load (Butlin et al. 2009), will be discussed later. Below, we first discuss the effects of the parameters in more detail.

Importance of a small number of loci regulating the sensory trait(s)

Figures 1, 2 and 3 suggest the importance of a small number of loci regulating the sensory trait. Generally, the probability of ecological speciation increases as the number of loci regulating ecological traits decreases because selection on individual loci becomes stronger (Gavrilets and Vose 2005). In sensory drive, the evolution of a sensory trait occurs first, and once a sensory trait has diverged, a mating trait is under strong sexual selection and will diverge itself under almost any circumstance. This is most likely why the number of loci regulating the sensory trait(s) has a larger impact on ecological speciation than does the number of loci regulating the mating trait(s) (Figs. 2, 3). Importantly, one opsin locus, the long-wavelength-sensitive opsin gene (LWS), is thought to play an important role in speciation of the cichlids in Lake Victoria (Terai et al. 2002, 2006; Seehausen et al. 2008). In addition, Miyagi et al. (2012) found remarkably higher between-species divergence at the LWS gene than at the other cone opsin genes (e.g., the blue-sensitive SWS opsin gene and green-sensitive RH2 opsin gene). This is in accordance with our model, which suggests that divergence occurs most easily when it is based on a single sensory locus (L S = 1).

Importance of intermediate levels of selection

Figures 1, 2 and 3 also suggest the importance of intermediate levels of viability selection and mating preference. The importance of intermediate selection pressures for ecological speciation has been repeatedly stressed in previous theoretical studies (Doebeli and Dieckmann 2003; Gavrilets and Vose 2007; Gavrilets et al. 2007; Kawata et al. 2007), and our results are consistent with those results. If natural or sexual selection is too weak, gene flow between different adapted phenotypes occurs and causes their admixture or extinction of one of them by genetic drift. In addition, in our model, the migration load becomes strong when sexual selection is weak because emigrants generate hybrids that have low viability and reduce of the population size. As shown in Figs 1, 2 and 3, the average number of phenotypes was always less than 3 when σ 2M  = 1, which suggests that the migration load makes it more difficult for evolution of the maximum number of phenotypes to occur. On the other hand, if natural or sexual selection is too strong, an invasion of new habitats or mating of individuals with new sensory or mating traits with the original phenotype becomes more difficult. Therefore, intermediate levels of selection pressure seem important for ecological speciation. Figures 1, 2 and 3 suggest that habitat preference has a smaller effect than do viability selection and mating preference and that adaptive radiation is possible in a wide range of habitat preference. Therefore, although Gavrilets (2004, pp 350–352) stated that habitat preference/selection is important for speciation to occur with gene flow, our results suggest that sensory drive can cause adaptive radiation even if habitat preference is very weak with almost no preference between two neighboring patches (when σ 2H  = 1, preference of an individual that has an optimal sensory trait in the current patch with that in the neighbor patch is 0.99 in L S = 1, see Table 1).

Importance of a high emigration rate and large carrying capacity

We also found that the number of adapted phenotypes is positively correlated with the emigration rate and the carrying capacity (Figs. 1 2, 3 and Supplementary Figs. 1, 2, 3). Both parameters increase the number of emigrants, some of whom may carry adaptive mutations into the new habitat, and thus increase the opportunities for adaptation to occur. Of course, higher emigration rates also increase gene flow between the differentially adapted phenotypes living at different depths and are expected to increase the migration load, which would lead to negative effects on adaptation and differentiation. However, in sensory drive, assortative mating is established rapidly and reproductive isolation will develop before substantial mixing of the differentially adapted phenotypes occurs unless sexual selection is too weak.

Here, because we started the simulations from a monomorphic population completely adapted to a single patch, divergence strongly depended on quick invasion of and adaptation to new habitats, which is facilitated by high emigration rates. Additionally, a larger carrying capacity may decrease the possibility that the invaders will become extinct. Because we did not examine other initial conditions, we do not know whether the effects of the emigration rate or the carrying capacity change when, for example, the initial population is widely distributed over the habitat space or when initial variation is present in the sensory traits or the mating trait.

Neutral variation and differentiation between phenotypes

We found that habitat preference, mating preference and the emigration rate significantly affected F ST between phenotypes. Therefore, we may be able to use F ST to infer the parameter ranges for the habitat preference, mating preference and emigration rates in natural populations, but an accurate estimation of each parameter value would not be possible from the F ST values alone. Another important result is that F ST values are robust to differences in the number of the mating trait loci. Therefore, inferring the parameters for the past population history (e.g., habitat preference, mating preference and the emigration rate) may be possible without knowing the detailed genetic architecture of the mating trait. In contrast, an increase in the number of sensory trait loci significantly changed the F ST values over a large range of parameters, and so identifying an accurate genetic model for the sensory trait is crucial for inferring the population history. As mentioned above, one opsin locus, LWS, is thought to play an important role in the sensory-drive speciation of the cichlids in Lake Victoria. Therefore, we can make some tentative inferences about this case using the results from the 1-locus model.

Previously, Samonte et al. (2007) concluded that populations of the same species genetically intermingle with those of the other species in the African cichlids of Lake Victoria based on an analysis of genetic variation at 11 loci. Moreover, Wagner et al. (2013) have recently shown that individuals of each species become monophyletic only when more than thousands of marker loci are used to infer their phylogeny based on a Genome-wide RAD analysis. The phylogenetic trees reconstructed in our study also showed a lack of monophyletic relationships among individuals with the same phenotype (Supplementary Fig. 4). Thus, the adaptive radiation of cichlids in Lake Victoria, which shows polyphyletic relationships among individuals belonging to a single species when a small number of loci are used for the inference, can be explained by our sensory drive model using parameter values such as those used in Supplementary Fig. 4. For example, Seehausen et al. (2008) estimated the F ST between two cichlid species, Pundamilia pundamilia and P. nyererei, to be 0.01–0.025 using data from 11 microsatellite loci. In our model, such low F ST values were found in only 11 parameter sets, 9 of which satisfied m = 0.1 and all of which satisfied σ 2M  ≥ 0.1. In addition, we may be able to use the number of adapted phenotypes for the parameter inference. In the case of Pundamilia, there are two types of LWS in the habitat (Seehausen et al. 2008). Of the above parameter sets, 9 of 11 showed two adapted phenotypes, 8 of which satisfied m = 1 and 7 of which satisfied σ 2M  = 1. This might suggest that P. pundamilia and P. nyererei had a high emigration rate and weak mating preference. Under such conditions, these two species should generate hybrids, and indeed, individuals with intermediate body color have been observed in natural populations (Seehausen et al. 2008). Terai et al. (2006) also estimated F ST between three populations of Neochromis greenwoodi, two populations (including its offshore incipient species N. omnicaeruleus) from one habitat and one population from another habitat. Using markers unlinked to LWS, the average F ST between populations with different habitats was estimated to be 0.032 and 0.043. Although these two habitats are separated by long distance and we may not be able to apply our results directly to infer the parameter values, the number of parameter sets under which F ST values were within 0.032–0.043 and only two phenotypes evolved was 38, 35 of which satisfied σ 2M  ≥ 0.1 and 29 of which satisfied σ 2H  ≥ 0.1. This might suggest that N. omnicaeruleus and N. greenwoodi had a weak mating preference and habitat preference.

In this study, we considered only selectively neutral genes unlinked to the selected loci to infer the population structure. However, F ST at selectively neutral genes may increase as linkage to the selected loci becomes stronger (Feder and Nosil 2010; Feder et al. 2012). Therefore, if we find an F ST value at a locus much larger than those at the other loci, the observation indicates that the locus is closely linked to or itself a selected locus.

No correlation between FST and strength of viability selection

In general, strong selection should result in strong reproductive isolation and is thus expected to increase F ST between divergently adapted phenotypes. However, in our study, F ST values did not increase as viability selection became stronger (Table 3). This means that although sexual selection affects the divergence at neutral loci, viability selection on the sensory trait does not. In sensory drive, evolution in the sensory trait is soon followed by evolution in the mating trait causing assortative mating. Then, the strength of viability selection does not matter much for the accumulation of the genetic differentiation at the neutral loci. In addition, the density-dependent viability selection (see Eq. 3) allows individuals with non-adaptive mutations to survive when the population density is low even if the viability selection is strong. These surviving individuals may migrate to other favorable patches. For these reasons, the effect of the level of viability selection on population differentiation would be smaller than those of the other factors.

Application to other organisms

As mentioned earlier, our result in a three-dimensional habitat would be applicable to fishes that speciated by sensory drive. Previous studies suggested that male coloration and female light sensitivity have caused speciation by sensory drive in guppies (Endler and Houde 1995; Hoffman et al. 2007) and sticklebacks (Boughman 2001). Therefore, we may be able to apply our model and results directly for these species. For example, if we know the number of loci encoding the opsins that caused the speciation, we may be able to infer the range of the model parameters such as mating preference and the emigration rate. Moreover, in East African cichlids, mating sound and olfaction are also known to affect the mate choice (Amorim et al. 2004; Plenderleith et al. 2005). As reviewed in Salzburger (2009), there is some evidence showing that these traits are ecologically adaptive. Therefore, if we know the genetic architecture of these traits, our theoretical result may help reveal the role of sensory drive in sound or olfaction in East African cichlids. Finally, a recent study suggested that Amazonian bird species have speciated by acoustic adaptation (Tobias et al. 2010). Because the sound transmission property is different in different habitats (bamboo forest and terra firme forest in this case), adaptation of the male song might contribute to the establishment of reproductive isolation. Again, this case might be analyzed using our theoretical results, but some modification such as an exchange of the horizontal and vertical axes and an adjustment of the number of environments may be necessary.

Finally, we briefly discuss the validity and effect of changing some of the assumptions we made in our model.

First, we assumed that emigration occurs after viability selection. If emigration occurs before viability selection, individuals with a new mutation in the sensory trait can move to patches in which they have higher fitness. This would increase the viability of new phenotypes produced by mutations and thus make adaptive evolution more likely to occur.

Second, in our model, a female might not mate if the probability of mating with this female for any male in the patch is less than one. Thus, if the mating preference is strong, a female with a new phenotype for the sensory trait may not be able to produce offspring. If we assume that all females can mate as assumed by Kawata et al. (2007) and Gavrilets et al. (2007), then adaptive evolution would be more likely to occur because a female having a new phenotype for the sensory trait can mate with the probability of one and produce offspring.

Third, we assumed that each individual had its habitat preference determined by its sensory trait and environment. Although the cichlids in Lake Victoria usually inhabit the optimal light environment, it is not clear whether this is because of habitat preference or the result of selection. However, as mentioned in Importance of intermediate levels of selection, adaptive radiation was possible even if habitat preference was very weak. Thus, we conclude that sensory drive can cause adaptive radiation without a strict qualification about habitat preference.

Fourth, we assumed that each gene independently expresses an integer-valued phenotype in the sensory trait [see eq. (1)]. We might have modeled the sensory trait in such a way that selection operates on the sum of the effects of the 2 × L S sensory trait alleles as in the mating trait rather than independently on the effect of each allele. However, it is known that African cichlids have multiple opsin genes and that each of them expresses a specific cone opsin with a different absorption wavelength (Carleton and Kocher 2001). Animals are also known to have many olfactory receptor genes and can distinguish many types of odor (Buck and Axel 1991). A recent study has shown that African cichlids also have and express many different olfactory receptor genes within one individual (Ota et al. 2012). Thus, our assumption that each allele independently expresses a different sensory trait would be more realistic than the model that assumes the additive determination of one sensory trait.

Fifth, we assumed free recombination between loci. Tight linkage between loci would accelerate the accumulation of good combinations of alleles and would make adaptation more likely to occur.

In summary, some of our assumptions are based on the nature of cichlids in Lake Victoria. On the other hand, the other assumptions were made because of lack of knowledge. However, all these assumptions make adaptive evolution more difficult. Thus, if one or more of them are violated, adaptive radiation would occur over an even wider parameter range.

Conclusion

In this study, we found that sensory drive leads to adaptive radiation even when gene flow occurs between the adapted phenotypes. This shows that sensory drive is one of the important factors that caused the adaptive radiation observed in the cichlids of Lake Victoria. We also found that the F ST values between rapidly diversifying phenotypes at neutral loci varied considerably depending on habitat preferences, mating preferences and emigration rates. Interestingly, the F ST values were almost the same regardless of small differences in the number of loci controlling the mating trait. These results suggest that F ST values can be used to estimate the past population history without a fully accurate genetic model for the mating trait.