1 Introduction

Negative density dependence (NDD) is regulating tree populations at every developmental stage from the seedling stage to maturity (Harms et al. 2000; Peters 2003; Wu et al. 2016a). Many previous studies, using seedlings, have attempted to document NDD by examining the relationship of plant survival, recruitment or growth with the densities of conspecific neighbors (Webb and Peart 2000; Chen et al. 2010; Comita et al. 2010; Johnson et al. 2012; Lin et al. 2012). Focal plants were negatively as well as positively impacted by conspecific and heterospecific neighbors, depending on local abiotic conditions, microbial activity, insects, and other animals (Lebrija-Trejos et al. 2014). Positive interactions found with heterospecific neighbors might result from facilitation, specific responses to abiotic conditions and/or the so-called species herd protection hypothesis (Comita and Hubbell 2009; Peters 2003; Lebrija-Trejos et al. 2014). Negative interactions with conspecific neighbors might be caused by shared pests and/or by competition for limiting resources (Lebrija-Trejos et al. 2014). A major mechanism, known as the Janzen-Connell hypothesis, proposed that the maintenance of diversity is facilitated by a conspecific negative density dependence (CNDD), whereby the proximity to adults of the same species reduces seedling survival rates through attacks by host-specific adversaries (Janzen 1970, 1972). Furthermore, distance dependence, a basic part of the Janzen-Connell hypothesis, has been extensively verified in tropical forests. Some evidence of distance dependence effects were found in temperate forests, and more evidence is required in order to to expand the range of the verification.

Previously, negative density dependence was not considered as a universal biodiversity maintaining mechanism in forest communities, because only a few abundant species showed it (Hubbell 1979; Hubbell et al. 1990). However, later studies found that negative density dependence was hidden by limitations of survey methods, spatial heterogeneity, and natural disturbances (Zhu et al. 2009). Research methods gradually improved. When spatial heterogeneity was excluded, it was found that population density was regulated by the effects of negative density dependence (Hubbell et al. 1990; Wills et al. 1997; Peters 2003; Zhu et al. 2009).

Simply dividing the species into conspecific and heterospecific species could hide the difference in the effect of different species on a focal species (Pacala et al. 1996). Recent studies have shown that phylogenetic relatedness of neighbors is an important predictor of NDD, a pattern referred to as phylogenetic density dependence (Webb et al. 2006; Gonzalez et al. 2010; Metz et al. 2010; Ness et al. 2011). This pattern is consistent with the observation that herbivorous insects are frequently associated with clades of host plants (Novotny et al. 2010) and that pathogenic transmission between pairs of tree species is more likely to occur if the species are phylogenetically related (Gilbert and Webb 2007). The indices of phylogenetic dissimilarity (Wu et al. 2016a), as well as phylogenetically correlated plant traits (Coley and Barone 1996), are accepted to be applications of the phylogenetic approach. However, the results of phylogenetic density dependence vary. Several previous studies have provided some evidence to support phylogenetic negative density dependence (PNDD; Bagchi et al. 2010; Metz et al. 2010; Paine et al. 2012). In addition, some studies have identified phylogenetic positive density dependence (PPDD; Lebrija-Trejos et al. 2014; Zhu et al. 2015; Wu et al. 2016a). Therefore, a large amount of evidence is required to confirm the existence of phylogenetic density dependence.

Both in theoretical models and in experiments, it has been shown that negative density dependence (NDD) could reduce interspecific competition exclusion and improve species diversity (Harms et al. 2000). Most field experiments did not consider the disturbance caused by other factors such as habitat heterogeneity which may affect the accuracy of the model parameters (Clark et al. 1998; Dieckmann et al. 1999; Condit et al. 2000; but see Zhu et al. 2009). Detecting NDD effects may therefore be difficult (He and Duncan 2000; Wright 2002; Getzin et al. 2008). Some studies divided the habitat into different types to examine habitat preferences of certain species by analyzing the association between species occurrence and certain habitat variables, like topography, radiation, and availability of soil nutrients and water, at the seedling stage (Webb and Peart 2000; Comita et al. 2007; John et al. 2007; Comita and Engelbrecht 2009; Metz 2012). It is thus likely that habitat preferences and NDD operate simultaneously to produce the observed species composition and resulting population dynamics (Comita et al. 2009; Chen et al. 2010; Bai et al. 2012; Piao et al. 2013). In order to verify such a scenario, nested models, which consider the effects of density, with and without an abiotic context, are required. Therefore, it would be of interest to know how habitat filtering regulates conspecific and phylogenetic density dependence.

The purpose of this study is to explore the effect of population density on population structure and the influence of different environmental gradients on negative density dependence (NDD). The specific questions which we strive to answer are the following: (i) How does population density affect the NDD? (ii) How does habitat filtering regulate the NDD? (iii) Can additional evidence be found for phylogenetic negative density dependence (PNDD) or phylogenetic positive density dependence (PPDD) and why? (iv) Is the intensity of negative density dependence affected by the distance between parent trees and seedlings?

2 Methods

2.1 Study site and seedling quadrates

Our study was conducted within a 20-ha mixed dynamic primary broad-leaved Korean pine forest plot (BKP) (Fig. 1a, b), located at the Changbai Mountain in Jilin Province, northeastern China (42° 20′ N, 127° 54′ E). The elevation of the study area ranges from 975 to 997 m (Fig. 1c). The area is characterized by a temperate continental climate, with cold windy winters and wet summers. The mean annual temperature is 2.9 °C, and precipitation amounts are 600 to 900 mm. The soil is classified as a dark brown forest soil (Wu et al. 2016b). The BKP plot (400 × 500 m) was established in 2014. All of the woody stems with diameters at breast height (DBH) ≥ 1 cm were tagged, identified, measured, and mapped (see detailed methods in Condit 1998). The dominant tree species are Korean pine (Pinus koraiensis Sieb. et Zucc.), Amur linden (Tilia amurensis Rupr.), Manchurian ash (Fraxinus mandschurica Rupr.), Mongolian oak (Quercus mongolica Fisch. ex Ledeb.), and Mono maple (Acer mono Maxim.). The basal area of our mixed primary broad-leaved Korean pine forest stand was 42.82 m2/ha.

Fig. 1
figure 1

The spatial distribution in 20-ha mixed dynamic primary broad-leaved Korean pine forest plot (BKP) in northeastern China. a Map of China showing location of the study area. b A topographic map of BKP plot. c Contour map of BKP plot. d The layout of seedling quadrats in BKP plot

Full size image

From May to August 2016, a total of 130 seedling quadrates (2 m × 2 m, 4 m2) were established in a regular pattern within the center of each 40 m × 40 m subplot in BKP plot (Fig. 1d). Where obstacles such as streams, large trees, rocks or fallen trees prevented the establishment of a seedling quadrate, we placed it instead in a nearby 5-m × 5-m subplots. In each of the 130 seedling quadrates, all of the woody plants (trees, shrubs and lianas) as well as seedlings with DBH < 1 cm were tagged, identified by species, and measured for height. In this study, all seedlings with DBH < 1 cm were used as focal seedlings, because what we want to explore is the density dependence of the whole seedling stage. Seedling quadrats were censused once a month through the whole growing season (from May to August), a total of four times. In each census, the states (alive or dead) of all the woody seedlings alive at the previous census were recorded and all new recruits were identified and tagged.

2.2 Neighborhood variables

We defined the total seedling neighbor density of each seedling quadrate as the number of seedlings within the quadrate. The conspecific and heterospecific seedling neighbor densities were defined similarly. The seedlings which were impossible to classify by species were included in the heterospecific neighbor counts, but not as the focal seedlings. The total adult neighbor density (TA) was calculated as the summed basal area (BA) of the nearby adults weighted by their distances to the focal seedling (Canham et al. 2004) as follows:

$$ \mathrm{TA}=\sum \limits_i^N\frac{{\mathrm{BA}}_i}{{\mathrm{Distance}}_i} $$

where N is the number of adult neighbors. The conspecific and heterospecific adult neighbor densities were calculated in the same way. The density models were calculated over distances of 5, 10, 15, and 20 m, in order to discuss which had the stronger support. All our seedling quadrates were within 40 m of the edge of the BKP plot and therefore had incomplete adult neighbor density values.

2.3 Construction of phylogenetic trees and indices of phylogenetic dissimilarity

Four phylogenetic diversity indices were used to quantify phylogenetic dissimilarities between the focal seedlings and their heterospecific neighbors in our analyses. The indices included total phylogenetic diversity (TOTPd), average phylogenetic diversity (AVEPd), relative average phylogenetic diversity (APd′) and relative nearest taxon phylogenetic diversity (NTPd′). A phylogenetic tree was built for adult trees, as well as for the seedling species occurring in the plot, using a Phylomatic program (Webb and Donoghue 2005) based on APGIII (Angiosperm Phylogeny Group 2009). Using this tree algorithm, the four phylogenetic diversity indices were separately calculated, from the focal seedlings to all the other heterospecific seedlings within the plot, as well as the heterospecific adult neighbors within each of the 5, 10, 15, and 20 m radial plots. We recalculated the four phylogenetic diversity indices; as a result, the models with densities calculated APd indices had stronger support than those indices with densities otherwise calculated. The sAPd′ and aAPd′ represent for phylodiversity between the focal seedling and the heterospecific seedling neighbors and adult neighbors, respectively.

2.4 Habitat variables

The habitat variables for each of the 130 target seedling quadrates were characterized using the following measurements: canopy openness (light), soil moisture (SM), pH, litter thickness (LT), and herbal density (HD). The BKP plot is relatively flat and has no special topography such as ditches, ridges and valleys, so the effect of topography on seedling survival was not considered in this study.

Canopy openness

For each seedling quadrat, we used hemispherical photographs to measured canopy openness, it indicated the light condition in the understory. Hemispherical photographs were taken 1.3 m aboveground at the center of each quadrat, using a Nikon Coolpix 4500 camera equipped with a Nikon FC-E8 Fisheye Converter lens (Tokyo, Japan) in January 2014. For each quadrat, photographs were taken in uniformly overcast weather once a mouth. The photograph showing the highest contrast between sky and foliage for each quadrat was selected. Digital Plant Canopy Imager CI-110(China) was used to process photos and to calculate the light transmittance.

Soil moisture

For each seedling quadrat, we used a HH2 Moisture Meter (made in the UK) to measure soil moisture once a mouth. By inserting the probe into the soil, instantaneous soil moisture data were obtained. Average values were used for each quadrat.

pH

We randomly chose two soil samples (500 g) which were taken from surface layers (0–20 cm) in each seedling quadrat and analyzed the samples at the laboratory. The Precision pH Meter Tester PHS-25 (made in China) was used to determine the soil pH value. The mean of the two samples was taken as the final value of each quadrat measurement.

Measuring effects of litter thickness

Surface litter intercepted and captured a large amount of rainwater, thus increasing soil moisture which effectively inhibited forest soil water evaporation and forest soil moisture. The effects of different litter thickness on soil moisture were assessed. We selected three random points in each seedling quadrat to measure litter thickness.

Herbal density

Two herb quadrats (1 m × 1 m) were placed on the side of each seedling quadrat (Fig. 1d); all the herbaceous plants were tagged, identified by species and measured for abundance. Each quadrat was surveyed once a month.

2.5 Statistical analysis

The seedlings individual survival (lived/died) from May to August 2016 was modeled as a logistic function of the neighborhood densities, habitat variables, and phylogenetic relatedness using generalized linear mixed models (GLMMs) applied in the lme4 package in R 3.0.2. To exclude spatial auto-correlation caused by some unexplored factors (Wu et al. 2016a), we added tree seedling species and quadrats as random effects our models in our study. The focal seedling height was log-transformed, and all continuous explanatory variables were standardized before analyses. Four models (Table 1) were established to explore the effect of habitat filtering on the detection of CNDD and PNDD, following the method applied by Paine et al. (2012) and Wu et al. (2016a). The four models are presented in Table 1: a density-dependent model (model I), a density + habitat model with the same neighborhood variables as those in the density-dependent model (model II), a phylogenetic density dependent model (model III), and a phylogenetic + habitat model with the same neighborhood variables as those in the phylogenetic density-dependent model (model IV). Akaike’s information criterion (AIC) was used to compare models.

Table 1 Description of the four models
Full size table

3 Results

3.1 Distance dependence for seedling survival in the best-fitting model

The density + habitat model (model II) and the phylogenetic + habitat model (model IV) were found to indicate stronger negative density effects than the other models (without habitat variables) at the 15-m radial scale (Table 2; Fig. 2). We did not observe any significant interactions between the probability of seedling survival and habitat variables and the seedling-seedling densities or the phylogenetic neighbor densities. Therefore, we assume that there is a density threshold in the density dependence. Models II and IV with a 15-m radius were chosen to verify whether seedling survival rates were affected by both neighboring seedlings and adult neighbors. Simultaneously, we explored whether or not phylogenetic density played a key role in the survival rates of seedlings.

Table 2 The AIC and BIC values for the comparison of four models at scales of 5, 10, 15, and 20 m
Full size table
Fig. 2
figure 2

Neighborhood effects of conspecific adult neighbor density on seedling survival at scales of 5, 10, 15, and 20 m with four models (models I, II, III, IV). a Density-dependent model (model I). b Density + habitat model (model II). c Phylogenetic density-dependent model (model III). d Phylogenetic + habitat model. Estimated coefficients (± SE) from models are shown separately for four scales. The black circles indicate significant effects (P < 0.05), gray circles signify marginally significant effects (0.05 < P < 0.1), and white circles mean no significance. Double negative signs: significant negative effects, single negative sign: signify marginally significant negative effects

Full size image

3.2 Density dependence affects by population density

The observed densities of the focal seedlings per 4 m2 (D4) were divided into four classes (less than 20 seedlings/4 m2; 20 ≤ D4 < 30 seedlings/4 m2; 30 ≤ D4< 40 seedlings/4 m2; D4 ≥ 40 seedlings/4 m2). Two models (II and IV) were established with these four units. The results show an increasing trend of significant seedling-seedling conspecific negative density dependence (CNDD) with rising of threshold (Table 3).The effect of seedling-seedling CNDD was especially significant, when densities were ranged from 20 to 40 seedlings/4 m2. Conspecific adult neighbor densities had a significantly negative impact on seedling survival only when the density threshold < 20 seedlings/4 m2 (Table 3). When density threshold was between 30 and 40 seedlings/4 m2, seedling survival rates were significantly positively impacted by heterospecific adult neighbors in model II (Table 3). We found negative effects of phylogenetic seedling-seedling diversity, indicating that increased phylogenetic similarities between heterospecific neighbors and focal seedlings caused an increase in seedling survival (Table 3). Seedling survival rates were significantly and positively impacted by the phylogenetic relatedness of seedling neighbors but were negatively affected by the phylogenetic relatedness of adult neighbors (Table 3). Seedlings surrounded by more closely related seedling neighbors had a higher probability of survival. Both seedling-seedling and seedling-adult phylogenetic density dependence were discovered when the focal seeding density< 20 seedlings/4 m2 (Table 3). Across all units, seedling survival rates were significant positively impacted by soil moisture.

Table 3 Coefficient estimates for all explanatory variables in the density + habitat model (model II) and the phylogenetic + habitat model (model IV) with four classes of focal seedling population density
Full size table

The population density of all 23 species was at least 30 seedlings/4 m2 in the study area. Fifteen of the 23 species had a density of 30 to 40 seedlings/4 m2. Only 4 of the 23 species had a density higher than 40 seedlings/4 m2. The species-level effects of conspecific seedling survival rates were negative for adult neighbors in 91% of the species and for seedling neighbors in 13% of species when the population density ≤ 20 seedlings/4 m2 (Fig. 3a, e). A population density> 20 and ≤ 30 seedlings/4 m2 also had a significant effect on seedling survival for adult neighbors in 13% and for seedling neighbors in 56% of species (Fig. 3b, f). A population density> 30 and ≤ 40 seedlings/4 m2 also had a significant effect on seedling survival for adult neighbors in 20% and for seedling neighbors in 93% of all species (Fig. 3c, g). Finally, a population density> 40 seedlings/4 m2 had no significant effect on either adult or seedling neighbors (Fig. 3d, h). Seedling survival rates decreased for most species with increasing conspecific population density.

Fig. 3
figure 3

Histograms showing P values (x-axis) of conspecific seedling and adult neighbor densities on survival for 23 tree species in the Changbai Mountain 20-ha forest dynamics plot. For seedlings, two types of neighbors were analyzed as follows: seedlings and adults. Dashed lines are at zero, so that bars to the left of the line indicate a negative effect of the neighborhood variable on survival, while bars to the right indicate a positive effect on survival. a, e D4 < 20 seedlings/4 m2. b, f 20 ≤ D4 < 30 seedlings/4 m2. c, g 30 ≤ D4 < 40 seedlings/4 m2. d, h D4 > 40 seedlings/4 m2

Full size image

3.3 How does habitat filtering regulate density dependence

As can be seen in Table 3, the seedling survival was affected by different habitat variables, and the strengths of these effects were different. We propose that habitat filtering will result in different effects of density dependence by controlling the habitat conditions. Soil moisture was highly correlated with survival in each density threshold unit with each model (Table 3). Therefore, we used soil moisture conditions as an example to verify the proposed hypothesis (Table 4). The median value of the soil moisture was 45%. Therefore, we divided the soil moisture variable into three gradients (soil moisture ≤ 42, 45, and 48%) to establish the models. Seedling survival rates were significantly negatively affected by conspecific seedling neighbors when soil moisture ≤42 or ≤45%. Conspecific adult neighbor densities had a significant negative impact on seedling survival only when soil moisture ≤ 42% (Table 4). Both heterospecific adult neighbor densities and heterospecific adult neighbor densities positively affected seedling survival when soil moisture ≤ 45%. Seedling survival rates were positively impacted by phylogenetic relatedness of seedling neighbors in each unit and positively affected by phylogenetic relatedness of adult neighbors when soil moisture ≤ 48% (Table 4). It seems that the intensity of radiation on seedling survival was enhanced with decreasing soil moisture thresholds (Table 4). Soil moisture was low where radiation was strong in microhabitats, and the survival limitation caused by soil moisture was reduced. We also found that seedling survival was only significantly and negatively correlated with litter thicknesses (LT) in one case (soil moisture ≤ 48%) (Table 4). We concluded that if the LT layer is thicker, it will prevent the rainwater to reach the soil. As a result, both the soil moisture and consequently the probability of seedling survival will be affected. Seedling survival was significantly and positively affected by herbal density, an important habitat variable (Table 4). This result was not expected.

Table 4 Coefficient estimates for all explanatory variables in the density + habitat model (model II) and the phylogenetic + habitat model (model IV) with three classes of soil moisture
Full size table

4 Discussion

In this study, we used a data set of 3268 seedlings including 23 woody plant species in a 20-ha primary broad-leaved Korean pine forest plot, located in the Changbai Mountain area of northeastern China. We explored the relative importance of conspecific negative density dependence (CNDD) and phylogenetic density dependence which affected by habitat filtering and population density in seedling survival using generalized linear mixed models (GLMMs). The models were built for seedlings which depend for their survival on the densities of conspecific and heterospecific neighbors, considering the phylogenetic dissimilarities between the heterospecific neighbors and the focal seedlings. Each of these models was developed with and without habitat variables, in order to determine the degree to which habitat filtering affected the apparent prevalence of NDD. We analyze the distance effect with each model to explore the influence of distance on density dependence. In particular, models were built with added thresholds of population density and habitat variable gradients.

4.1 The effects of distance on density dependence

The effects of density dependence and distance dependence on plant populations are both important. Distance dependence is the most fundamental part of the Janzen-Connell hypothesis and widely verified (Hubbell and Foster 1983; Wright 2002). Connell (1971) found a strong dependence effect on seedling survival and parent tree distance. Hyatt et al. (2003) found greater distance dependence effects in seedlings than in seeds. Wright (2002) and Petermann et al. (2008) reported that the effects of distance dependence on community dynamics were underestimated in most studies including BCI plots. There are many reasons why the universality of distance-dependent effects in plant communities is underestimated, one of the reasons being seed dispersal limitation. Seeds usually fall close to the parent trees, and the seedlings are mostly gathered near their parent trees. If we just analyzed neighboring individual numbers, we might underestimate the effect of distance dependence. Therefore, we calculated the total adult neighbor density (TA) as the summed basal area (BA) of the nearby adults weighted by their distances to the focal seedling. Four radial scales with distances from the adult neighbors to the seedling quadrates of 5, 10, 15, and 20 m were used.

In this study, we analyzed the relationships between seedling survival rates, environmental factors, phylogenetic relatedness, seedling neighbor densities, and adult neighbor densities within these four radii (5, 10, 15, and 20 m). The models with densities calculated over a distance of 15 m were found to have the strongest effect. The effect of distance on density dependence was very strong. Wu et al. (2016a) found that a distance of less than 20 m had strong effects when they studied density dependence within a tropical forest in Xishuangbanna China.

4.2 The evidence of PNDD and PPDD

Several studies have focused on the effect of heterospecific neighbors on negative density dependence based on phylogenetic relatedness (Metz et al. 2010; Paine et al. 2012). For example, Liu et al. (2012) evaluated the phylogenetic Janzen-Connell effect, which may be caused by associated host-specific fungal pathogens in subtropical forests. In our results, we found evidence of phylogenetic negative density dependence (PNDD) in agreement with the studies of Metz et al. (2010) who found that seedling survival rates increased in cases where nearby adult neighbors were more distantly related to the focal seedlings. The critical factors affecting pathogen infection of a host plant are known to be morphological and biochemical, which are often phylogenetically conserved (Mitter et al. 1991). There is much empirical evidence that closely related species which also have several similar key functional traits are more likely to share the same or similar pests and pathogens (Novotny et al. 2006; Gilbert and Webb 2007; Gilbert et al. 2012; Liu et al. 2012; Yang et al. 2014). Therefore, the effects of neighbors on a focal plant should be dependent on their phylogenetic similarities and should be less negative for plants which are less related. However, Wu et al. (2016a) found that seedling survival rates were higher among closely related heterospecific neighbors and considered the possibility that unobserved habitat factors may have confounding effects, although it was not very clear what these factors might be. However, there was evidence of phylogenetic positive density dependence (PPDD). Our results were in line with the findings of Wu et al. (2016a, b) in the Xi Shuang Ban Na tropical forest and consistent with the studies of Lebrija-Trejos et al. (2014) and Zhu et al. (2015) in the Barro Colorado (BCI) plot. Especially, Lebrija-Trejos et al. (2014) found a positive relationship between first-year seedling survival and the proportion of closely related heterospecific neighbors in the BCI plot. In our study, all of the woody plants as well as seedlings with DBH < 1 cm were tagged; earlier seedling stages were explored in models, including the first-year seedlings. The strength of the effect of seedling-seedling PPDD increased with higher soil moisture (Table 4). Such evidence is likely the result of closely related species sharing similar habitat resources (Zhu et al. 2015).

4.3 The effects of the population density on density dependence

Conspecific seedling-seedling and seedling-adult negative density dependence (CNDD) have been reported many times in tropical and subtropical as well as temperate forests (Johnson et al. 2012, Piao et al. 2013, Zhu et al. 2015; Wu et al. 2016a, b). In less dense patches (population density < 20), both conspecific adult and seedling neighbor densities had significant negative impact on seedling survival. A possible explanation is that there may be many more conspecific seedling neighbors around their presence over the wider area (beyond our seedling quadrats) (Wu et al. 2016a). The impact of conspecific neighbors on survival appears to be substantially larger with high population density; such relationships may be involved in intraspecific competition for shared resources. Meanwhile, clustering of conspecific individuals may attract more natural special pests and pathogens (Janzen 1970; Connell 1971). Another possible explanation for conspecific density dependence might be expected to be stronger in communities with high population density, reaching its limit in single species stands where self-thinning laws are applicable (Niklas et al. 2003).

4.4 The effects of the habitat filtering on density dependence

The role of density dependence, without considering the effect of habitat heterogeneity, may be misinterpreted (Piao et al. 2013). Tests for community level consequences of density dependence must account for habitat heterogeneity (Chen et al. 2010). Without accounting for habitat heterogeneity, conspecific thinning is likely to result from unfavorable habitat and not from tree-tree interactions (Piao et al. 2013). Habitat heterogeneity may explain more mechanisms of density dependence. Wu et al. (2016a) have shown that focal species with habitat variables considered suffered stronger negative density dependence effects than those without habitat variables considered.

In our study, soil moisture was factored out. Seedling survival was most strongly impacted by the density of conspecific seedling and adult neighbors in relatively low soil moisture habitats and weaker conspecific negative density dependence (CNDD) in relatively high soil moisture habitats. Usually, the activities of species- or genera-specific pathogens and other specialized natural enemies may decrease in areas of low soil moisture, which might also cause weaker CNDD there, an example of habitat filtering. Such relationships are likely the result of the effect of habitat carrying capacity; because of the scarcity of resources (e.g., soil moisture), the competition of conspecific individuals for shared resources should lead to negative density dependence. Seedling-seedling competition is strong in our forest, in contrast to some studies in tropical forests (Paine et al. 2008; Svenning et al. 2008; Zhu et al. 2015). For example, Lebrija-Trejos et al. (2014) discounted direct seedling-seedling competition, because they considered that seedling neighbors rarely have direct contact with one another and their impacts on resource use are likely to be slight. However, unfortunately, we could not disentangle the activities of specialized natural enemies from intraspecific competition given the observational nature of our analyses. Both heterospecific seedling neighbor densities and heterospecific adult neighbor densities positively impact on seedling survival. Seedling survival was significantly positively impacted by herbal density at medium soil moisture localities. These findings may provide evidence for another theory, the “species herd protection hypothesis” (Peters 2003). Heterospecific neighborhoods are known to result in fewer encounters between a host and its species-specific pests and pathogens, a condition which reduces the transmission of natural enemies and increases rates of survival (Peters 2003). The hypothesis explicitly considered the implications of biotic interactions mediated by heterospecific neighbors, which may be seen as an extension of the Janzen-Connell hypothesis.

In our study, we found evidence of a phylogenetic positive density dependence (PPDD) among the seedling-seedling relations when soil moisture ≤ 42% and ≤ 45% (Table 4). This result is in line with the study of Sedio et al. (2012), who found that both hydraulic traits and species’ responses to water availability were phylogenetically based, leading to phylogenetic clustering of species within microhabitats. The effect of seedling-seedling PPDD was enhanced with an increase in soil moisture, which may reflect habitat preferences resulting from habitat filtering: trees survive well and occur at higher densities in the most suitable habitat for the species (Zhu et al. 2015). The heterogeneity of spatial resources may be masking the signs of potential negative density dependence. It is not only difficult to test negative density dependence in habitats suitable for population growth, but it will also give the illusion of a positive density dependence effect (Zhu et al. 2009). It has also been suggested that perhaps other habitat variables or some as-yet unrecognized mechanism may be effective (Wu et al. 2016a, b).

5 Conclusion

In this study, we integrated effect of distance dependence, conspecific negative density dependence (CNDD), phylogenetic density dependence (PNDD/PPDD), and habitat filtering in temperate tree seedling survival. Some results were found. (1) The strongest CNDD was found within a radius of 15 m. (2) Seedling survival were most strongly impacted by the density of conspecific seedling- and adult- neighbors in habitats with relatively low soil moisture and less in higher moisture habitats. (3) The effect of seedling-seedling CNDD was especially significant, when densities were ranged from 20 to 40 seedlings/4 m2. (4) There was some evidence of phylogenetic positive density dependence (PPDD), and the effect of seedling-seedling PPDD was increasing with an increase in soil moisture. Our results demonstrate that conspecific negative density dependence played an important role in seedling survival, which is closely related to habitat filtering and population density. However, we found some evidences of phylogenetic positive density dependence. We suggest that future studies of neighborhood density dependence should increase awareness of evolutionary relationships.

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.