Many wildlife-monitoring programmes have long time series of species abundance that cannot be summarized adequately by linear trend lines. To describe long time series better, generalized additive models may be used to obtain a smooth trend line through abundance data. We describe another approach to estimate a smoothed trend line through time series consisting of one observation per time point, such as year or month. This method is based on structural time-series models in combination with the Kalman filter and is computerized in the TrendSpotter software. One of its strengths is the possibility to test changes in smoothed abundances between years, taking into account serial correlation. The trend method is applied in the Dutch Waterbird Monitoring Scheme (DWMS), a monitoring scheme for migrating and overwintering waterbirds. Taking the numbers of overwintering Greater Scaup (Aythia marila) in the Netherlands as an example, we demonstrate three applications of the method: (1) trend calculation and classification for each year in the time series, (2) assessing alerts for alarming population declines and (3) testing yearly abundance against a population threshold. We discuss the situations where TrendSpotter is to be preferred over other methods.
Long time series of wildlife data are becoming more widespread now that many countries have well-established monitoring programmes. In the Netherlands nature monitoring has been strongly intensified and standardized in the past decade of the twentieth century. This development was mainly driven by the wish to base nature policy on sound knowledge of trends in animal and plant numbers and by international regulations like the European Unions’ Habitats and Birds Directive. The Dutch Network for Ecological Monitoring (NEM; van Strien 2006) now contains 14 monitoring schemes for different species groups, ranging form butterflies and plants to birds. Most time series cover periods of 10–15 years, but time series of bird species span more than 20 years. The longest time series, since 1975, come from the Dutch Waterbird Monitoring Scheme.
In most NEM monitoring schemes data analysis is performed using log-linear Poisson regression, a form of generalized linear models (GLMs) (McCullagh and Nelder 1989; ter Braak et al. 1994). The programme used is TRends and Indices for Monitoring data (TRIM) (Pannekoek and van Strien 2001), which calculates yearly indices per species and also provides an overall linear trend estimate that is based on the yearly indices and is meant to describe the change over the entire study period. However, long time series may show alternating periods of increase and decrease, and linear trends do not summarize these time series adequately. Furthermore, to separate patterns of genuine change from annual fluctuations, it is helpful to apply a smoothing algorithm. Although TRIM is able to assess trends in parts of the time series by using change points, smoothing would require other techniques such as Loess estimators or generalized additive models (GAMs).
GAMs are among the most widely used methods to smooth time series. The smoothed trend line of a GAM will usually look much like a moving average, but, in addition, it provides information on the uncertainty of the trend by calculating confidence limits. GAMs are being used for the analysis of bird monitoring data, for instance in the Breeding Bird Scheme and the Wetland Bird Survey in the UK (Fewster et al. 2000; Atkinson et al. 2006). However, in some situations, GAMs are not satisfactory to apply. In such cases, an alternative smoothing method from the class of structural time-series analysis, in combination with the Kalman filter, may be helpful (Visser 2004, 2005, 2007). In this paper we demonstrate the main characteristics of this method and some applications, using the TrendSpotter software package. We first give a brief introduction to TrendSpotter and compare characteristics of three methods of analysis (TRIM, GAM and TrendSpotter). We then focus on the application of TrendSpotter to data of overwintering Greater Scaup (Aythya marila) in The Netherlands.
Materials and methods
Data and data processing
In the Dutch Waterbird Monitoring Scheme (DWMS) waterbirds are counted on all important water bodies in the Netherlands. These numbers concern mainly migrating and overwintering birds. The waterbird counts are performed in 8–12 months per season, depending on the region (Soldaat et al. 2004; van Roomen et al. 2006a, b).
Although there are many missing values present in the data because not all water bodies are counted each month, the scheme comes close to a total census for many species. For these species a large part of the total population is counted, and the sum of all birds at individual sites is a good approximation of the number of birds in the whole country. For the Greater Scaup, for instance, yearly, more than 90% of all the birds present in the country have probably been counted in the DWMS since 1990.
To calculate yearly total numbers of Greater Scaup and any other species in the DWMS, missing values in the dataset were first imputed in two steps:
First, missing monthly counts were imputed with UINDEX (Bell 1995; Underhill and Prŷs-Jones 1994) at the level of so-called monitoring sites, which include all important wetlands. We did not use TRIM here, because TRIM cannot cope with monthly counts. Before imputation, the sites were grouped into 11 regional strata in which a more or less comparable population development was expected.
After summation of the imputed monthly counts to seasonal sums per monitoring site, the seasonal sums with > 90% imputed birds were assigned a missing value, in order to exclude extremely high imputed values that are occasionally generated in the first step. These missing values were imputed again within a database that was not stratified in regional strata.
After imputation, the seasonal sums of all sites were aggregated to a country yearly total. In the standard trend analysis procedure of the DWMS, yearly totals consisting of > 90% imputed birds are assigned a missing value, but, for the Greater Scaup, all yearly totals had < 90% imputation. The yearly totals were divided by 12 to give mean monthly numbers, which give better understandable bird numbers than seasonal sums or so-called bird-days. The last step was to apply TrendSpotter to these mean values.
Characteristics of the methods of analysis
TrendSpotter estimates smoothed population numbers μ 1, μ 2, . . . , μ t , . . . , μ N for a time series with N equidistant measurements over time (y 1, y 2, . . . , y t , . . . , y N ). The expectations of these smoothed population numbers are denoted as m 1, m 2, . . . , m t , . . . , m N . TrendSpotter also estimates the standard deviations of the smoothed population numbers: SD1, SD2, . . . , SD t , . . . , SD N . Finally, it estimates the standard deviations of the differences μ N −μ t , denoted here as SDN−t. The estimation of confidence intervals is based on the deviations of time point values from the smoothed line. This emphasizes temporal variation as the main source of error, although measurement errors and errors due to imputation are implicitly incorporated in the yearly measurements. Missing data are easily incorporated into the model. In the present version of TrendSpotter it is not possible to add information on differences in reliability of individual measurements. In TrendSpotter the flexibility of a trend is set by the value of a standard deviation of a noise process. This standard deviation (parameter) can be chosen by hand, or can be optimized by maximum likelihood optimization (the default choice). If the parameter is set to zero, the estimated trend equals the well-known regression trend line (straight line). If the parameter is set to a very large number, the trend will go through all measurements y 1, y 2, . . . , y t , . . . , y N . All values for m I , SD I , and SDN−t are estimated by use of the Kalman filter. This filter is attractive, because it gives estimates with optimal statistical properties. Although normally distributed residuals (in jargon innovations) are not a necessary condition for the proper use of the Kalman filter, it still is a desirable property. For example, all confidence limits assessed by ±1.96 × standard error can be interpreted as 95% confidence limits. Normality of residuals can be tested by a so-called normality plot. For mathematical details about structural time-series analysis and the Kalman filter please refer to Harvey (1989).
Table 1 lists characteristics of three methods of analysis. Both TRIM and GAMs can be used to estimate missing counts in the raw data. TrendSpotter, however, is not able to impute missing values in the raw data. Thus, for the DWMS, TrendSpotter can only be used to smooth indices obtained by other methods, e.g. TRIM or UINDEX (Underhill and Prŷs-Jones 1994; Bell 1995). In contrast, one of the strengths of a GAM is its ability to smooth, using the raw data (Fewster et al. 2000), which makes it a promising method, although it is very computer-intensive. Both TRIM and TrendSpotter apply analytical methods to calculate confidence intervals. GAMs typically compute confidence intervals using bootstrapping, because of difficulties in applying an analytical method (Fewster et al. 2000). However, bootstrapping of sites is not adequate in the case of a census, such as the DWMS. Bootstrapping of sites emphasizes the between-site variation, i.e. the variation in changes between sites, as the main source of uncertainty, whereas this is not relevant for a census. Bootstrapping might even lead to unrealistic high upper limits of confidence intervals when, e.g., the Wadden Sea or other areas with exceptionally high numbers of particular species are selected several times per bootstrap sample. Underhill and Prŷs-Jones (1994) have already acknowledged that there are logical difficulties with incorporating between-site variation in the case of a census. In such a situation, we prefer to apply TrendSpotter in combination with an imputing method.
All three trend methods discussed can be used to identify change points (Table 1), but the methods differ in their ability to test differences between years. In TRIM, each yearly index can be statistically tested against the base year, taking into account serial correlation. TrendSpotter is also able to test indices between years. More particularly, TrendSpotter tests the difference between the smoothed population number of the last year against each of the preceding years, thereby taking into account serial correlation between years. GAM indices may also be compared between years, but a statistical test is limited because serial correlation cannot be taken into account. Substantial positive serial correlation may be present in bird monitoring data, because partly the same individual birds are counted each year at the same sites. If such serial correlation is not taken into account, confidence intervals may be underestimated, leading to more type I errors.
Trend calculation and classification
The mean monthly numbers of overwintering Greater Scaup in the Netherlands nowadays are slightly lower than in the 1970s and early 1980s, and they are much lower than in the second part of the 1980s and the early 1990s (Fig. 1). The figure makes clear that one overall linear trend estimate for this time series is not very informative, because of the relatively high bird numbers in the middle part of the time series. Instead, the smoothed line calculated by TrendSpotter gives a much better description of the population trajectory. TrendSpotter enables us to calculate trend estimates based on the smoothed curve. We distinguish between (1) the total change rates (TCRs), i.e. the smoothed population number in the last year compared with the smoothed population number in each year in the time series and (2) the mean yearly change rate (YCR), i.e. the TCR expressed as a mean change rate per year. The YCR corresponds with the multiplicative yearly slope in TRIM.
If TrendSpotter is applied to untransformed data, the calculations proceed by first approximating the total change rate as:
We note that the expectation of μ N /μ t could also be calculated by Monte Carlo simulation. However, this option is not incorporated into TrendSpotter.
For log-transformed data the TCR can be approximated by:
For both untransformed and log-transformed data, the YCR is calculated as:
The smoothed curve values for Greater Scaup numbers indicate a decline from 1978 to 2004 (Fig. 1). The corresponding log-transformed values for m N and m t are 9.805 and 10.296, respectively (Table 2). The analysis was performed on log-transformed data, so substituting these figures in Eq. 1b gives a total change rate of:
and substituting this TCR in Eq. 2 gives a yearly change rate of
In other words, the population underwent an almost 40% decline [100 × (1.00−0.612) = 38.8], between 1978 and 2004, and a mean yearly decline between 1978 and 2004 of 1.9% [100 × (1.00−0.981) = 1.9].
The next step is to take the confidence limits into account in the trend calculation. By using the approximation for μ N /μ t given in Eq. 1a, we find the 95% confidence limits of the TCR and YCR
if TrendSpotter is run on untransformed data, or
in the case of log-transformed data. For both untransformed and log-transformed data, the confidence interval of the YCR can be calculated by substituting CITCR in Eq. 4 by the lower and upper confidence limits from Eq. 3a or b
YCR estimates and confidence intervals can be used to classify the trends per year. We applied the trend classification scheme given in Table 3. The results for Greater Scaup in The Netherlands are shown in Table 4 (fourth column), indicating that the species has undergone a significant moderate decline since 1979–1980 and 1988–1995.
An extension of trend calculation is the assessment of population alerts. A population alert highlights any declines in bird numbers that are of conservation concern (de Nobel et al. 2002). The British Trust for Ornithology (BTO) alert system (http://www.bto.org/birdtrends2005/alerts.htm) uses strong declines (>50%) and moderate declines (>25%) as alert thresholds, over different time periods (the whole time series and the last 25, 10 and 5 years of the time series). The BTO system uses GAMs to calculate smoothed indices of animal abundance and takes into account the standard errors of the indices to test for significance of the alerts.
TrendSpotter can also be used to calculate alerts through the conversion of the TCR values. The population of Greater Scaup has a TCR of 0.375 since 1990 (see Table 4, fifth column), which is similar to a 62.5% decline [100 × (1.00−0.375) = 62.5]. This decline is significantly larger than 25%, as the maximal TCR gives an estimated 39.3% decline for 1990–2004 [100 × (1.00−0.607) = 39.3], but not significantly larger than 50%. If an alert threshold of 25% decline is used, a moderate alert for the Greater Scaup is generated for the decline since 1990 (Table 4). Alert-thresholds may be any arbitrary value, e.g. >10%, >25% or >50% decline, and may also be based on a confidence interval of 90% instead of 95%.
Testing against a population threshold
Another application of TrendSpotter is the possibility to test if a smoothed index is above or below a certain population threshold. If the standard deviations SD t that are estimated by TrendSpotter are used, the smoothed population number for each year in the time series can be compared statistically with some predefined aim, for instance the favourable conservation status in the framework of the EU Birds Directive. For the Greater Scaup this conservation status was set at a monthly mean of at least 25,000 birds (LNV 2006). The smoothed population numbers for 1988–1995 were significantly above this value, as can be concluded from a comparison of the lower confidence limit in Fig. 1 with the solid grey line. From 1999 until 2003, however, the upper confidence limit was below the favourable conservation status, indicating a deviation from the threshold. In 2004 no significant difference from the favourable conservation status was found, partly due to a slight increase in the numbers of Greater Scaup counted, but also due to the larger confidence intervals at the end of the time series. The widening of confidence intervals at the start and end of time series is inherent to smoothing techniques.
Time series analysis with TrendSpotter has proved useful in the Dutch Waterbird Monitoring Scheme. Time series of up to 30 years, with alternating periods of increases and declines, were adequately described by a smoothed line. Changes in population abundance since any year could be statistically tested and trends could be classified for each individual year and converted into population alerts, as well as tested against population thresholds. GAMs offer similar advantages, but, as said earlier, do not take serial correlation into account with respect to trend classification and population alerts.
TrendSpotter may also be useful in other monitoring schemes, although there are some restrictions to the minimum length of the time series and the maximum percentage of missing yearly values at the start of the time series. Furthermore, the data should not contain too many zero values or values close to zero. Its main limitation, however, is its inability to take into account the uncertainties of the imputing models applied to the raw data, and, in that respect, a GAM applied to raw data is to be preferred. Currently, TrendSpotter is best applied in situations where GAMs to smooth the time series would be less satisfactory. This is not only in the case of a census; TrendSpotter may also be helpful in assessing confidence limits of composite indices for species groups, a type of biodiversity indicator that is becoming increasingly popular (see, e.g. Gregory et al. 2005). TrendSpotter is also the method of choice to smooth time series that contain only one value per time point, e.g. the first laying date of Lapwing in the Netherlands. Many applications of TrendSpotter to composite indices and to single value per year time series are found in http://www.natuurcompendium.nl.
A future development of TrendSpotter will be the inclusion of weights for each year in the time series, which will enable one to incorporate confidence intervals of the indices computed by TRIM or other imputing models. Such a combination of TRIM and TrendSpotter would offer many of the advantages of a GAM based on raw data and is easier to apply than a GAM.
Unfortunately, even such a combination may not completely solve the conceptual problems in the calculation of confidence intervals of trends in waterbirds. For a census, currently, there does not seem to be an analysis method available to take into account imputation uncertainties without, at the same time, including between-site variation. We have chosen to apply TrendSpotter for the DWMS because we prefer to neglect the imputation uncertainties rather than to include them and unavoidably incorporate between-site variation as obtained from bootstrapping using a GAM. This preference is based on the assumption that imputation uncertainties in the DWMS are probably small, as the number of missing counts is limited and extremely high imputed values are excluded from the trend analysis.
Two additional features of TrendSpotter should be noted. First, the programme may be run on time series with cyclic patterns, as are usually present in monthly counts of waterbirds. This could be interesting in the analysis of shifts in the seasonal patterns of birds as an effect of, e.g., climate change. Second, similar to applying covariate models in a GAM, explanatory variables may be added to each record in the dataset, which enables the detection of causal factors for observed changes in population abundance.
Atkinson PW, Austin GE, Rehfisch MM, Baker H, Cranswick P, Kershaw M, Robinson J, Langston RHW, Stroud DA, van Turnhout C, Maclean MD (2006) Identifying declines in waterbirds: the effects of missing data, population variability and count period on the interpretation of long-term survey data. Biol Conserv 130:549–559
Bell MC (1995) UINDEX4. A computer program for estimating population index numbers by the Underhill method. The Wildfowl & Wetland Trust, Slimbridge
Braak CJF ter, van Strien A, Meijer R, Verstrael TJ (1994) Analysis of monitoring data with many missing values: which method? In: Hagemeijer W, Verstrael TJ (eds) Bird numbers 1992. Distribution, monitoring and ecological aspects. Proceedings of the 12th International Conference of IBCC and EOAC, Noordwijkerhout, The Netherlands. Statistics Netherlands, Voorburg/Heerlen & SOVON, Beek-Ubbergen, pp 663–673
Fewster RM, Buckland ST, Siriwardena GM, Baillie SR, Wilson JD (2000) Analysis of population trends for farmland birds using generalized additive models. Ecology 81:1970–1984
Gregory RD, van Strien AJ, Vorisek P, Gmelig Meyling AW, Noble DG, Foppen RPB, Gibbons DW (2005) Developing indicators for European birds. Philos Trans R Soc B 360:269–288
Harvey AC (1989) Forecasting structural time series models and the Kalman filter. Cambridge University Press, London
LNV (2006) Natura 2000 doelendocument. Dutch Ministry of Agriculture, Nature and Food Quality, Den Haag
McCullagh P, Nelder JA (1989) Generalised linear models. 2nd edn. Chapman & Hall, London
Nobel P de, van Turnhout C, van der Winden J, Foppen R (2002) An Alert System for bird population changes on a national level and for EU Bird Directive monitoring: a Dutch approach. SOVON-onderzoeksrapport. SOVON Vogelonderzoek, Nederland, Beek-Ubbergen
Pannekoek J, van Strien A (2001) TRIM (TRends and Indices for Monitoring data). Research paper no. 1020, Centraal Bureau voor de Statistiek (CBS), Voorburg
van Roomen M, Koffijberg K, Noordhuis R, Soldaat L (2006a) Long-term waterbird monitoring in The Netherlands: a tool for policy and management. In: Boere GC, Galbraith CA, Stroud DA (eds) Waterbirds around the world. The Stationary Office, Edinburgh, pp 463–467
van Roomen M, van Winden E, Koffijber K, Ens B, Hustings F, Kleefstra R, schoppers J, van Turnhout C, SOVON Ganzen- en zwanenwerkgroep, Soldaat L (2006b) Watervogels in Nederland in 2004/2005. SOVON-monitoringrapport 2006/02, RIZA-rapport BM06.14, SOVON Vogelonderzoek Nederland, Beek-Ubbergen
Soldaat L, van Winden E, van Turnhout C, Berrevoets C, van Roomen M, van Strien A (2004) De berekening van indexen en trends bij het watervogelmeetnet. SOVON-onderzoeksrapport 2004/02. Centraal Bureau voor de Statistiek, Voorburg/Heerlen
van Strien A (2006) Landelijke Natuurmeetnetten van het NEM in 2005. Kwaliteitsrapportage NEM. Centraal Bureau voor de Statistiek, Voorburg/Heerlen
Underhill LG, Prŷs-Jones RP (1994) Index numbers for waterbird populations. (I) review and methodology. J Appl Ecol 31:463–480
Visser H (2004) Estimation and detection of flexible trends. Atmos Environ 38:4135–4145
Visser H (2005) The significance of climate change in the Netherlands. An analysis of historical and future trends (1901–2020). MNP report 550002007. http://www.mnp.nl/en/publications/2005/The_significance_of_climate_change_in_the_Netherlands.html
Visser H, Petersen AC (2007) Outdoor skating and climate change. Climatic Change (in press)
This publication would not have been possible without the data collected by many volunteer bird counters, and the financial support of the Dutch Ministry of Agriculture, Nature and Food quality, the Ministry of Transport, Public Works and Water Management and Vogelbescherming Nederland. Thanks are also due to A. Gmelig Meyling for computerizing the calculations described in the paper and to J. Blew and an anonymous reviewer for helpful comments on a previous draft of the paper.
Communicated by F. Bairlein.
This is a paper on the occasion of a presentation at the “population alerts” symposium at the 24th IOC Congress in Hamburg.
TrendSpotter is available from the second author (http://firstname.lastname@example.org) at no cost.
About this article
Cite this article
Soldaat, L., Visser, H., van Roomen, M. et al. Smoothing and trend detection in waterbird monitoring data using structural time-series analysis and the Kalman filter. J Ornithol 148 (Suppl 2), 351–357 (2007). https://doi.org/10.1007/s10336-007-0176-7
- Kalman filter
- Structural time-series analysis
- Population alert
- Trend analysis
- Generalized additive model (GAM)
- TRends and Indices for Monitoring data (TRIM)
- Aythia marila
- Greater scaup