Introduction: Sensitivity Analysis – What and Why?

Abstract

Demography is a science that connects individual processes and events to the development of cohorts and then to the dynamics of populations.

1 Introduction

Demography is a science that connects individual processes and events to the development of cohorts and then to the dynamics of populations. It does so with mathematical models that distinguish among individuals based on their characteristics.¹ The most familiar such model is the life table, which records mortality and fertility of the individual as a function of age, and is used to calculate properties of cohorts (e.g., the distribution of age at death) and populations (e.g., the intrinsic rate of increase).

The life table is the most familiar, but demography has proceeded far beyond that in both models and analyses. In any case, though, a model is defined first by its structure (the states of individuals and the transitions possible among them), then by the rates at which individuals develop, survive, and reproduce throughout the life cycle, then by the functional dependence of those rates (time-invariant or time-varying, density-independent or density-dependent, deterministic or stochastic), and finally by the values of the parameters that define the rates. A set of parameters operating within a given model generates the demographic outcomes calculated from the model (population growth rate, population structure, equilibria, cycles, measures of longevity, state occupancy times, transient behavior and projections, and so on). The sensitivity problem is to understand how the outcome[s] change in response to changes in the parameters.

Why should we care about the effects of change?

• We may be concerned with particular changes that we have reason to believe will occur (due to, e.g., changes in society, changes in the environment, changes in policy) and want to know how they will affect the outcome of interest.

• We may want to evaluate the effect of changes that we hope to cause as matters of policy, or to compare alternative policies for their effects.

• We may be interested in evolutionary demographic questions. Natural selection is a process that explores the consequences for fitness (which is itself a demographic outcome) of changing the phenotypic traits that influence demographic parameters.

• We may want to identify the parameters have the biggest effect on the outcome, in order to allocate sampling or measurement efforts where they are most needed.

• At a very basic level, we may simply want to know how the system works, how the outcomes are determined. Just as an empirical study might include experiments to manipulate factors and see how outcomes change, sensitivity analysis of a mathematical model reveals how outcomes respond to parameter changes.

It is not an overstatement to say that no model is every fully understood if it does not include a sensitivity analysis.

2 Sensitivity, Calculus, and Matrix Calculus

The change in an outcome in response to a change in a parameter can be treated as a problem in differential calculus. Let ξ denote some dependent variable and θ some parameter. The sensitivity problem can be approached via the derivative

$$\displaystyle \begin{aligned} {d \xi \over d \theta} {} \end{aligned} $$

(1.1)

or the elasticity, or proportional sensitivity²

$$\displaystyle \begin{aligned} {\epsilon \xi \over \epsilon \theta} = \frac{\theta}{\xi} {d \xi \over d \theta} = {d \log \xi \over d \log \theta} \end{aligned} $$

(1.2)

Note that I will use “sensitivity analysis” to refer generically to both sensitivity and elasticity.

The sensitivity problem is a challenging task, rather than an exercise in undergraduate calculus, because the dependence of ξ on θ may be complicated, and because ξ may be a scalar (e.g. life expectancy at birth, or population growth rate) or a vector (e.g., a stable stage distribution or a projected population structure) or a matrix (e.g., the matrix of mean occupancy times). Similarly, θ may be a scalar (e.g., the Gompertz rate of aging) or a vector (e.g., the age schedule of mortality rates), or a matrix (e.g., the transition matrix among life cycle stages). In addition, the chains of causation in even simple demographic models are complicated. Tracing the causal chains from a set of parameters (of which there may be many) to a set of outcomes (again, many) with complicated interactions is hard.

This book is an in depth exploration of sensitivity analyses based on matrix formulations of demographic calculations. Matrix formulations are designed precisely to map transformations from one multidimensional space to another. Thus they simplify computations, clarify notation, and increase analytical power.³

The premise of this book is that demography as a discipline is neither defined by, nor limited to, a taxon. You will find here examples and analyses of humans, of non-human animals, and plants. Human demography and population biology have mutually informed each other from the beginning, and I see no reason for them to stop now.

It is important to remember that the diversity of complex life histories among the species that occupy our world poses a challenge to demographic analysis that is identical to the challenge posed by the complicated lives of humans. The dynamics of health status, family structure, or socio-economic status introduce complications to the life course exactly comparable to the dynamics of size growth in plants, metamorphosis in insects, or breeding status in birds.

A bit of history

The earliest focus of demographic sensitivity analysis was population growth rate λ (or the intrinsic rate of increase $r=\log \lambda$) in linear demographic models. Hamilton (1966) was the first to solve this, in the context of the evolution of senescence. Demetrius (1969) derived a corresponding matrix expression, apparently unaware of Hamilton’s results. Goodman (1971) was the first to notice the connection to reproductive value (see Chap. 3). Keyfitz (1971) derived the sensitivity of r, but also of life expectancy, mean age at death,and other outcomes.

All these analyses were based on age-classified demographic models. These results were generalized to stage-classified models by applying eigenvalue perturbation theory (Caswell 1978), followed by elasticity calculations (de Kroon et al. 1986), sensitivities of eigenvectors (Caswell 1982), lower-level parameters (Caswell 1989b), second derivatives of eigenvalues (Caswell 1996), the population spreading rate (Neubert and Caswell 2000), transient dynamics (Caswell 2007) and other things. Following the important early work of Tuljapurkar (1990), the sensitivity analysis of stochastic models developed in parallel with that of deterministic models (e.g. Tuljapurkar et al. 2003; Haridas and Tuljapurkar 2005; Horvitz et al. 2005; Steinsaltz et al. 2011).

Matrix calculus, permitting differentiation of scalar-, vector-, or matrix-valued functions of scalar, vector, or matrix arguments, began to be developed in the 1960s (see Nel (1980) for some history and comparison of different methods). The approach we will use here was introduced by Neudecker (1969) and expanded by Magnus and Neudecker (1985). A comprehensive, but mathematically difficult, treatment is given in Magnus and Neudecker (1988). Chapter 2 gives a brief presentation of the matrix calculus methods we will utilize in this book.

3 Some Issues

Sensitivity analysis is more than an algebraic exercise; it is a tool for making inferences and drawing conclusions about substantive demographic issues. It is useful to bring to the discussion a perspective on some questions.

3.1 Prospective and Retrospective Analyses: Sensitivity and Decomposition

If some variable ξ is a function of a set of parameters θ ₁, …, θ _p, then ${\partial \boldsymbol {\xi } \over \partial \theta _i}$ gives the rate of change of ξ in response to a change in the ith parameter, holding the rest constant. Contrary to what is sometimes assumed, this calculation requires no assumption that it is actually possible to change the parameters. If the flight velocity of pigs is one of the parameters in the model, the analysis will happily answer the question of what would happen if pigs could fly.

Nor is there any assumption that changes in θ _i have ever happened in the past. The sensitivity analysis looks forward, asking what would happen if this or that parameter were to change. It is thus referred to as prospective analysis (Caswell 2000).

On the other hand, suppose you find yourself considering two values of ξ, that have resulted from two different situations (times, places, conditions), each with its own set of parameters:

$$\displaystyle \begin{aligned} \begin{array}{rcl} \theta_1^{(1)}, \theta_2^{(1)}, \ldots &\displaystyle \longrightarrow&\displaystyle \xi^{(1)} \\ \theta_1^{(2)}, \theta_2^{(2)}, \ldots &\displaystyle \longrightarrow&\displaystyle \xi^{(2)} \end{array} \end{aligned} $$

You ask, what caused the difference between ξ ⁽²⁾ and ξ ⁽¹⁾. Knowing the derivatives ${\partial \xi \over \partial \theta _i}$ cannot tell you, because you are not asking the counterfactual question of what would happen if, but the very factual question of what actually happened between the two situations. This is a retrospective analysis, familiar to human demographers as a decomposition problem (e.g., Kitagawa 1955; Canudas Romo 2003).

One widely used approach to understanding the causes of observed differences is life table response experiment (LTRE) analysis,⁴ which uses a first-order approximation to decompose the differences,

$$\displaystyle \begin{aligned} \Delta \xi = \xi^{(2)} - \xi^{(1)} \approx \sum_i {\partial \xi \over \partial \theta_i} \left( \theta_i^{(2)} - \theta_i^{(1)} \right) . {} \end{aligned} $$

(1.3)

The ith term in the summation is the contribution of the difference in the parameter θ _i to the difference in the outcome, Δξ. These contributions reflect both the sensitivity of ξ to the parameters and the differences between conditions in each of the parameters. Parameters to which ξ is not very sensitive can make large contributions if the difference Δθ _i is big enough. Contributions to which ξ is very sensitive can make small contributions if θ _i does not change much. The matrix calculus version of this decomposition is given in Sect. 2.9, applied to differences in life disparity in Chap. 4, to periodic environments in Chap. 8, and explored in the challenging context of stochastic models in Chap. 9.

The distinction between prospective and retrospective analysis is obvious once the questions they address are specified, but it has challenged a number of authors (e.g., Wisdom and Mills 1997; Manlik et al. 2017). A particularly insightful discussion of these ideas, in somewhat different terminology, appears in Nathan Keyfitz’s essay, How do we know the facts of demography?, which now appears as Chapter 20 of Keyfitz and Caswell (2005).

3.2 Uncertainty Propagation

Suppose that ξ is a function of θ, but θ is known only imperfectly. Then ξ is also known only imperfectly; the uncertainty in θ is propagated from θ to ξ. The sensitivity dξ∕dθ alone says nothing about uncertainty, and the uncertainty in ξ says nothing about the sensitivity.

Uncertainty propagation can be calculated by simulation if a probability distribution is known that can describe the uncertainty in θ. Sampling from this distribution and calculating ξ for each sampled parameter gives the distribution of ξ resulting from the uncertainty in θ (e.g. Caswell et al. 1998; Salomon et al. 2001). If the distribution of θ comes from an empirical set of measurements, this approach converges to the bootstrap (Efron and Tibshirani 1993). If θ has a parametric distribution (e.g., the multivariate normal distribution returned by maximum likelihood estimation) the technique is sometimes known as a parametric bootstrap (e.g., Regehr et al. 2010).

Sensitivity analysis can contribute to uncertainty propagation analysis through the first order, small variance approximation to the variance in ξ,

$$\displaystyle \begin{aligned} V(\xi) \approx \sum_{i,j} \left( {\partial \xi \over \partial \theta_i} \right) \left( {\partial \xi \over \partial \theta_j} \right) \mbox{Cov}(\theta_i,\theta_j) \end{aligned} $$

(1.4)

Notice again that sensitivity does not, by itself, say anything about uncertainty, but it does show how the (co)variance in parameters will propagate to the variance in the outcome ξ.

3.3 Why Not Just Simulate?

If you work on these problems, or if you apply these methods in particular studies, eventually you will be asked (often by a reviewer), why not just do it all by simulation?Just evaluate ξ at the value θ, and at θ +  Δθ, and then approximate the derivative as

$$\displaystyle \begin{aligned} \frac{\Delta \xi}{\Delta \theta} = \frac{\xi(\theta+\Delta \theta) - \xi(\theta)}{\Delta \theta} \end{aligned} $$

(1.5)

for some very small value of Δθ.

Three answers come to mind. First, if θ and the model are of sufficiently high dimension, there can be a lot of these perturbations to be calculated. For example, population projections of the type analyzed by Caswell and Sanchez Gassen (2015), with 102 ages, 2 sexes, 3 vital rates, and projections on the order of 50 years, have over 30,000 parameters. A numerical perturbation of each of these would be painful.

Second, the computation of derivatives by numerical perturbations is a notoriously ill-behaved problem. A standard reference on computations in applied mathematics says that this approximation is “almost guaranteed to produce inaccurate results” (Press et al. 1992, p. 185). It is subject to truncation error (caused by making the perturbation too large) and roundoff error (caused by making the perturbation too small). In some applications these errors will be unimportant, but in others they can be crucial (e.g., Hunter and Caswell 2009, for an example in mark-recapture analysis)s.

Third, and more basic and telling: an exact answer is always an improvement over an approximation. When an exact answer is available, in an easily computable form, there must be strong arguments to support the idea that a less efficient and less accurate approximation is just as good. And having both exact and approximate methods is even better.

These arguments apply to numerical calculation of derivatives. But simulation has an important place in analyzing scenarios; i.e., the results of specified collections of parameters, usually with multiple and large differences among them. When population projections are reported with “high,” “medium,” and “low” fertility scenarios, the point is to compare a range of multivariate alternatives. Other examples include comparisons of screening procedures for colorectal cancer (Wu et al. 2006), or projections based on IPCC global climate models (e.g., Jenouvrier et al. 2012). In principle, sensitivity analysis could support these calculations by suggesting interesting scenarios, highlighting the parameters with the biggest impact on the outcome.

3.4 Sensitivity and Identifying Targets for Intervention

To intervene is to change something. Population biologists concerned with endangered species would like to intervene to increase the population growth rate. Those concerned with invasive pests would like to do the same, but in the opposite direction. Human demographers focused on aging societies wonder about how policies would change age distributions or dependency ratios. In all these cases, the interventions operate through changes in demographic parameters, and thus sensitivity analysis can reveal something about their effects.

This logic has led to the use of prospective perturbation analyses in conservation biology, using the sensitivity or elasticity of population growth rate to identify promising targets for intervention. The first such use involved the loggerhead sea turtle (Crouse et al. 1987). Standard practice at the time was to focus on protecting eggs and hatchlings on nesting beaches. But a sensitivity analysis showed that population growth rate was not very sensitive to these stages, and much more sensitive to changes in survival of adults at sea. This led to a recommendation, and then a policy, to install “turtle excluder devices” on the nets used in coastal shrimp fisheries in the United States, to reduce mortality due to adult turtles being captured in those nets.

This basic idea has become a part of the toolkit for conservation biology, but has also fallen victim to a kind of magical thinking that first makes unrealistic expectations of the sensitivity analysis and then blames the analysis for failing to meet those expectations. For a recent example see Manlik et al. (2017); for a thorough description of the issues and some of their solutions, see Caswell (2001, Chapter 18).

The fact remains that knowing the sensitivity of some outcome ξ to some parameter θ gives the rate of change of ξ in response to an intervention that changes θ. That is valuable information to have in considering the various interventions that might bring about a desired change.

3.5 The Dream of Easy Interpretation

This book is full of long and complicated formulas. Occasionally, these formulas yield easy, readily apparent, qualitative interpretations.⁵ But not often. There is a reason for this. The formulas are complicated because the processes are complicated, and because the results are given at a high level of generality. Chapter 10, for example, analyzes the sensitivity of nonlinear, density-dependent models. It derives a complicated formula for the sensitivity of any function of the equilibrium population, to changes in any parameter affecting any of the vital rates, in any age- or stage-specific way, for any choice of stage classification and any survival, fertility, and transition rates, with any pattern of density dependence, for any species with any kind of life history. Accounting for that web of dependencies, in such generality, makes finding an easily interpretable formula an unlikely dream.

Not an impossible dream, but in general, insights of that kind arise from simplifying general methods to address particular situations. Specifying a particular demographic structure, choosing an outcome variable of interest, and carefully specifying the functional dependencies, if done skillfully, can lead to qualitative results.

4 The Importance of Change

Questions of change lurk in almost every demographic (every scientific?) study. We ask how things have changed in the past, how they differ among populations in the present, and how they will, or may, change in the future. Even apparently simple descriptive statements (the results of a census in a particular time and place, for example) are almost immediately examined in comparison with other times and/or places.

Sensitivity analysis is a powerful tool for analyzing change, in the special case of demographic outcomes that are calculated as functions of some set of parameters. As the chapters to come will make clear, this covers a wide landscape of interesting demographic questions. And the list is not yet complete.