What to Use and When

Abstract

Given our discussion of the various methods above, a natural question is: What is the (best) method to use? We phrase this question with “best” in parentheses because we do not believe that there is an absolutely “best sensitivity method”. In fact, even the answer to the simpler question “of which method should be used” is multifaceted.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Given our discussion of the various methods above, a natural question is: What is the (best) method to use? We phrase this question with “best” in parentheses because we do not believe that there is an absolutely “best sensitivity method”. In fact, even the answer to the simpler question “of which method should be used” is multifaceted.

The first and most important factor is the setup of the sensitivity analysis . We need to establish the managerial question. This is followed by a decision to adopt a certain method, which requires awareness of the fact that each method has limitations.

For example, we might be interested in running a model through different scenarios in order to appreciate how a profit forecast varies given different combinations of model inputs . If a model runs sufficiently fast, why not aim for a full decomposition of the scenario results, as described in the previous sections? However, the model runs maybe computationally expensive. It may be that only a series of one-at-a-time sensitivities (i.e., $n+1$ model evaluations) can be achieved, which would allow us to form a tornado diagram in which the sensitivity measures are first-order finite change sensitivity indices . If we can afford $2n+2$ calculations, then we are in a position to derive a generalized tornado diagram, which enables us to appreciate individual effects as well as the relevance of interactions . As such, we obtain a more complete explanation of the change than with a simple one at a time sensitivity. Alternatively, we might be interested in inspecting the behavior of our model as a function of each model input, as the individual model inputs vary over the predetermined ranges. In this case, the most relevant method is one-way sensitivity functions. Alternatively, we might be interested in small perturbations about a reference point. In that case, differential sensitivity methods should be used.

Notably, however, we must recall each method’s limitations before drawing any conclusions. We can launch a little horse race among the proposed techniques. Consider the two mappings in Eqs. (3.17) and (3.19). Let us start with the mapping in Eq. (3.17). Tornado diagrams and one-way sensitivity functions both fail to provide evidence of the dependence of g on $x_{1}$ and $x_{2}$. (Recall that the base case is $\mathbf {x}^{0}=(\dfrac{1}{2},\dfrac{1}{2},\dfrac{1}{2}))$. Perhaps we can reach a better answer by differentiating the model. We find:

$$\begin{aligned} \begin{array}{ccc} \dfrac{\partial y(\mathbf {x}^{0})}{\partial x_{1}}=0&\dfrac{\partial y( \mathbf {x}^{0})}{\partial x_{2}}=0&\dfrac{\partial y(\mathbf {x}^{0})}{ \partial x_{3}}=0, \end{array} \end{aligned}$$

(10.1)

because $\mathbf {x}^{0}$ is a critical point for the mapping in Eq. (3.17). This suggests that differentiation also works poorly with this model. Let us instead try finite change sensitivity indices. Given the same scenarios found in the tornado diagram in Fig. 8.1, we derive the generalized tornado diagram in Fig. 10.1.

Fig. 10.1

Generalized tornado diagram for the model in Eq. (3.17)

The diagram in Fig. 10.1 reveals $x_{1}$ and $x_{2}$ as active model inputs . Therefore, the generalized tornado diagram is more informative in this case than partial derivatives or a traditional tornado diagram.

Let us also perform a sensitivity analysis on the model in Eq. (3.19). We recall that a tornado diagram (as well as a generalized tornado diagram) would only register null sensitivity measures, while one-way sensitivity functions would only reveal an active role for $x_{3}$. The same would be true if we used generalized tornado diagrams . By using differentiation, we arrive at (recall that $\mathbf {x}^{0}=[ \begin{array}{ccc} 0&0&0 \end{array} ]$):

$$\begin{aligned} \begin{array}{ccc} \dfrac{\partial y(\mathbf {x}^{0})}{\partial x_{1}}=0&\dfrac{\partial y( \mathbf {x}^{0})}{\partial x_{2}}=0&\dfrac{\partial y(\mathbf {x}^{0})}{ \partial x_{3}}=-2\pi . \end{array} \end{aligned}$$

(10.2)

As such, differentiation produces the same indication of a one-way sensitivity function and signals that the only relevant variable is $x_{3}$.

These results demonstrate the drawbacks of performing a local sensitivity analysis , regardless of the method. The reason is that we are exploring the model-input space around only a single point of interest or a very limited number of points.

What can we do to improve the situation? We can, for example, more thoroughly explore the model-input space. However, doing so requires that we abandon deterministic local methods and turn to global methods. Before we describe such methods, we wish to explore the concept of “value of information” as a bridge between local and global sensitivity analyses, and the problem of local sensitivity analysis with input constraints.