FormalPara Key Points for Decision Makers

In health economic models, uncertain ordered parameters are common. For example, utility and treatment costs associated with different severity levels of a disease are commonly ordered.

For ordered parameters, which are believed to be related such that the value of one variable is always greater than another, this information on ordering should always be appropriately incorporated within PSA.

The proposed sampling approach generates ordered parameters with both clinical and statistical validity for use in PSA.

1 Introduction

In health technology assessment, probabilistic sensitivity analysis (PSA) represents the generally accepted approach for characterising the uncertainty in parameters included in an economic model and for producing accurate results in non-linear models [1, 2]. This involves generating a large number of realisations of the economic model, each time sampling values from the distributions applied to each uncertain parameter included in the model.

Sometimes there is an absolute belief that the value of one variable is greater than the value of another. There may be uncertainty around the true mean values of the variables, but the ordering of the values is known. For example, if an individual rates his/her general health as ‘good’, then later as ‘fair’, we might be uncertain about how to map the ‘good’ and ‘fair’ health evaluations onto a numeric scale, but can assume the ‘good’ general health score will be higher than the ‘fair’ general health score. Health-related quality of life (HRQoL) for two different severity levels of a disease may also be related in this manner.

Failure to account for constraints between samples of ordered parameters may result in PSA values that do not accurately characterise the uncertainty present in a decision problem. In theory, this could result in decisions made on the allocation of scarce healthcare resources being suboptimal, although the direction of bias would depend on the specific model. Other outputs and analyses that are reliant on the PSA, such as cost-effectiveness acceptability curves (CEACs) and frontiers (CEAFs), and value of information analyses, are likely to also be inaccurate if the constraint that the value of one variable is greater than another is not accounted for appropriately.

Based on the authors’ experience, as National Institute for Health and Care Excellence (NICE) committee members, Evidence Review Group members dealing with a large number of NICE submissions every year, and peer reviewers, we have identified multiple examples where sampling of ordered parameters has not been handled correctly. Independent sampling from the distributions of the ordered parameters could result in a PSA that lacks clinical validity as in some realisations the logical constraint may be violated, with the sampled value of parameters potentially implying that having a disease makes people healthier. We refer to this method as the independent sampling approach. For example, in an NICE Single Technology Appraisal [3] the use of the independent sampling approach results in the utility in the progressed state being higher than in the stable state. Another sampling method used in practice used a common random number to draw from both distributions [4], which induces extreme dependence: one variable is a deterministic function of another. In addition to the extreme dependence induced by this method, the sampled realisations of PSA could still violate the known ordering if the cumulative density functions of the two parameters cross. We refer to this method as the common random number generator approach. A flawed alternative observed in papers sent to the authors for peer review is a method whereby samples are excluded when the ordering assumption is violated. We refer to this method as the modified independent sampling approach. This approach results in the summary statistics of the sampled realisations not equalling that of the source data, and provides biased sampled realisations.

Typical approaches described above lack either statistical or clinical validity. To have clinical validity when sampling parameters where it is known that one value is greater than another, all PSA realisations should exhibit the logical order of the given parameters. To have statistical validity, we have deemed that the following criteria should be met: (1) if the parameters of interest are bounded then all PSA realisations should be in these bounded ranges; (2) the summary statistics of the sampled values of the parameters of interest from PSA should match closely to the given summary statistics of the parameters of interest; (3) the induced correlation between the sampled values of parameters of interest should be plausible.

Introducing an additional parameter that presents either an absolute difference or a ratio of the ordered parameters is a common approach for modelling the ordered parameters explicitly. In Sect. 2, we apply this modelling idea in generating samples for PSA, and propose a new approach for sampling ordered parameters where summary statistics have been provided for the parameters of interest. We call this method the difference method (DM) approach. This approach is illustrated with an example sampling ordered parameters for utility and cost in Sect. 3, while Sect. 4 provides the results of comparing the performance of the DM approach with the typical approaches used in practice. Discussion is given in Sect. 5.

2 Methods

2.1 Unbounded Ordered Parameters

Suppose that there are two variables \(X\) and \(Y\), where the value of \(Y\) is greater than the value of \(X\), the distribution of \(X\) has mean \(\mu_{X}\) and variance \(\sigma_{X}^{2}\), and the distribution of \(Y\) has mean \(\mu_{Y}\) and variance \(\sigma_{Y}^{2}\). Let \({\Delta }\) denote the difference between \(X\) and \(Y\), and \(\mu\) and \(\sigma^{2}\) denote the mean and variance of the distribution of the difference \({\Delta }\), respectively. The DM approach samples from either \(X\) and \({\Delta }\) or \(Y\) and \({\Delta }\), depending on the magnitude of the variances of \(X\) and \(Y\). If \(\sigma_{Y}^{2} > \sigma_{X}^{2}\), then define

$$Y = X + {\Delta }$$
(1)

but if \(\sigma_{Y}^{2} < \sigma_{X}^{2}\), then define

$$X = Y - {\Delta }.$$
(2)

Assuming \(X\) and \({\Delta }\) in Eq. 1 are independent, and \(Y\) and \({\Delta }\) in Eq. 2 are independent, we get \(\mu_{Y} = \mu_{X} + \mu\) and \(\sigma_{Y}^{2} = \sigma_{X}^{2} + \sigma^{2}\) from Eq. 1, and \(\mu_{X} = \mu_{Y} - \mu\) and \(\sigma_{X}^{2} = \sigma_{Y}^{2} + \sigma^{2}\) from Eq. 2. Rearrange these equations, and we have \(\mu = \mu_{Y} - \mu_{X}\) and \(\sigma^{2} = \left| {\sigma_{Y}^{2} - \sigma_{X}^{2} } \right|\).

If both variable \(X\) and \(Y\) are unbounded, we propose using a Gamma(\(s\),\(r\)) distribution for \({\Delta }\) so that the difference is always positive, where \(s\) is the shape parameter and \(r\) is the rate parameter. Hence, \(\mu = \frac{s}{r}\) and \(\sigma^{2} = \frac{s}{{r^{2} }}\). These can be solved simultaneously to give

$$s = \frac{{\mu^{2} }}{{\sigma^{2} }},$$
(3)
$$r = \frac{\mu }{{\sigma^{2} }}.$$
(4)

Other distributions with two model parameters for a non-negative random variable such as log normal could also be used, and will produce sampled realisations with both clinical and statistical validity, but the induced correlations might be different.

The sampling procedure depends on the magnitude of the variances of \(X\) and \(Y\).

  • When \(\sigma_{Y}^{2} > \sigma_{X}^{2}\), it involves sampling \(X\) from Normal (\(\mu_{X}\),\(\sigma_{X}^{2}\)) and \({\Delta }\) from Gamma (\(s\),\(r\)). Sampled values of \(Y\) are derived from sampled values of \(X\) and \({\Delta }\) using Eq. 1.

  • When \(\sigma_{Y}^{2} < \sigma_{X}^{2}\), it involves sampling \(Y\) from Normal(\(\mu_{Y}\),\(\sigma_{Y}^{2}\)) and \({\Delta }\) from a Gamma(\(s\),\(r\)). Sampled values of \(X\) are derived from sampled values of \(Y\) and \({\Delta }\) using Eq. 2.

The normal distribution is chosen because both \(X\) and \(Y\) are unbounded and can take any values on the real line. In addition, the input parameter of interest in an economic model is typically the mean of a random variable, which is approximately normally distributed because of the central limit theorem. We have proposed to use a simulation method to derive sampled values using either Eqs. 1 or 2; however, an analytic solution might exist.

2.2 Bounded Ordered Parameters

If both \(X\) and \(Y\) are bounded between 0 and 1, such as probabilities and most HRQoLs, or bounded to be positive, such as cost, we suggest a four-step sampling procedure.

  • Step 1: Given mean and variance of \(X\) and \(Y\), sample \(X\) and \(Y\) from Beta distributions if the \(\left( {X,Y} \right) \in [0,1]\), and sample from Gamma distributions if \(\left( {X,Y} \right) \in [0,\infty )\). The Gamma parameters can be derived using Eqs. 3 and 4. To derive the model parameter for a Beta distribution given the mean and variance, let us assume \(X\) is from a Beta(\(a,b\)) distribution. Hence

$$\mu_{X} = \frac{a}{a + b},$$
$$\sigma_{X}^{2} = \frac{ab}{{\left( {a + b} \right)^{2} \left( {a + b + 1} \right)}}.$$

These two equations can be solved simultaneously to give

$$a = \left( {\frac{{\left( {1 - \mu_{X} } \right)\mu_{X} }}{{\sigma_{X}^{2} }} - 1} \right)\mu_{X} ,$$
(5)
$$b = a\frac{{1 - \mu_{X} }}{{\mu_{X} }}.$$
(6)

If Y is from a \({\text{Beta}}(a,b)\), the Beta model parameters can be derived using Eqs. 5 and 6 by replacing \(\mu_{X}\) and \(\sigma_{X}^{2}\) with \(\mu_{Y}\) and \(\sigma_{Y}^{2}\).

  • Step 2: Transform the sampled \(X\) and \(Y\) to the real line, using \(X^{\prime } = {\text{logit}} \left( X \right) = \log \left( {\frac{X}{1 - X}} \right)\) and \(Y^{\prime } = {\text{logit}}\left( Y \right) = \log (\frac{Y}{1 - Y})\) if the \(\left( {X,Y} \right) \in [0,1]\); and using \(X^{\prime } = \log (X)\) and \(Y^{\prime } = \log (Y)\) if the \(\left( {X,Y} \right) \in [0,\infty\)). Calculate the mean and variance for sampled \(X^{\prime }\) and \(Y^{\prime }\), where \(X^{\prime }\) and \(Y^{\prime }\) are the logit transformed \(X\) and \(Y\) if \(\left( {X,Y} \right) \in [0,1]\), and the log transformed \(X\) and \(Y\) if \(\left( {X,Y} \right) \in [0,\infty )\).

  • Step 3: Use the DM approach for unbounded variables described in Sect. 2.1 to redefine \({\Delta }\) to be the difference between \(X^{\prime }\) and \(Y^{\prime }\). First, sample \({\Delta }\) from a Gamma distribution, where the model parameters can be derived using Eqs. 3 and 4. Then either using sampled values for \(X^{\prime }\) from step 2 and sampled values for \({\Delta }\) to derive sampled values for \(Y^{\prime }\), or using sampled values for \(Y^{\prime }\) from step 2 and sampled values for \({\Delta }\) to derive sampled values for \(X\), depending on the magnitude of the variance of \(X^{\prime }\) and \(Y^{\prime }\).

  • Step 4: Back transform sampled values for \(X^{\prime }\) and \(Y^{\prime }\) to obtain sampled values for \(X\) and \(Y\). If \(\left( {X,Y} \right) \in [0,1]\), then use \(Y = \frac{{e^{{Y^{\prime}}} }}{{1 + e^{{Y^{\prime}}} }}\) and \(X = \frac{{e^{{X^{\prime } }} }}{{1 + e^{{X^{\prime } }} }}\). If \(\left( {X,Y} \right) \in [0,\infty\)), then use \(X = e^{{X^{\prime } }}\) and \(Y = e^{{Y^{\prime } }}\).

The logit or log transformation for the bounded \(X\) and \(Y\) is to make sure that the sampled realisations will be in the appropriate bounded range. Depending on the given summary statistics, some samples may fall outside of the bounded ranges without the transformation.

3 Example

In this section, we illustrate how the proposed DM approach can be implemented in a hypothetical example. Suppose that there is an active (worse) and remission (better) state in an economic model for a particular condition. The input parameters are mean HRQoL and mean cost. In health technology assessments, it is common that an analyst does not have access to the individual patient-level data (IPD), but only summary statistics derived from the IPD. For ordered parameters, often only the summary statistics such as mean and standard deviation/variance for each of the parameters are provided, but not the correlation structure. Suppose that the analyst is given the following information regarding the input parameters.

  • Active (worse) state: the input parameter for HRQoL (\(X_{u}\)) has mean \(\mu_{{X_{u} }} = 0.54\) and standard error 0.138 (variance \(\sigma_{{X_{u} }}^{2} = 0.019\)); the input parameter for cost (\(X_{c}\)) has mean \(\mu_{{X_{c} }} = 110\) and standard error 3.872 (variance \(\sigma_{{X_{c} }}^{2} = 15\)).

  • Remission (better) state: the input parameter for HRQoL (\(Y_{u}\)) has mean \(\mu_{{Y_{u} }} = 0.70\) and standard error 0.126 (variance \(\sigma_{{Y_{u} }}^{2} = 0.016\)); the input parameter for cost (\(Y_{c}\)) has mean \(\mu_{{Y_{c} }} = 100\) and standard error 3.162 (variance \(\sigma_{{Y_{c} }}^{2} = 10\)).

Suppose the analyst requires 5000 samples of realisations for each parameter in each state. It was known that the HRQoL in the remission state is higher than the active state, \(Y_{u} > X_{u}\), and HRQoL is bounded between 0 and 1, \(\left( {X_{u} ,Y_{u} } \right) \in [0,1]\). It was also known that the cost in the remission state is lower than in the active state, \(Y_{c} < X_{c}\), and cost is positive, \(\left( {X_{c} ,Y_{c} } \right) \in [0,\infty )\). The four-step sampling procedure for sampling mean HRQoL and mean cost is given in Table 1.

Table 1 Four-step sampling procedure for generating mean health-related quality of life and mean cost
Full size table

For comparison, we also performed the independent sampling approach, the modified independent sampling approach, and the common random number generator approach to sample mean HRQoL and mean cost in the active and remission states. For the mean HRQoL parameters, the independent sampling approach generates \(Y_{u}\) and \(X_{u}\) by sampling \(Y_{u}\) from \({\text{Beta}}(6.52, 5.55)\) and \(X_{u}\) from \({\text{Beta}}(8.49,3.64)\) independently. The modified independent sampling approach samples independent realisations of \(Y_{u}\) and \(X_{u}\) first and then excludes the draws where the ordering assumption is violated until it reaches the required number of realisations. The common random number generator approach samples \(Y_{u}\) and \(X_{u}\) from the same Beta distributions as the other two approaches, but the same random number generator was used for both parameters. For the mean cost parameters, the distributions used in these three standard sampling approaches were \({\text{Gamma}}(1000,0.100)\) and \({\text{Gamma}}\left( {806.67, 0.136} \right)\) for \(Y_{c}\) and \(X_{c}\), respectively.

We also developed a Microsoft Excel workbook (Microsoft Corporation, Redmond, WA, USA) to implement the DM approach, which is included as online supplementary material to this paper (Online Resource 1). The Microsoft Excel workbook used the example presented in Sect. 3 for an illustration.

4 Results

Table 2 shows that the mean and variance of 5000 sampled realisations using four sampling approaches (the DM, independent sampling, modified independent sampling, and common random number generator approaches). The generated mean and variance of the sampled realisations using all the approaches closely matched the given summary statistics, with the exception of the modified independent sampling approach, which produces biased sampled realisations and should be avoided in practice for handling the logical problem in PSA. This method will always underestimate the uncertainty in the ordered parameters, overestimate the mean of the parameter with a higher value, and underestimate the mean of the parameter with a lower value. The magnitude of the bias depends on the percentages of the sampled realisations that violated the constraint. In the example, 20% of the sampled values in 5000 samples did not meet the logical order for the HRQoL parameter, whereas 2% of the samples violated the order for the cost parameter. The generated summary statistics were closer to the given values for mean cost than mean HRQoL.

Table 2 Mean and variance of 5000 sampled realisations using four sampling approaches
Full size table

The scatterplots of sampled pairs of values using the three methods that produce unbiased sampled realisations are provided in Fig. 1. These scatterplots show that the DM approach guarantees to maintain the constraint that one value is greater than another for each sampled pair, and induced a positive correlation between the ordered variables. The independent sampling approach is likely to lack both clinical and statistical validity since (1) it may produce a sampled value of mean HRQoL in the worse state that is higher than the sampled value of mean HRQoL in the better state (20% of the samples violated the order), and a sampled value of mean cost in the better state that is higher than the sampled value of mean cost in the worse state (2% of the samples violated the order); and (2) there are no correlations between the sampled realisations in the two states. The common random number generator approach is likely to lack statistical validity regarding the correlations since this method implies that given the value of one variable, the value of the other is fixed/determined, i.e. two variables are perfectly correlated.

Fig. 1
figure 1

Scatterplots of pairs of samples generated by three methods: a for HRQoL, where \(X\) is HRQoL in the disease-active state and \(Y\) is HRQoL in the disease-remission state; b for cost, where \(X\) is cost in the disease-remission state and \(Y\) is cost in the disease-active state. HRQoL health-related quality of life, DM difference method, Ind independent sampling method, RN common random number sampling method

Full size image

Using different sampling approaches in generating samples for ordered parameters will have an impact on the uncertainty of the incremental cost-effectiveness ratio (ICER), and hence also the CEAC/CEAF; however, we believe that this would rarely affect conclusions of policy. Hence, we did not compare the methods based on the change in uncertainty in ICER or CEAC/CEAF. Nevertheless, we should not stop improving the rigour of a method just because it may not affect the conclusions.

5 Discussion

Sampling from ordered variables is a common task that an analyst faces in conducting a PSA in cost-effectiveness analysis (CEA). Given the summary statistics for the ordered parameters, the only way to handle the logical problem is by using an appropriate sampling approach in the PSA. We have illustrated that typical approaches used in practice, such as independent sampling, modified independent sampling, and using a common random number generator, lack either statistical or clinical validity. This problem may only rarely affect conclusions or decisions made based on economic evaluation results, but we believe there is a need for a new sampling method to handle the logical problem to improve the rigour of PSAs in CEA.

The proposed DM approach has been shown to be effective in generating bivariate variables that satisfy both clinical and statistical validity in the given examples. It provides a solution to an issue that in theory may have important implications for the interpretation of economic evaluations of health technologies. An earlier version of the method without the transformation step has been used in recent work in health techonology assessment [5, 6], but has been refined and made more generalisable in the process of writing this paper.

When performing a PSA, often only the summary statistics of the sampled realisations are compared with their given values to check the statistical validity. We contend that when sampling from ordered variables, it is also important to consider the clinical validity of the induced correlations between the sampled values. If it is believed that the ordered variables are neither independent nor perfectly dependent, then the independent sampling approach or the common random number generator approach should be avoided. Assuming different parametric distributions in the proposed DM approach may result in induced correlations of different magnitudes; whether there is a best choice in specific scenarios is subject to further research. The modified independent sampling approach should be avoided in all cases because it produces biased sampled realisations.

One limitation of this paper is that we have not conducted a review of the methods used in handling the logical problem in ordered parameters, and have only undertaken one example to illustrate the properties of the proposed method. One drawback of the DM approach is that it does not work if utilities are believed to be below 0. A modification of the method is needed and this requires future research. Some ordered parameters may have a multiplicative relationship rather than an additive. A multiplier approach could be an alternative approach to handle the logical problem. Assigning an appropriate distribution for the multiplier parameter in PSA may be a challenge because it may not have properties as convenient as those in the difference parameter.

We have illustrated how the DM approach works in the case of two ordered parameters. It can be extended to the case of more than two ordered parameters by taking an iterative approach of the proposed method. For example, if there are three variables \(X, Y\) and \(Z\), where the value of \(Z\) is greater than the value of \(Y\), and the value of \(Y\) is greater than the value of \(X\), then we need to define two difference parameters: \({\Delta }_{1}\) represents the difference between \(X\) and \(Y\), and \({\Delta }_{2}\) represents the difference between \(Y\) and \(Z\). We firstly sample \(X\) and \(Y\) using the proposed DM approach, then apply the DM approach again to sampled \(Y\) and \({\Delta }_{2}\) to derive samples for \(Z\).

This logical problem of one variable being strictly greater than another also exists in one-way sensitivity analysis. The proposed method can be used in probabilistic one-way sensitivity analysis to handle ordered parameters. For deterministic one-way sensitivity analysis, making the same change in the related variables will make sure the parameter order remains unchanged; this ‘same change' could either be relative (i.e. a multiple) or absolute (i.e. an increment). This approach is similar to the common random number generator approach in PSA.

6 Conclusions

When producing PSA samples, both clinical and statistical validity should be checked. Where there is a strong belief that variables are constrained in that one value is greater than another, the DM approach should be considered as a method of ensuring the clinical and statistical validity of PSA samples, and the analyses derived from these samples.