A New Approach for Sampling Ordered Parameters in Probabilistic Sensitivity Analysis
Abstract
Background
Probabilistic sensitivity analysis (PSA) in costeffectiveness analysis involves sampling a large number of realisations of an economic model. For some parameters, we may be uncertain around the true mean values of the variables, but the ordering of the values is known. Typical sampling approaches lack either statistical or clinical validity. For example, sampling using a common number generator results in extreme dependence, and independent sampling can lead to realisations with incorrect ordering.
Methods
We propose a new sampling approach for ordered parameters, the difference method (DM) approach, which samples the parameters of interest via a difference parameter. If the parameters of interest are bounded, it involves transforming the variables so that they are unbounded and then sampling via the difference parameter. We have provided a Microsoft Excel workbook to implement the method. The proposed approach is illustrated with an example sampling ordered parameters for utility and cost.
Results
The DM approach has a number of advantages when comparing with the typical approaches used in practice. It generates PSA samples that have similar summary statistics as the given values in our examples, while maintaining the constraint that one value was greater than another. The method also implies plausible positive correlation between the two ordered variables.
Conclusions
Both clinical and statistical validity should be checked when producing PSA samples. The DM approach should be considered as a solution to potential problems in generating PSA samples for ordered parameters.
Key Points for Decision Makers
In health economic models, uncertain ordered parameters are common. For example, utility and treatment costs associated with different severity levels of a disease are commonly ordered. 
For ordered parameters, which are believed to be related such that the value of one variable is always greater than another, this information on ordering should always be appropriately incorporated within PSA. 
The proposed sampling approach generates ordered parameters with both clinical and statistical validity for use in PSA. 
1 Introduction
In health technology assessment, probabilistic sensitivity analysis (PSA) represents the generally accepted approach for characterising the uncertainty in parameters included in an economic model and for producing accurate results in nonlinear models [1, 2]. This involves generating a large number of realisations of the economic model, each time sampling values from the distributions applied to each uncertain parameter included in the model.
Sometimes there is an absolute belief that the value of one variable is greater than the value of another. There may be uncertainty around the true mean values of the variables, but the ordering of the values is known. For example, if an individual rates his/her general health as ‘good’, then later as ‘fair’, we might be uncertain about how to map the ‘good’ and ‘fair’ health evaluations onto a numeric scale, but can assume the ‘good’ general health score will be higher than the ‘fair’ general health score. Healthrelated quality of life (HRQoL) for two different severity levels of a disease may also be related in this manner.
Failure to account for constraints between samples of ordered parameters may result in PSA values that do not accurately characterise the uncertainty present in a decision problem. In theory, this could result in decisions made on the allocation of scarce healthcare resources being suboptimal, although the direction of bias would depend on the specific model. Other outputs and analyses that are reliant on the PSA, such as costeffectiveness acceptability curves (CEACs) and frontiers (CEAFs), and value of information analyses, are likely to also be inaccurate if the constraint that the value of one variable is greater than another is not accounted for appropriately.
Based on the authors’ experience, as National Institute for Health and Care Excellence (NICE) committee members, Evidence Review Group members dealing with a large number of NICE submissions every year, and peer reviewers, we have identified multiple examples where sampling of ordered parameters has not been handled correctly. Independent sampling from the distributions of the ordered parameters could result in a PSA that lacks clinical validity as in some realisations the logical constraint may be violated, with the sampled value of parameters potentially implying that having a disease makes people healthier. We refer to this method as the independent sampling approach. For example, in an NICE Single Technology Appraisal [3] the use of the independent sampling approach results in the utility in the progressed state being higher than in the stable state. Another sampling method used in practice used a common random number to draw from both distributions [4], which induces extreme dependence: one variable is a deterministic function of another. In addition to the extreme dependence induced by this method, the sampled realisations of PSA could still violate the known ordering if the cumulative density functions of the two parameters cross. We refer to this method as the common random number generator approach. A flawed alternative observed in papers sent to the authors for peer review is a method whereby samples are excluded when the ordering assumption is violated. We refer to this method as the modified independent sampling approach. This approach results in the summary statistics of the sampled realisations not equalling that of the source data, and provides biased sampled realisations.
Typical approaches described above lack either statistical or clinical validity. To have clinical validity when sampling parameters where it is known that one value is greater than another, all PSA realisations should exhibit the logical order of the given parameters. To have statistical validity, we have deemed that the following criteria should be met: (1) if the parameters of interest are bounded then all PSA realisations should be in these bounded ranges; (2) the summary statistics of the sampled values of the parameters of interest from PSA should match closely to the given summary statistics of the parameters of interest; (3) the induced correlation between the sampled values of parameters of interest should be plausible.
Introducing an additional parameter that presents either an absolute difference or a ratio of the ordered parameters is a common approach for modelling the ordered parameters explicitly. In Sect. 2, we apply this modelling idea in generating samples for PSA, and propose a new approach for sampling ordered parameters where summary statistics have been provided for the parameters of interest. We call this method the difference method (DM) approach. This approach is illustrated with an example sampling ordered parameters for utility and cost in Sect. 3, while Sect. 4 provides the results of comparing the performance of the DM approach with the typical approaches used in practice. Discussion is given in Sect. 5.
2 Methods
2.1 Unbounded Ordered Parameters
Other distributions with two model parameters for a nonnegative random variable such as log normal could also be used, and will produce sampled realisations with both clinical and statistical validity, but the induced correlations might be different.

When \(\sigma_{Y}^{2} > \sigma_{X}^{2}\), it involves sampling \(X\) from Normal (\(\mu_{X}\),\(\sigma_{X}^{2}\)) and \({\Delta }\) from Gamma (\(s\),\(r\)). Sampled values of \(Y\) are derived from sampled values of \(X\) and \({\Delta }\) using Eq. 1.

When \(\sigma_{Y}^{2} < \sigma_{X}^{2}\), it involves sampling \(Y\) from Normal(\(\mu_{Y}\),\(\sigma_{Y}^{2}\)) and \({\Delta }\) from a Gamma(\(s\),\(r\)). Sampled values of \(X\) are derived from sampled values of \(Y\) and \({\Delta }\) using Eq. 2.
The normal distribution is chosen because both \(X\) and \(Y\) are unbounded and can take any values on the real line. In addition, the input parameter of interest in an economic model is typically the mean of a random variable, which is approximately normally distributed because of the central limit theorem. We have proposed to use a simulation method to derive sampled values using either Eqs. 1 or 2; however, an analytic solution might exist.
2.2 Bounded Ordered Parameters

Step 1: Given mean and variance of \(X\) and \(Y\), sample \(X\) and \(Y\) from Beta distributions if the \(\left( {X,Y} \right) \in [0,1]\), and sample from Gamma distributions if \(\left( {X,Y} \right) \in [0,\infty )\). The Gamma parameters can be derived using Eqs. 3 and 4. To derive the model parameter for a Beta distribution given the mean and variance, let us assume \(X\) is from a Beta(\(a,b\)) distribution. Hence

Step 2: Transform the sampled \(X\) and \(Y\) to the real line, using \(X^{\prime } = {\text{logit}} \left( X \right) = \log \left( {\frac{X}{1  X}} \right)\) and \(Y^{\prime } = {\text{logit}}\left( Y \right) = \log (\frac{Y}{1  Y})\) if the \(\left( {X,Y} \right) \in [0,1]\); and using \(X^{\prime } = \log (X)\) and \(Y^{\prime } = \log (Y)\) if the \(\left( {X,Y} \right) \in [0,\infty\)). Calculate the mean and variance for sampled \(X^{\prime }\) and \(Y^{\prime }\), where \(X^{\prime }\) and \(Y^{\prime }\) are the logit transformed \(X\) and \(Y\) if \(\left( {X,Y} \right) \in [0,1]\), and the log transformed \(X\) and \(Y\) if \(\left( {X,Y} \right) \in [0,\infty )\).

Step 3: Use the DM approach for unbounded variables described in Sect. 2.1 to redefine \({\Delta }\) to be the difference between \(X^{\prime }\) and \(Y^{\prime }\). First, sample \({\Delta }\) from a Gamma distribution, where the model parameters can be derived using Eqs. 3 and 4. Then either using sampled values for \(X^{\prime }\) from step 2 and sampled values for \({\Delta }\) to derive sampled values for \(Y^{\prime }\), or using sampled values for \(Y^{\prime }\) from step 2 and sampled values for \({\Delta }\) to derive sampled values for \(X\), depending on the magnitude of the variance of \(X^{\prime }\) and \(Y^{\prime }\).

Step 4: Back transform sampled values for \(X^{\prime }\) and \(Y^{\prime }\) to obtain sampled values for \(X\) and \(Y\). If \(\left( {X,Y} \right) \in [0,1]\), then use \(Y = \frac{{e^{{Y^{\prime}}} }}{{1 + e^{{Y^{\prime}}} }}\) and \(X = \frac{{e^{{X^{\prime } }} }}{{1 + e^{{X^{\prime } }} }}\). If \(\left( {X,Y} \right) \in [0,\infty\)), then use \(X = e^{{X^{\prime } }}\) and \(Y = e^{{Y^{\prime } }}\).
The logit or log transformation for the bounded \(X\) and \(Y\) is to make sure that the sampled realisations will be in the appropriate bounded range. Depending on the given summary statistics, some samples may fall outside of the bounded ranges without the transformation.
3 Example

Active (worse) state: the input parameter for HRQoL (\(X_{u}\)) has mean \(\mu_{{X_{u} }} = 0.54\) and standard error 0.138 (variance \(\sigma_{{X_{u} }}^{2} = 0.019\)); the input parameter for cost (\(X_{c}\)) has mean \(\mu_{{X_{c} }} = 110\) and standard error 3.872 (variance \(\sigma_{{X_{c} }}^{2} = 15\)).

Remission (better) state: the input parameter for HRQoL (\(Y_{u}\)) has mean \(\mu_{{Y_{u} }} = 0.70\) and standard error 0.126 (variance \(\sigma_{{Y_{u} }}^{2} = 0.016\)); the input parameter for cost (\(Y_{c}\)) has mean \(\mu_{{Y_{c} }} = 100\) and standard error 3.162 (variance \(\sigma_{{Y_{c} }}^{2} = 10\)).
Fourstep sampling procedure for generating mean healthrelated quality of life and mean cost
Sampling procedure  Sampling mean HRQoL  Sampling mean cost 

Step 1: Sample \(Y\) and \(X\)  Sample 5000 values of \(Y_{u}\) from \({\text{Beta}}(6.52, 5.55)\) and 5000 values of \(X_{u}\) from \({\rm Beta}(8.49,3.64)\), where the Beta distribution parameters were calculated using Eqs. (5) and (6).  Sample 5000 values of \(Y_{c}\) from \({\text{Gamma}}(1000,0.100)\) and 5000 values of \(X_{c}\) from \({\text{Gamma}}(806.67, 0.136)\), where the Gamma distribution parameters were calculated using Eqs. (3) and (4). 
Step 2: Transform sampled \(Y\) and \(X\) from step 1 to unbounded range  Transform sampled \(Y_{u}\) and \(X_{u}\) from step 1 to unbounded range using logit function: \(Y_{u}^{\prime } = {\text{logit}}(Y_{u} )\) and \(X_{u}^{\prime } = {\text{logit}}(X_{u} )\). The mean and variance for the transformed variables are \(\mu_{{Y_{u} }}^{\prime} = 0.928\), \(\sigma_{{Y_{u} }}^{\prime 2} = 0.436\), \(\mu_{{X_{u} }}^{\prime } = 0.186\), and \(\sigma_{{X_{u} }}^{\prime 2} = 0.364\).  Transform sampled \(Y_{c}\) and \(X_{c}\) from step 1 to unbounded range using log function: \(Y_{c}^{\prime } = \log (Y_{c} )\) and \(X_{c}^{\prime } = \log (X_{c} )\). The mean and variance for the transformed variables are \(\mu_{{Y_{c} }}^{\prime } = 4.60\), \(\sigma_{{Y_{c} }}^{\prime 2} = 0.00101\), \(\mu_{{X_{c} }}^{\prime } = 4.70\), and \(\sigma_{{X_{c} }}^{\prime 2} = 0.00124\). 
Step 3: Sample the difference between transformed \(Y\) and \(X\)  Since \(\sigma_{{Y_{u} }}^{\prime 2} > \sigma_{{X_{u} }}^{\prime 2}\), we define \(Y_{u}^{\prime } = X_{u}^{\prime } + \Delta_{u}^{\prime }\), where the mean and variance of \(\Delta_{u}^{\prime }\) is 0.742 and 0.072, respectively. Sample \(\Delta_{u}^{\prime }\) from \({\rm Gamma}(7.64, 0.10)\), where the Gamma distribution parameters were calculated using Eqs. (3) and (4). Compute sampled values of \(Y_{u}^{\prime}\) by adding sampled \({\Delta }_{u}^{\prime}\) and sampled \(X_{u}^{\prime}\) from step 2.  Since \(\sigma_{{Y_{c} }}^{\prime 2} < \sigma_{{X_{c} }}^{\prime 2}\), we define \(X_{c}^{\prime} = Y_{c}^{\prime} + {\Delta }_{c}^{\prime}\), where the mean and variance of \(\Delta_{c}^{\prime }\) is 0.10 and 0.0002, respectively. Sample \(\Delta_{c}^{\prime }\) from \({\rm Gamma}(41.07, 0.002)\), where the Gamma distribution parameters were calculated using Eqs. (3) and (4). Compute sampled values of \(X_{c}^{\prime}\) by subtracting sampled \({\Delta }_{c}^{\prime}\) from sampled \(Y_{c}^{\prime}\) from step 2. 
Step 4: Back transform  Back transform sampled \(Y_{u}^{\prime}\) and \(X_{u}^{\prime}\) to \(Y_{u}\) and \(X_{u}\) using \(Y_{u} = \frac{{e^{{Y_{u}^{\prime} }} }}{{1 + e^{{Y_{u}^{\prime} }} }}\) and \(X_{u} = \frac{{e^{{X_{u}^{\prime} }} }}{{1 + e^{{X_{u}^{\prime} }} }}\).  Back transform sampled \(X_{c}^{\prime}\) and \(Y_{c}^{\prime}\) to \(X_{c}\) and \(Y_{c}\) using \(X_{c} = e^{{X_{c}^{\prime} }}\) and \(Y_{c} = e^{{Y_{c}^{\prime} }}\). 
For comparison, we also performed the independent sampling approach, the modified independent sampling approach, and the common random number generator approach to sample mean HRQoL and mean cost in the active and remission states. For the mean HRQoL parameters, the independent sampling approach generates \(Y_{u}\) and \(X_{u}\) by sampling \(Y_{u}\) from \({\text{Beta}}(6.52, 5.55)\) and \(X_{u}\) from \({\text{Beta}}(8.49,3.64)\) independently. The modified independent sampling approach samples independent realisations of \(Y_{u}\) and \(X_{u}\) first and then excludes the draws where the ordering assumption is violated until it reaches the required number of realisations. The common random number generator approach samples \(Y_{u}\) and \(X_{u}\) from the same Beta distributions as the other two approaches, but the same random number generator was used for both parameters. For the mean cost parameters, the distributions used in these three standard sampling approaches were \({\text{Gamma}}(1000,0.100)\) and \({\text{Gamma}}\left( {806.67, 0.136} \right)\) for \(Y_{c}\) and \(X_{c}\), respectively.
We also developed a Microsoft Excel workbook (Microsoft Corporation, Redmond, WA, USA) to implement the DM approach, which is included as online supplementary material to this paper (Online Resource 1). The Microsoft Excel workbook used the example presented in Sect. 3 for an illustration.
4 Results
Mean and variance of 5000 sampled realisations using four sampling approaches
Variable  Health state  Given mean (variance)  Generated mean (variance)  

Difference method  Independent sampling  Modified independent sampling  Common random number  
HRQoL  Remission  0.70 (0.016)  0.70 (0.016)  0.70 (0.016)  0.73 (0.012)  0.70 (0.016) 
Active  0.54 (0.019)  0.54 (0.019)  0.54 (0.019)  0.51 (0.016)  0.54 (0.020)  
Cost  Remission  100 (10)  100.02 (9.94)  100.02 (9.94)  99.9 (9.54)  100.02 (9.95) 
Active  110 (15)  109.98 (15.29)  109.98 (15.10)  110.15 (13.87)  110.03 (14.93) 
Using different sampling approaches in generating samples for ordered parameters will have an impact on the uncertainty of the incremental costeffectiveness ratio (ICER), and hence also the CEAC/CEAF; however, we believe that this would rarely affect conclusions of policy. Hence, we did not compare the methods based on the change in uncertainty in ICER or CEAC/CEAF. Nevertheless, we should not stop improving the rigour of a method just because it may not affect the conclusions.
5 Discussion
Sampling from ordered variables is a common task that an analyst faces in conducting a PSA in costeffectiveness analysis (CEA). Given the summary statistics for the ordered parameters, the only way to handle the logical problem is by using an appropriate sampling approach in the PSA. We have illustrated that typical approaches used in practice, such as independent sampling, modified independent sampling, and using a common random number generator, lack either statistical or clinical validity. This problem may only rarely affect conclusions or decisions made based on economic evaluation results, but we believe there is a need for a new sampling method to handle the logical problem to improve the rigour of PSAs in CEA.
The proposed DM approach has been shown to be effective in generating bivariate variables that satisfy both clinical and statistical validity in the given examples. It provides a solution to an issue that in theory may have important implications for the interpretation of economic evaluations of health technologies. An earlier version of the method without the transformation step has been used in recent work in health techonology assessment [5, 6], but has been refined and made more generalisable in the process of writing this paper.
When performing a PSA, often only the summary statistics of the sampled realisations are compared with their given values to check the statistical validity. We contend that when sampling from ordered variables, it is also important to consider the clinical validity of the induced correlations between the sampled values. If it is believed that the ordered variables are neither independent nor perfectly dependent, then the independent sampling approach or the common random number generator approach should be avoided. Assuming different parametric distributions in the proposed DM approach may result in induced correlations of different magnitudes; whether there is a best choice in specific scenarios is subject to further research. The modified independent sampling approach should be avoided in all cases because it produces biased sampled realisations.
One limitation of this paper is that we have not conducted a review of the methods used in handling the logical problem in ordered parameters, and have only undertaken one example to illustrate the properties of the proposed method. One drawback of the DM approach is that it does not work if utilities are believed to be below 0. A modification of the method is needed and this requires future research. Some ordered parameters may have a multiplicative relationship rather than an additive. A multiplier approach could be an alternative approach to handle the logical problem. Assigning an appropriate distribution for the multiplier parameter in PSA may be a challenge because it may not have properties as convenient as those in the difference parameter.
We have illustrated how the DM approach works in the case of two ordered parameters. It can be extended to the case of more than two ordered parameters by taking an iterative approach of the proposed method. For example, if there are three variables \(X, Y\) and \(Z\), where the value of \(Z\) is greater than the value of \(Y\), and the value of \(Y\) is greater than the value of \(X\), then we need to define two difference parameters: \({\Delta }_{1}\) represents the difference between \(X\) and \(Y\), and \({\Delta }_{2}\) represents the difference between \(Y\) and \(Z\). We firstly sample \(X\) and \(Y\) using the proposed DM approach, then apply the DM approach again to sampled \(Y\) and \({\Delta }_{2}\) to derive samples for \(Z\).
This logical problem of one variable being strictly greater than another also exists in oneway sensitivity analysis. The proposed method can be used in probabilistic oneway sensitivity analysis to handle ordered parameters. For deterministic oneway sensitivity analysis, making the same change in the related variables will make sure the parameter order remains unchanged; this ‘same change' could either be relative (i.e. a multiple) or absolute (i.e. an increment). This approach is similar to the common random number generator approach in PSA.
6 Conclusions
When producing PSA samples, both clinical and statistical validity should be checked. Where there is a strong belief that variables are constrained in that one value is greater than another, the DM approach should be considered as a method of ensuring the clinical and statistical validity of PSA samples, and the analyses derived from these samples.
Notes
Acknowledgements
The authors would like to thank Jeremy Oakley for the discussions during the development of the method, and the editor and two referees for their helpful and constructive comments.
Author Contributions
Shijie Ren and Jonathan Minton were primarily responsible for drafting the manuscript. Sophie Whyte, Nicholas R. Latimer and Matt Stevenson contributed towards writing the article and commented on various versions of the manuscript. Sophie Whyte also led the development of the Microsoft Excel workbook.
Compliance with Ethical Standards
Data Availability Statement
All data generated or analysed during this study are included in this published article.
Funding
No funding was received for the preparation of this study.
Conflict of interest
Shijie Ren, Jonathan Minton, Sophie Whyte, Nicholas R. Latimer and Matt Stevenson declare that they have no conflicts of interest.
Supplementary material
References
 1.NICE. Guide to the methods of technology appraisal 2013. NICE; 2013. https://www.nice.org.uk/process/pmg9/chapter/foreword. Accessed 3 Apr 2017.
 2.Claxton K, Sculpher M, McCabe C, Briggs A, Akehurst R, Buxton M, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14:339–47.CrossRefPubMedGoogle Scholar
 3.Kearns B, Lloyd Jones M, Stevenson M, Littlewood C. Cabazitaxel for the secondline treatment of metastatic hormonerefractory prostate cancer: A NICE single technology appraisal. Pharmacoeconomics. 2013;31:479–88.CrossRefPubMedGoogle Scholar
 4.Carroll C, Stevenson M, Scope A, Evans P, Buckley S. Hemiarthroplasty and total hip arthroplasty for treating primary intracapsular fracture of the hip: a systematic review and costeffectiveness analysis. Health Technol Assess. 2011;15:1–74.PubMedCentralGoogle Scholar
 5.Harnan SE, Tappenden P, Essat M, Gomersall T, Minton J, Wong R, et al. Measurement of exhaled nitric oxide concentration in asthma: a systematic review and economic evaluation of NIOX MINO, NIOX VERO and NObreath. Health Technol Assess. 2015;19:1–330.CrossRefPubMedPubMedCentralGoogle Scholar
 6.Archer R, Tappenden P, Ren S, MartynSt James M, Harvey R, Basarir H, et al. Infliximab, adalimumab and golimumab for treating moderately to severely active ulcerative colitis after the failure of conventional therapy (including a review of TA140 and TA262): clinical effectiveness systematic review and economic model. Health Technol Assess. 2016;20:1–326.CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons AttributionNonCommercial 4.0 International License (http://creativecommons.org/licenses/bync/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.