Fuzzy approach to decision analysis with multiple criteria and uncertainty in health technology assessment


Decision making in health technology assessment (HTA) involves multiple criteria (clinical outcomes vs. cost) and risk (criteria measured with estimation error). A survey conducted among Polish HTA experts shows that opinions how to trade off health against money should be treated as fuzzy. We propose an approach that allows to introduce fuzziness into decision making process in HTA. Specifically, in the paper we (i) define a fuzzy preference relation between health technologies using an axiomatic approach; (ii) link it to the fuzzy willingness-to-pay and willingness-to-accept notions and show the survey results in Poland eliciting these; (iii) incorportate uncertainty additionally to fuzziness and define two concepts to support decision making: fuzzy expected net benefit and fuzzy expected acceptability (the counterparts of expected net benefit and cost-effectiveness acceptability curves, CEACs, often used in HTA). Illustrative examples show that our fuzzy approach may remove some problems with other methods (CEACs possibly being non-monotonic) and better illustrate the amount of uncertainty present in the decision problem. Our framework can be used in other multiple criteria decision problems under risk where trade-off coefficients between criteria are subjectively chosen.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7


  1. 1.

    Defined in the Reimbursement Act to be equal to triple annual gross domestic product per capita.

  2. 2.

    There are other criteria that may be used in HTA, e.g., the decision maker may want to give priority to treatments being offered to particularly vulnerable populations like patients with rare or ultra-rare diseases.

  3. 3.

    Another issue arises here, beyond the main interest of this paper: based on what grounds can we aggregate utilities between patients, neglecting the problem of inter-personal utility comparisons? Briefly, due to ethical issues we do not want to differentiate treatment provided to individual patients because of differences in their preferences. That is why in HTA we typically measure the clinical improvements in single patients objectively: longevity of life is totally objective, and we might use some definition of health-related quality of life, like EQ-5D. We can then translate health states into utilities using a single population-based value set. Hence, we do not aggregate utilities but rather apply a single utility function to all the patients.

  4. 4.

    It is sometimes claimed that higher drug prices and higher WTP ratios are rationalized in these situations, as the only way to stimulate the R&D process, when the demand is small.

  5. 5.

    Throughout the paper we use the notation that if X is a name of a fuzzy set, then \(X(\cdot )\) denotes a membership function defining this set, and \(X_\alpha \) denotes the \(\alpha \)-cut. Obviously, the membership function and \(\alpha \)-cuts can be derived from each other, and both can serve equally well to define a fuzzy set.

  6. 6.

    By \({\mathbf {1}}_A(x)\) we denote the standard indicator function, whose value is 1 if \(x\in A\), and 0 if \(x\notin A\).

  7. 7.

    In Sect. 2 we denoted by \(E_i\) and \(C_i\) random variables describing the uncertainty of the effect and cost assessment of technology i. To clarify, here \(E_{(i)}\) and \(C_{(i)}\) denote the sequence of random variables, with each element of this sequence (denoted by i) measuring the uncertainty of the difference between the effect and cost of two alternatives being compared.

  8. 8.

    Whenever the underlying first-order uncertainty permits the central limit theorem to hold, i.e., the inter-patient variability is such that increasing the sample size allows to estimate the average effect and cost more precisely.

  9. 9.

    In standard Euclidean metric in \({\mathbb {R}}^2\), denoted by \(d(\cdot ,\cdot )\).


  1. Arrow, K. (1963). Uncertainty and the welfare economics of medical care. The American Economic Review, 53(5), 141–149.

    Google Scholar 

  2. Arrow, K., & Lind, R. (1970). Uncertainty and the evaluation of public intervention decisions. The American Economic Review, 60, 364–378.

    Google Scholar 

  3. Billingsley, P. (1999). Convergence of probability measures (2nd ed.). New York: Wiley.

    Google Scholar 

  4. Black, W. (1990). The CE Plane: A graphic representation of cost-effectiveness. Medical Decision Making, 10, 212–214.

    Article  Google Scholar 

  5. Bleichrodt, H., Wakker, P., & Johannesson, M. (1997). Characterizing QALYs by risk neutrality. Journal of Risk and Uncertainty, 15, 107–114.

    Article  Google Scholar 

  6. Briggs, A., Claxton, K., & Sculpher, M. (2006). Decision modelling for health economic evaluation. Oxford: Oxford University Press.

    Google Scholar 

  7. Briggs, A., & Fenn, P. (1998). Confidence intervals or surfaces? Uncertainty on the cost-effectiveness plane. Health Economics, 7(8), 723–740.

    Article  Google Scholar 

  8. Briggs, A., Weinstein, M. C., Fenwick, E. A., Karnon, J., Sculpher, M. J., & Paltiel, A.D., on behalf of the ISPOR-SMDM Modeling Good Research Practices Task Force. (2012). Model parameter estimation and uncertainty analysis: A report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making, 32(5), 722–732.

  9. Claxton, K. (1999). The irrelevance of inference: A decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics, 18(3), 341–364.

    Article  Google Scholar 

  10. Devlin, N., & Parkin, D. (2004). Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Economics, 13, 437–452.

    Article  Google Scholar 

  11. Eckermann, S., & Willan, A. (2011). Presenting evidence and summary measures to best inform societal decisions when comparing multiple strategies. Pharmacoeconomics, 29(7), 563–577.

    Google Scholar 

  12. Eichler, H. G., Kong, S., Gerth, W., Mavros, P., & Jönsson, B. (2004). Use of cost-effectiveness analysis in health-care resources allocation decision-making: How are cost-effectiveness thresholds expected to emerge? Value in Health, 7(5), 518–528.

    Article  Google Scholar 

  13. Fenwick, E., Claxton, K., & Sculpher, M. (2001). Representing uncertainty: The role of cost-effectiveness acceptability curves. Health Economics, 10(8), 779–787.

    Article  Google Scholar 

  14. Fenwick, E., O’Brien, B., & Briggs, A. (2004). Cost-effectiveness acceptability curves facts, fallacies and frequently asked questions. Health Economics, 13, 405–415.

    Article  Google Scholar 

  15. Gafni, A., & Birch, S. (2006). Incremental cost-effectiveness ratios (ICERs): The silence of the lambda. Social Science & Medicine, 62, 2091–2100.

    Article  Google Scholar 

  16. Garber, A. (2000). Advances in cost-effectiveness analysis of health interventions. In A. J. Culyer (Ed.), Handbook of health economics (Vol. 1A, pp. 181–221). Amsterdam: North-Holland.

    Google Scholar 

  17. Gold, M., Siegel, J., Russell, L., & Weinstein, M. (Eds.). (1996). Cost-effectiveness in health and medicine. Oxford: Oxford University Press.

    Google Scholar 

  18. Hunink, M. G., Bult, J. R., de Vries, J., & Weinstein, M. C. (1998). Uncertainty in decision models analyzing cost-effectiveness: The joint distribution of incremental costs and effectiveness evaluated with a nonparametric bootstrap method. Medical Decision Making, 18(3), 337–346.

    Article  Google Scholar 

  19. Jakubczyk, M., & Kamiński, B. (2010). Cost-effectiveness acceptability curves-caveats quantified. Health Economics, 19, 955–963.

    Article  Google Scholar 

  20. Kahneman, D., Knetsch, J., & Thaler, R. (2009). Experimental tests of the endowment effect and the coase theorem. In E. L. Khalil (Ed.), The new behavioral economics. Volume 3. Tastes for endowment, identitiy and the emotions (Vol. 3, pp. 119–142). London: Elgar.

    Google Scholar 

  21. Klir, G., & Yuan, B. (1995). Fuzzy sets and fuzzy logic: Theory and applications. Englewood Cliffs NJ: Prentice Hall.

    Google Scholar 

  22. Lee, K. (2005). First course on fuzzy theory and applications. Berlin: Springer.

    Google Scholar 

  23. Löthgren, M., & Zethraeus, N. (2000). Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Economics, 9, 623–630.

    Article  Google Scholar 

  24. Moreno, E., Girón, F., Vázquez-Polo, F., & Negrín, M. (2010). Optimal healthcare decisions: Comparing medical treatments on a cost-effectiveness basis. European Journal of Operational Research, 204, 180–187.

    Article  Google Scholar 

  25. Moreno, E., Girón, F., Vázquez-Polo, F., & Negrín, M. (2013). Optimal treatments in cost-effectiveness analysis in the presence of covariates: Improving patient subgroup definition. European Journal of Operational Research, 226, 173–182.

    Article  Google Scholar 

  26. Obenchain, R. (1997). Issues and algorithms in cost-effectiveness inference. Biopharmaceutical Report, 5(2), 1–7.

    Google Scholar 

  27. Obenchain, R. (2008). ICE preference maps: Nonlinear generalizations of net benefit and acceptability. Health Services and Outcomes Research Methodology, 8, 31–56.

    Article  Google Scholar 

  28. O’Brien, B., Gertsen, K., Willan, A., & Faulkner, L. (2002). Is there a kink in consumers’ threshold value for cost-effectiveness in health care? Health Economics, 11, 175–180.

    Article  Google Scholar 

  29. Pliskin, J., Shepard, D., & Weinstein, M. (1980). Utility functions for life years and health status. Operations Research, 28(1), 206–224.

    Article  Google Scholar 

  30. Sadatsafavi, M., Najafzadeh, M., & Marra, C. (2008). Acceptability curves could be misleading when correlated strategies are compared. Medical Decision Making, 28(3), 306–307.

    Article  Google Scholar 

  31. Severens, J., Brunenberg, D., Fenwick, E., O’Brien, B., & Joore, M. (2005). Cost-effectiveness acceptability curves and a reluctance to lose. Pharmacoeconomics, 23(12), 1207–1214.

    Article  Google Scholar 

  32. van Hout, B., Al, M., Gordon, G., & Rutten, F. (1994). Costs, effects and C:E-ratios alongside a clinical trial. Health Economics, 3, 309–319.

    Article  Google Scholar 

  33. Weinstein, M., & Zeckhauser, R. (1973). Critical ratios and efficient allocation. Journal of Public Economics, 2, 147–157.

    Article  Google Scholar 

  34. Zaric, G. (2010). Cost-effectiveness analysis, health-care policy, and operations research models. In Wiley Encyclopedia of operations research and management science, Wiley. doi:10.1002/9780470400531.eorms0202.

  35. Zivin, J., & Bridges, J. (2002). Addressing risk preferences in cost-effectiveness analysis. Applied Health Economics and Health Policy, 1(3), 135–139.

    Google Scholar 

Download references


It would not have been possible to collect the survey results presented in the paper without the help from M. Niewada, who facilitated the contact with the respondents. We would like to acknowledge the help of HTA experts who participated in the survey: K. J. Filipiak, K. Jahnz-Różyk, and the others, who opted to remain anonymous. We also express our gratitude to D. Golicki, T. Macioch, W. Wrona, and again M. Niewada, who commented on the first version of the survey.

Author information



Corresponding author

Correspondence to Michał Jakubczyk.



Proof of Proposition 1

For \(e=0\) we have \(\text {fNB}(x)=\mu (0,c+x)\) and so is equal to 1 for \(x\ge -c\), and to 0 for \(x<-c\). Hence \(\text {fNB}_\alpha =]-\infty ,-c]\) for \(\alpha \in ]0,1]\),while \(\tau _\alpha =-c\).

For \(e>0\):

$$\begin{aligned}&\sup (\text {fNB}_\alpha )=\sup (\left\{ x\in {\mathbb {R}}:\, \mu (e,c+x)\ge \alpha \right\} )=\sup \left( \left\{ x\in {\mathbb {R}}:\, \mu \left( 1,\frac{c+x}{e}\right) \ge \alpha \right\} \right) \\&\quad =\sup (\left\{ ye-c:\, y\in {\mathbb {R}}\wedge \mu (1,y)\ge \alpha \right\} )\\&\quad =e\times \sup (\left\{ y\in {\mathbb {R}}:\, \mu (1,y)\ge \alpha \right\} )-c\\&\quad =e\times \sup (\text {fWTP}_\alpha )-c, \end{aligned}$$

where the first equality comes from the definition of fNB; the second from scale invariance, the third from substitution \(x=ye-c\), the fourth from taking irrelevant terms out of \(\sup \) function, the fifth from the definition of fWTP.

For \(e<0\) we proceed analogously (we replace \(\sup \) with \(\inf \) as we take \(e<0\) out):

$$\begin{aligned}&\sup (\text {fNB}_\alpha )=\sup (\left\{ x\in {\mathbb {R}}:\, \mu (e,c+x)\ge \alpha \right\} )\\&\quad =\sup \left( \left\{ x\in {\mathbb {R}}:\, \mu \left( -1,\frac{-c-x}{e}\right) \ge \alpha \right\} \right) \\&\quad =\sup (\left\{ ye-c:\, y\in {\mathbb {R}}\wedge \mu (-1,-y)\ge \alpha \right\} )\\&\quad =e\times \inf (\left\{ y\in {\mathbb {R}}:\, \mu (-1,-y)\ge \alpha \right\} )-c\\&\quad =e\times \inf (\text {fWTA}_\alpha )-c. \end{aligned}$$

Obviously, if for \(\alpha \in ]0,1]\) the sets \(\left\{ (e,c)\in {\mathbb {R}}^2:\, \mu (\cdot ,\cdot )\ge \alpha \right\} \) are closed, then \(\text {fNB}_\alpha (e,c)\) is closed. \(\square \)

Proof of Proposition 2

Due to monotonicity axiom \(\text {fNB}(\cdot )\) is non-increasing. Therefore \(\forall \alpha _1,\alpha _2\in ]0,1],\alpha _1<\alpha _2\), we have \(\sup (\text {fNB}_{\alpha _2})\le \sup (\text {fNB}_{\alpha _1})\), and this inequality is preserved under integration in Definition 4. Regarding fEA, obviously \({\mathbf {1}}_{[\alpha ,1]}\left( \mu (e,c)\right) \) is non-increasing with \(\alpha \), and this (weak) inequality is preserved under integration. \(\square \)

Proof of Proposition 3

When WTP and WTA are crisp then \(\mu (\cdot ,\cdot )\) only takes values in \(\left\{ 0,1 \right\} \). Therefore, for any \(\alpha >0\) all intervals \(\text {fENB}_\alpha \) are identical, and the same holds for \(\text {fEA}_\alpha \). Thus it suffices to calculate \(\text {fENB}_1\) and \(\text {fEA}_1\).

We first focus on \(\text {fENB}_1\). From Eq. (2) we get \(\phi _1=\int _\varOmega \sup \left( {\text {fNB}}_1\left( e,c\right) \right) \,{\mathrm {d}}{\fancyscript{P}}\). Now we remember that WTP and WTA are crisp and have the same value t, so by applying Proposition 1 we get:

$$\begin{aligned} \phi _1=\int _\varOmega (et-c)\,{\mathrm {d}}{\fancyscript{P}} = {\mathbb {E}}(Et-C). \end{aligned}$$

This means that \(\text {fENB}(x)\) is equal to 1 for \(x\le {\mathbb {E}}(Et-C)\), and drops to 0 if x is larger. This is exactly the result given in the theorem. Now we move to \(\text {fEA}_1\). From Eq. 3 we have:

$$\begin{aligned} \text {fEA}_1=\left[ 0,\int _\varOmega {\mathbf {1}}_{\{1\}}(\mu (e,c)) \,{\mathrm {d}}{\fancyscript{P}}\right] . \end{aligned}$$

However, because WTP and WTA have exactly the same crisp value t, we see that \(\mu (e,c)=1\) when \(et-c\ge 0\) which leads to \(\text {fEA}_1=\left[ 0,{\mathbb {P}}(Et-C)\right] \). This means that \(\text {fEA}(x)\) is equal to 1 for \(x\le {\mathbb {P}}(Et-C)\), and drops to 0 if x is larger. This is exactly the result given in the theorem. \(\square \)

Proof of Proposition 4

We start with \(\text {fEA}_{(i)}\). We will show that for \(\beta >\mu (e^*,c^*)\) we have \(\sup \left( \text {fEA}_{(i),\beta }\right) \rightarrow 0\) and for \(\beta <\mu (e^*,c^*)\) we have \(\sup \left( \text {fEA}_{(i),\beta }\right) \rightarrow 1\). This will show that for fixed \(x\in ]0,1[\) we have \(\sup \{\alpha \in [0,1]:\, x\in \text {fEA}_{(i),\alpha }\}\rightarrow \mu (e^*,c^*)\). We start with \(\beta >\mu (e^*,c^*)\). We have:

$$\begin{aligned} \text {fEA}_{i,\beta }=\left[ 0,\int _\varOmega {\mathbf {1}}_{[\beta ,1]}(\mu (e,c)) \,{\mathrm {d}}{\fancyscript{P}}_i\right] . \end{aligned}$$

As \(\mu \) is continuous in \((e^*,c^*)\), there exists \(\delta >0\) such that for all (ec) closerFootnote 9 to \((e^*,c^*)\) than \(\delta \) we have \(\mu (e,c)<\beta \). Then:

$$\begin{aligned} \int _\varOmega {\mathbf {1}}_{[\beta ,1]}(\mu (e,c)) \,{\mathrm {d}}{\fancyscript{P}}_i&= \int _{\varOmega ,d((e,c),(e^*,c^*))<\delta } {\mathbf {1}}_{[\beta ,1]}(\mu (e,c)) \,{\mathrm {d}}{\fancyscript{P}}_i \,+\\&\quad +\,\int _{\varOmega ,d((e,c),(e^*,c^*))\ge \delta } {\mathbf {1}}_{[\beta ,1]}(\mu (e,c)) \,{\mathrm {d}}{\fancyscript{P}}_i. \end{aligned}$$

The first term is simply equal to 0, because in the region of integration \(\mu \) is less than \(\beta \). The second term converges to 0 with \(i\rightarrow +\infty \) because \((E_{(i)},C_{(i)}\) converges in probability to \((e^*,c^*)\) and \({\mathbf {1}}_{[\beta ,1]}(\mu (e,c))\) is bounded. Thus \(\sup (\text {fEA}_{(i),\beta })\rightarrow 0\).

The proof for \(\beta <\mu (e^*,c^*)\) is analogous, but then the first term is equal to

$$\begin{aligned} {\mathbb {P}}(d((E_{(i)},C_{(i)}),(e^*,c^*))<\delta ), \end{aligned}$$

and this value converges to 1 because \((E_{(i)},C_{(i)})\) converges in probability to \((e^*,c^*)\).

Now we move to the proof of convergence of \(\text {fENB}_{(i),\alpha }\). We have to show that \(\phi _{i,\alpha }=\int _\varOmega \sup \left( {\text {fNB}}_\alpha \left( e,c\right) \right) \,{\mathrm {d}}{\fancyscript{P}}_i\) from Eq. (2) converges to \(\sup \left( {\text {fNB}}_\alpha \left( e^*,c^*\right) \right) \). Because we assume that \((E_{(i)},C_{(i)})\) converges in probability to \((e^*,c^*)\) then by the continuous mapping theorem (Billingsley 1999) it is enough to show that \(g(e,c)=\sup \left( {\text {fNB}}_\alpha \left( e,c\right) \right) \) is continuous at \((e^*,c^*)\). Observe from Eq. (1) and the proof of Proposition 1 that \(\sup (fNB_\alpha (e,c))=\tau _\alpha (e,c)\), and is obviously continuous in the whole \({\mathbb {R}}^2\). \(\square \)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jakubczyk, M., Kamiński, B. Fuzzy approach to decision analysis with multiple criteria and uncertainty in health technology assessment. Ann Oper Res 251, 301–324 (2017). https://doi.org/10.1007/s10479-015-1910-9

Download citation


  • Multiple criteria decision making
  • Fuzzy preferences
  • Uncertainty
  • Health technology assessment
  • Willingness to pay
  • Preference elicitation