Skip to main content


Log in

Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists

  • Original Paper
  • Published:
The European Journal of Health Economics Aims and scope Submit manuscript



Published health utility studies are increasingly cited in cost–utility analyses to inform reimbursement decision-making. However, there is limited guidance for investigators looking to systematically evaluate the methodological quality of health utility studies or their applicability to decision contexts.


To describe how health utility concepts are reflected in tools intended for use with the health economic literature, particularly with respect to the evaluation of methodological quality and context applicability.


We reviewed the critical appraisal and reporting tools described in a 2012 report published by the Agency for Healthcare Research and Quality (AHRQ), supplemented with a keyword search of MEDLINE and EMBASE, to identify existing tools which include health utility constructs. From these tools, a list of relevant items was compiled and grouped into domain categories based on the methodological or applicability aspect they were directed toward.


Of the 24 tools we identified, 12 contained items relevant to the evaluation of health utilities. Sixty-five items were considered relevant to the evaluation of quality, while 44 were relevant to the evaluation of applicability. Items were arranged into four domains: health state descriptions; selection and description of respondents; elicitation and measurement methods; and other considerations.


As key inputs to cost–utility analyses, health utilities have the potential to significantly impact estimates of cost-effectiveness. Existing tools contain only general items related to the conduct or use of health utility studies. There is a need to develop tools that systematically evaluate the methodological quality and applicability of health utility studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others


  1. Klarman, H.E., Francis, J.O.S., Rosenthal, G.D.: Cost Effectiveness Analysis Applied to the Treatment of Chronic Renal Disease. Med. Care 6(1), 48–54 (1968)

    Article  Google Scholar 

  2. Bremner, K.E., Chong, C.A., Tomlinson, G., Alibhai, S.M., Krahn, M.D.: A review and meta-analysis of prostate cancer utilities. Med. Decis. Making 27(3), 288–298 (2007).

    Article  PubMed  Google Scholar 

  3. Sturza, J.: A review and meta-analysis of utility values for lung cancer. Med. Decis. Making 30(6), 685–693 (2010).

    Article  PubMed  Google Scholar 

  4. Paracha, N., Thuresson, P.O., Moreno, S.G., MacGilchrist, K.S.: Health state utility values in locally advanced and metastatic breast cancer by treatment line: a systematic review. Expert Rev. Pharmacoecon. Outcomes Res. 16(5), 549–559 (2016).

    Article  PubMed  Google Scholar 

  5. Peasgood, T., Ward, S.E., Brazier, J.: Health-state utility values in breast cancer. Expert Rev. Pharmacoecon. Outcomes Res. 10(5), 553–566 (2010).

    Article  PubMed  Google Scholar 

  6. Hao, Y., Wolfram, V., Cook, J.: A structured review of health utility measures and elicitation in advanced/metastatic breast cancer. Clin. Econ. Outcomes Res. CEOR 8, 293–303 (2016).

    Article  Google Scholar 

  7. Schiller-Fruhwirth, I.C., Jahn, B., Arvandi, M., Siebert, U.: Cost-effectiveness models in breast cancer screening in the general population: a systematic review. Appl. Health Econ. Health Policy 15(3), 333–351 (2017).

    Article  PubMed  Google Scholar 

  8. Carter, G.C., King, D.T., Hess, L.M., Mitchell, S.A., Taipale, K.L., Kiiskinen, U., Rajan, N., Novick, D., Liepa, A.M.: Health state utility values associated with advanced gastric, oesophageal, or gastro-oesophageal junction adenocarcinoma: a systematic review. J. Med. Econ. 18(11), 954–966 (2015).

    Article  PubMed  Google Scholar 

  9. Djalalov, S., Rabeneck, L., Tomlinson, G., Bremner, K.E., Hilsden, R., Hoch, J.S.: A review and meta-analysis of colorectal cancer utilities. Med. Decis. Making 34(6), 809–818 (2014).

    Article  PubMed  Google Scholar 

  10. Jeong, K., Cairns, J.: Systematic review of health state utility values for economic evaluation of colorectal cancer. Heal. Econ. Rev. 6(1), 36 (2016).

    Article  Google Scholar 

  11. Richardson, J., Khan, M.A., Iezzi, A., Maxwell, A.: Comparing and explaining differences in the magnitude, content, and sensitivity of utilities predicted by the EQ-5D, SF-6D, HUI 3, 15D, QWB, and AQoL-8D multiattribute utility instruments. Med. Decis. Making 35(3), 276–291 (2015).

    Article  PubMed  Google Scholar 

  12. Dolan, P.: Modeling valuations for EuroQol health states. Med. Care 35(11), 1095–1108 (1997).

    Article  CAS  PubMed  Google Scholar 

  13. Mulhern, B.J., Bansback, N., Norman, R., Brazier, J.: Valuing the SF-6Dv2 classification system in the United Kingdom using a discrete-choice experiment with duration. Med. Care 58(6), 566–573 (2020).

    Article  PubMed  Google Scholar 

  14. Galante, J., Augustovski, F., Colantonio, L., Bardach, A., Caporale, J., Marti, S.G., Kind, P.: Estimation and comparison of EQ-5D health states' utility weights for pneumoccocal and human papillomavirus diseases in Argentina, Chile, and the United Kingdom. Value Health 14(5, Supplement), S60–S64 (2011).

  15. Takemoto, M.L., Lopes da Silva, N., Ribeiro-Pereira, A.C., Schilithz, A.O., Suzuki, C.: Differences in utility scores obtained through Brazilian and UK value sets: a cross-sectional study. Health Qual. Life Outcomes 13, 119 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Pollard, C., Hartz, S., Leage, S.L., Paget, M.A., Cook, J., Enstone, A.: Elicitation of health state utilities associated with varying severities of flares in systemic lupus erythematosus. Health Qual. Life Outcomes 13, 66 (2015).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Brazier, J., Ara, R., Azzabi, I., Busschbach, J., Chevrou-Severac, H., Crawford, B., Cruz, L., Karnon, J., Lloyd, A., Paisley, S., Pickard, A.S.: Identification, review, and use of health state utilities in cost-effectiveness models: an ISPOR Good practices for outcomes research task force report. Value Health 22(3), 267–275 (2019).

    Article  PubMed  Google Scholar 

  18. Xie, F., Zoratti, M., Chan, K., Husereau, D., Krahn, M., Levine, O., Clifford, T., Schunemann, H., Guyatt, G.: Toward a centralized, systematic approach to the identification, appraisal, and use of health state utility values for reimbursement decision making: introducing the health utility book (HUB). Med. Decis. Making 39(4), 370–378 (2019).

    Article  PubMed  Google Scholar 

  19. Walker, D.G., Wilson, R.F., Sharma, R., Bridges, J., Niessen, L., Bass, E.B., Frick, K.: Best practices for conducting economic evaluations in health care: a systematic review of quality assessment tools. AHRQ Publication No. 12(13)-EHC132-EF. Agency for Healthcare Research and Quality (2012)

  20. Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A.H., Mauskopf, J., Loder, E.: Consolidated health economic evaluation reporting standards (CHEERS)–explanation and elaboration: a report of the ISPOR health economic evaluation publication guidelines good reporting practices task force. Value Health 16(2), 231–250 (2013).

    Article  PubMed  Google Scholar 

  21. Evers, S., Goossens, M., de Vet, H., van Tulder, M., Ament, A.: Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. Int. J. Technol. Assess. Health Care 21(2), 240–245 (2005)

    Article  Google Scholar 

  22. Nerich, V., Saing, S., Gamper, E.M., Holzner, B., Pivot, X., Viney, R., Kemmler, G.: Critical appraisal of health-state utility values used in breast cancer-related cost-utility analyses. Breast Cancer Res. Treat. (2017).

    Article  PubMed  Google Scholar 

  23. Chiou, C.F., Hay, J.W., Wallace, J.F., Bloom, B.S., Neumann, P.J., Sullivan, S.D., Yu, H.T., Keeler, E.B., Henning, J.M., Ofman, J.J.: Development and validation of a grading system for the quality of cost-effectiveness studies. Med. Care 41(1), 32–44 (2003).

    Article  PubMed  Google Scholar 

  24. Simoens, S.: Assessment of methodological quality of economic evaluations in belgian drug reimbursement applications. PLoS One 8(12), e85411 (2013).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Stalmeier, P.F., Goldstein, M.K., Holmes, A.M., Lenert, L., Miyamoto, J., Stiggelbout, A.M., Torrance, G.W., Tsevat, J.: What should be reported in a methods section on utility assessment? Med. Decis. Making 21(3), 200–207 (2001)

    Article  CAS  Google Scholar 

  26. Petrou, S., Rivero-Arias, O., Dakin, H., Longworth, L., Oppe, M., Froud, R., Gray, A.: Preferred reporting items for studies mapping onto preference-based outcome measures: the MAPS statement. Pharmacoeconomics 33(10), 985–991 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Xie, F., Pickard, A.S., Krabbe, P.F., Revicki, D., Viney, R., Devlin, N., Feeny, D.: A checklist for reporting valuation studies of multi-attribute utility-based instruments (CREATE). Pharmacoeconomics 33(8), 867–877 (2015).

    Article  PubMed  PubMed Central  Google Scholar 

  28. Brazier, J., Deverill, M., Green, C.: A review of the use of health status measures in economic evaluation. J. Health Serv. Res. Policy 4(3), 174–184 (1999).

    Article  CAS  PubMed  Google Scholar 

  29. Drummond, M.F., Jefferson, T.O.: Guidelines for authors and peer reviewers of economic submissions to the BMJ. The BMJ Economic Evaluation Working Party. BMJ 313(7052), 275–283 (1996)

    Article  CAS  Google Scholar 

  30. Ungar, W.J., Santos, M.T.: The pediatric quality appraisal questionnaire: an instrument for evaluation of the pediatric health economics literature. Value Health 6(5), 584–594 (2003).

    Article  PubMed  Google Scholar 

  31. Clemens, K., Townsend, R., Luscombe, F., Mauskopf, J., Osterhaus, J., Bobula, J.: Methodological and conduct principles for pharmacoeconomic research. Pharmaceutical Research and Manufacturers of America. PharmacoEconomics 8(2), 169–174 (1995)

    Article  CAS  Google Scholar 

  32. Adams, M.E., McCall, N.T., Gray, D.T., Orza, M.J., Chalmers, T.C.: Economic analysis in randomized control trials. Med. Care 30(3), 231–243 (1992)

    Article  CAS  Google Scholar 

  33. Gerard, K.: Cost-utility in practice: a policy maker’s guide to the state of the art. Health Policy 21(3), 249–279 (1992)

    Article  CAS  Google Scholar 

  34. Sacristan, J.A., Soto, J., Galende, I.: Evaluation of pharmacoeconomic studies: utilization of a checklist. Ann. Pharmacother. 27(9), 1126–1133 (1993).

    Article  CAS  PubMed  Google Scholar 

  35. Drummond, M., Manca, A., Sculpher, M.: Increasing the generalizability of economic evaluations: recommendations for the design, analysis, and reporting of studies. Int. J. Technol. Assess. Health Care 21(2), 165–171 (2005)

    Article  Google Scholar 

  36. Ramsey, S., Willke, R., Briggs, A., Brown, R., Buxton, M., Chawla, A., Cook, J., Glick, H., Liljas, B., Petitti, D., Reed, S.: Good research practices for cost-effectiveness analysis alongside clinical trials: the ISPOR RCT-CEA Task Force report. Value Health 8(5), 521–533 (2005).

    Article  PubMed  Google Scholar 

  37. Goetghebeur, M.M., Wagner, M., Khoury, H., Levitt, R.J., Erickson, L.J., Rindress, D.: Evidence and value: Impact on decisionMaking–the EVIDEM framework and potential applications. BMC Health Serv. Res. 8, 270 (2008).

    Article  PubMed  PubMed Central  Google Scholar 

  38. Davis, J.C., Robertson, M.C., Comans, T., Scuffham, P.A.: Guidelines for conducting and reporting economic evaluation of fall prevention strategies. Osteoporos Int. 22(9), 2449–2459 (2011).

    Article  CAS  PubMed  Google Scholar 

  39. Vintzileos, A.M., Beazoglou, T.: Design, execution, interpretation, and reporting of economic evaluation studies in obstetrics. Am. J. Obstet. Gynecol. 191(4), 1070–1076 (2004).

    Article  PubMed  Google Scholar 

  40. Grutters, J.P., Seferina, S.C., Tjan-Heijnen, V.C., van Kampen, R.J., Goettsch, W.G., Joore, M.A.: Bridging trial and decision: a checklist to frame health technology assessments for resource allocation decisions. Value Health 14(5), 777–784 (2011).

    Article  PubMed  Google Scholar 

  41. Russell, L.B., Gold, M.R., Siegel, J.E., Daniels, N., Weinstein, M.C.: The role of cost-effectiveness analysis in health and medicine. Panel on cost-effectiveness in health and medicine. Jama 276(14), 1172–1177 (1996)

    Article  CAS  Google Scholar 

  42. Siegel, J.E., Weinstein, M.C., Russell, L.B., Gold, M.R.: Recommendations for reporting cost-effectiveness analyses. Panel on cost-effectiveness in health and medicine. Jama 276(16), 1339–1341 (1996)

    Article  CAS  Google Scholar 

  43. Weinstein, M.C., Siegel, J.E., Gold, M.R., Kamlet, M.S., Russell, L.B.: Recommendations of the panel on cost-effectiveness in health and medicine. JAMA 276(15), 1253–1258 (1996)

    Article  CAS  Google Scholar 

  44. Sanders, G.D., Neumann, P.J., Basu, A., Brock, D.W., Feeny, D., Krahn, M., Kuntz, K.M., Meltzer, D.O., Owens, D.K., Prosser, L.A., Salomon, J.A., Sculpher, M.J., Trikalinos, T.A., Russell, L.B., Siegel, J.E., Ganiats, T.G.: Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA 316(10), 1093–1103 (2016).

    Article  PubMed  Google Scholar 

  45. Higgins, J.P.T., Altman, D.G., Gøtzsche, P.C., Jüni, P., Moher, D., Oxman, A.D., Savović, J., Schulz, K.F., Weeks, L., Sterne, J.A.C.: The Cochrane Collaboration’s tool for assessing risk of bias in randomized trials. BMJ 343 (2011).

  46. Mokkink, L.B., Terwee, C.B., Patrick, D.L., Alonso, J., Stratford, P.W., Knol, D.L., Bouter, L.M., de Vet, H.C.: The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Quality Life Res. 19(4), 539–549 (2010).

    Article  Google Scholar 

Download references


No funding was received for this research.

Author information

Authors and Affiliations



Corresponding author

Correspondence to Feng Xie.

Ethics declarations

Conflicts of interest

None to declare.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 59 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zoratti, M.J., Pickard, A.S., Stalmeier, P.F.M. et al. Evaluating the conduct and application of health utility studies: a review of critical appraisal tools and reporting checklists. Eur J Health Econ 22, 723–733 (2021).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: