Advertisement

PharmacoEconomics

, Volume 29, Issue 1, pp 35–49 | Cite as

Calibrating Models in Economic Evaluation

A Seven-Step Approach
  • Tazio Vanni
  • Jonathan Karnon
  • Jason Madan
  • Richard G. White
  • W. John Edmunds
  • Anna M. Foss
  • Rosa Legood
Review Article Calibrating Models in Economic Evaluation

Abstract

In economic evaluation, mathematical models have a central role as a way of integrating all the relevant information about a disease and health interventions, in order to estimate costs and consequences over an extended time horizon. Models are based on scientific knowledge of disease (which is likely to change over time), simplifying assumptions and input parameters with different levels of uncertainty; therefore, it is sensible to explore the consistency of model predictions with observational data. Calibration is a useful tool for estimating uncertain parameters, as well as more accurately defining model uncertainty (particularly with respect to the representation of correlations between parameters). Calibration involves the comparison of model outputs (e.g. disease prevalence rates) with empirical data, leading to the identification of model parameter values that achieve a good fit.

This article provides guidance on the theoretical underpinnings of different calibration methods. The calibration process is divided into seven steps and different potential methods at each step are discussed, focusing on the particular features of disease models in economic evaluation. The seven steps are (i) Which parameters should be varied in the calibration process? (ii) Which calibration targets should be used? (iii) What measure of goodness of fit should be used? (iv) What parameter search strategy should be used? (v) What determines acceptable goodness-of-fit parameter sets (convergence criteria)? (vi) What determines the termination of the calibration process (stopping rule)? (vii) How should the model calibration results and economic parameters be integrated?

The lack of standards in calibrating disease models in economic evaluation can undermine the credibility of calibration methods. In order to avoid the scepticism regarding calibration, we ought to unify the way we approach the problems and report the methods used, and continue to investigate different methods.

Keywords

Economic Evaluation Markov Chain Monte Carlo Calibration Process Calibration Target Global Extremum 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

No sources of funding were used to conduct this study or prepare this manuscript. The authors have no conflicts of interest that are directly relevant to the content of this review.

For the invaluable advice provided, the authors thank Michael Pickles and Andrew Cox.

Supplementary material

40273_2012_29010035_MOESM1_ESM.xls (2.7 mb)
Supplementary material, approximately 2804 KB.

References

  1. 1.
    Drummond MFSM, Torrance GW, et al. Methods for the economic evaluation of health care programmes. 3rd ed. New York: Oxford University Press, 2005Google Scholar
  2. 2.
    Box G, Draper N. Empirical model-building and response surfaces. 1st ed. New York: John Wiley & Sons, 1987Google Scholar
  3. 3.
    Philips Z, Ginnelly L, Sculpher M, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess 2004 Sep; 8 (36): iii-iiv, ix-xi, 1–158PubMedGoogle Scholar
  4. 4.
    Cooper BS. Confronting models with data. J Hosp Infect 2007 Jun; 65 Suppl. 2: 88–92CrossRefPubMedGoogle Scholar
  5. 5.
    Weinstein MC. Recent developments in decision-analytic modelling for economic evaluation. Pharmacoeconomics 2006; 24 (11): 1043–53CrossRefPubMedGoogle Scholar
  6. 6.
    Van de Velde N, Brisson M, Boily M-C. Modeling human papillomavirus vaccine effectiveness: quantifying the impact of parameter uncertainty. Am J Epidemiol 2007 Apr 1; 165 (7): 762–75CrossRefPubMedGoogle Scholar
  7. 7.
    Jit M, Choi YH, Edmunds WJ. Economic evaluation of human papillomavirus vaccination in the United Kingdom. BMJ 2008 Jul 17; 337: a769CrossRefGoogle Scholar
  8. 8.
    Stout NK, Knudsen AB, Kong CY, et al. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics 2009; 27 (7): 533–45CrossRefPubMedGoogle Scholar
  9. 9.
    National Institute for Health and Clinical Excellence. Guide to the methods of technology appraisal. London: NICE, 2008 [online]. Available from URL: http://www.nice.org.uk/media/B52/A7/TAMethodsGuideUpdatedJune2008.pdf [Accessed 2009 Jun 23]Google Scholar
  10. 10.
    Canadian Agency for Drugs and Technologies in Health. Guidelines for the economic evaluation of health technologies: Canada. 3rd ed. Ottawa (ON): CADTH, 2006 [online]. Available from URL: http://www.cadth.ca/media/pdf/186_EconomicGuidelines_e.pdf [Accessed 2009 Jun 23]Google Scholar
  11. 11.
    Pharmaceutical Benefits Advisory Committee. 1995 guidelines for the pharmaceutical industry on preparation of submission to the Pharmaceutical Benefits Advisory Committee. Woden (SA): PBAC, 1995 [online]. Available from URL: http://www.health.gov.au/internet/main/publishing.nsf/Content/health-pbs-general-pubs-pharmpac-part1.htm [Accessed 2009 Jun 23]Google Scholar
  12. 12.
    Weinstein MC, O’Brien B, Hornberger J, et al. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices–Modeling Studies. Value Health 2003 Jan-Feb; 6 (1): 9–17CrossRefPubMedGoogle Scholar
  13. 13.
    Drummond MF, McGuire A. Economic evaluation in health care: merging theory with practice. 1st ed. New York: Oxford University Press, 2001Google Scholar
  14. 14.
    Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodnessof-fit, parameter search strategies and convergence criteria. Pharmacoeconomics 2011; 29 (1): 51–62CrossRefPubMedGoogle Scholar
  15. 15.
    Ades AE, Claxton K, Sculpher M. Evidence synthesis, parameter correlation and probabilistic sensitivity analysis. Health Econ 2006; 15: 373–81CrossRefPubMedGoogle Scholar
  16. 16.
    Kim JJ, Kuntz KM, Stout NK, et al. Multiparameter calibration of a natural history model of cervical cancer. Am J Epidemiol 2007 Jul 15; 166 (2): 137–50CrossRefPubMedGoogle Scholar
  17. 17.
    Kim L, Thompson S. Uncertainty and validation of health economic decision models. Health Econ 2010; 19 (1): 43–55PubMedGoogle Scholar
  18. 18.
    Popper K. Conjectures and refutations: the growth of scientific knowledge. London: Routledge, 1963Google Scholar
  19. 19.
    Foss AM, Watts CH, Vickerman P, et al. Could the CARESHAKTI intervention for injecting drug users be maintaining the low HIV prevalence in Dhaka, Bangladesh? Addiction 2007; 102 (1): 114–25CrossRefPubMedGoogle Scholar
  20. 20.
    Williams JR, Foss AM, Vickerman P, et al. What is the achievable effectiveness of the India AIDS initiative intervention among female sex workers under target coverage? Model projections from southern India. Sex Transm Infect 2006; 82 (5): 372–80Google Scholar
  21. 21.
    Goldhaber-Fiebert JD, Stout NK, Ortendahl J, et al. Modeling human papillomavirus and cervical cancer in the United States for analyses of screening and vaccination. Popul Health Metr 2007; 5 (1): 11CrossRefPubMedGoogle Scholar
  22. 22.
    Jit M, Gay N, Soldan K, et al. Estimating progression rates for human papillomavirus infection from epidemiological data. Med Decis Making 2010 Jan-Feb; 306 (1): 84–98CrossRefGoogle Scholar
  23. 23.
    Suárez E, Smith JS, Bosch FX, et al. Cost-effectiveness of vaccination against cervical cancer: a multi-regional analysis assessing the impact of vaccine characteristics and alternative vaccination scenarios. Vaccine 2008; 26 Suppl. 5: F29–45CrossRefGoogle Scholar
  24. 24.
    Berkhof J, Bruijne MCD, Zielinski GD, et al. Evaluation of cervical screening strategies with adjunct high-risk human papillomavirus testing for women with borderline or mild dyskaryosis. Int J Cancer 2006; 118 (7): 1759–68CrossRefPubMedGoogle Scholar
  25. 25.
    Welton NJ, Ades AE. Estimation of markov chain transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis, and model calibration. Med Decis Making 2005 Nov-Dec; 25 (6): 633–45CrossRefPubMedGoogle Scholar
  26. 26.
    Anderson R, Haas M, Shanahan M. The cost-effectiveness of cervical screening in Australia: what is the impact of screening at different intervals or over a different age range? Aust N Z J Public Health 2008; 32 (1): 43–52CrossRefPubMedGoogle Scholar
  27. 27.
    Mandelblatt J, Schechter CB, Lawrence W, et al. The SPECTRUM population model of the impact of screening and treatment on US breast cancer trends from 1975 to 2000: principles and practice of the model methods. J Natl Cancer Inst Monogr 2006; (36): 47–55CrossRefPubMedGoogle Scholar
  28. 28.
    Karnon J, Goyder E, Tappenden P, et al. A review and critique of modelling in prioritising and designing screening programmes. Health Technol Assess 2007 Dec; 11 (52): iii-iv, ix-xi, 1–145Google Scholar
  29. 29.
    Pickles M, Foss AM, Vickerman P, et al. Interim modelling analysis to validate reported increases in condom use and assess HIV infections averted among female sex workers and clients in southern India following a targeted HIV prevention programme. Sex Transm Infect 2010 Feb; 86 Suppl. 1: i33–43CrossRefGoogle Scholar
  30. 30.
    McMahon PM, Kong CY, Weinstein MC, et al. Adopting helical CT screening for lung cancer. Cancer 2008; 113 (12): 3440–9CrossRefPubMedGoogle Scholar
  31. 31.
    Kim JJ, Wright TC, Goldie SJ. Cost-effectiveness of human papillomavirus DNA testing in the United Kingdom, the Netherlands, France, and Italy. J Natl Cancer Inst 2005; 97 (12): 888–95CrossRefPubMedGoogle Scholar
  32. 32.
    Press WH, Teukolsky SA, Vetterling WT, et al. Numerical recipes in C: the art of scientific computing. 2nd ed. Cambridge (NY): Cambridge University Press, 1992Google Scholar
  33. 33.
    Trotter CL, Edmunds WJ. Modelling cost effectiveness of meningococcal serogroup C conjugate vaccination campaign in England and Wales. BMJ 2002; 324 (7341): 809CrossRefPubMedGoogle Scholar
  34. 34.
    Keeling M. Modeling infectious diseases in humans and animals. 1st ed. Princeton (NJ): Princeton University Press, 2007Google Scholar
  35. 35.
    Rao SS. Engineering optimization: theory and practice. 4th ed. Chichester: John Wiley & Sons, 2009Google Scholar
  36. 36.
    Freitas AA. A critical review of multi-objective optimization in data mining: a position paper. SIGKDD Explorations 2004; 6 (2): 77–86CrossRefGoogle Scholar
  37. 37.
    Goldie SJ, Kim JJ, Kobus K, et al. Cost-effectiveness of HPV 16, 18 vaccination in Brazil. Vaccine 2007; 25 (33): 6257–70CrossRefPubMedGoogle Scholar
  38. 38.
    Chung Yin K, Pamela MM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value Health 2009; 12 (4): 521–9CrossRefGoogle Scholar
  39. 39.
    Tappenden P, Chilcott J, Eggington S, et al. Option appraisal of population-based colorectal cancer screening programmes in England. Gut 2007 May 1; 56 (5): 677–84CrossRefPubMedGoogle Scholar
  40. 40.
    Wong S. Computational methods in physics and engineering. 2nd ed. Singapore: World Scientific Publishing Co. Pte. Ltd, 1997CrossRefGoogle Scholar
  41. 41.
    Forrester M, Pettitt A, Gibson G. Bayesian inference of hospital-acquired infectious diseases and control measures given imperfect surveillance data. Biostat 2007 Apr 1; 8 (2): 383–401CrossRefGoogle Scholar
  42. 42.
    Goldie SJ, Weinstein MC, Kuntz KM, et al. The costs, clinical benefits, and cost-effectiveness of screening for cervical cancer in HIV-infected women. Ann Intern Med 1999; 130: 97–107CrossRefPubMedGoogle Scholar
  43. 43.
    Dayhoff JE, De Leo JM. Artificial neural networks. Cancer 2001; 91 (S8): 1615–35CrossRefPubMedGoogle Scholar
  44. 44.
    Tan SYGL, van Oortmarssen GJ, Piersma N. Estimating parameters of a microsimulation model for breast cancer screening using the score function method. Ann Operat Res 2003; 119 (1): 43–61CrossRefGoogle Scholar
  45. 45.
    Vanni T, Legood R, Franco E, et al. Cost-effectiveness of strategies for managing women presenting atypical squamous cells of unknown significance in Brazil. London: London School of Hygiene and Tropical Medicine, 2008Google Scholar
  46. 46.
    Blower S, Dowlatabadi H. Sensitivity and uncertainty analysis of complex-models of disease transmission–an HIV model, as an example. Int Stat Rev 1994; 62 (2): 229–43CrossRefGoogle Scholar
  47. 47.
    Fylstra D, Lasdon L, Watson J, et al. Design and use of the Microsoft Excel Solver. INTERFACES 1998 Sep 1; 28 (5): 29–55CrossRefGoogle Scholar
  48. 48.
    Lasdon LS, Waren AD, Jain A, et al. Design and testing of a generalized reduced gradient code for nonlinear programming. ACM Trans Math Softw 1978; 4 (1): 34–50CrossRefGoogle Scholar
  49. 49.
    Gill PE, Murray W, Wright MH. Practical optimization. San Diego (CA): Academic Press, 1981Google Scholar
  50. 50.
    Taylor DCA. Methods of model calibration: a comparative approach. ISPOR 12th Annual International Meeting; 2007 May 18-23; Arlington (VA) [online]. Available from URL: http://www.ispor.org/awards/12meet/MC1-Taylor.pdf [Accessed 2009 Dec 12]Google Scholar
  51. 51.
    Vanni T, Legood R, White RG. Calibration of disease simulation model using an engineering approach. Value Health 2010; 13 (1): 157CrossRefPubMedGoogle Scholar
  52. 52.
    Fryback DG, Stout NK, Rosenberg MA, et al. The Wisconsin breast cancer epidemiology simulation model. J Natl Cancer Inst Monogr 2006; (36): 37–47CrossRefPubMedGoogle Scholar
  53. 53.
    Karnon J, Czoski-Murray C, Smith KJ, et al. A hybrid cohort individual sampling natural history model of agerelated macular degeneration: assessing the cost-effectiveness of screening using probabilistic calibration. Med Decis Making 2009 May 1; 29 (3): 304–16CrossRefPubMedGoogle Scholar
  54. 54.
    Karnon J, Campbell F, Czoski-Murray C. Model-based cost-effectiveness analysis of interventions aimed at preventing medication error at hospital admission (medicines reconciliation). J Eval Clin Pract 2009; 15 (2): 299–306CrossRefPubMedGoogle Scholar
  55. 55.
    Carlton J, Karnon J, Czoski-Murray C, et al. The clinical effectiveness and cost-effectiveness of screening programmes for amblyopia and strabismus in children up to the age of 4–5 years: a systematic review and economic evaluation. Health Technol Assess 2008 Jun; 12 (25): iii, xi-194Google Scholar
  56. 56.
    Karnon J, Jones R, Czoski-Murray C, et al. Cost-utility analysis of screening high-risk groups for anal cancer. J Public Health 2008; 30 (3): 293–304CrossRefGoogle Scholar
  57. 57.
    Karnon J, McIntosh A, Dean J, et al. A prospective hazard and improvement analytic approach to predicting the effectiveness of medication error interventions. Saf Sci 2007; 45 (4): 523–39CrossRefGoogle Scholar
  58. 58.
    Gelman A, Carlin JB, Stern HS, et al. Bayesian data analysis. 2nd ed. Boca Raton (FL): Chapman and Hall/CRC, 2004Google Scholar
  59. 59.
    Welton NJ, Ades AE. Estimations of Markov transition probabilities and rates from fully and partially observed data: uncertainty propagation, evidence synthesis and model calibration. Med Decis Making 2005; 25 (6): 633–45CrossRefPubMedGoogle Scholar
  60. 60.
    O’Hagan A, Buck CE, Daneshkhah A, et al. Uncertain judgements: eliciting expert probabilities. Chichester: Wiley, 2006CrossRefGoogle Scholar
  61. 61.
    De Angelis D, Sweeting M, Ades A, et al. An evidence synthesis approach to estimating hepatitis C prevalence in England and Wales. Stat Methods Med Res 2009; 18: 361–79CrossRefPubMedGoogle Scholar
  62. 62.
    Welton NJ, Ades AE. A model of toxoplasmosis incidence in the UK: evidence synthesis and consistency of evidence. J R Stat Soc Ser C Appl Stat 2005; 54 (2): 385–404CrossRefGoogle Scholar
  63. 63.
    Goubar A, Ades AE, Angelis DD, et al. Estimates of human immunodeficiency virus prevalence and proportion diagnosed based on Bayesian multiparameter synthesis of surveillance data. J R Stat Soc Ser A Stat Soc 2008; 171 (3): 541–80CrossRefGoogle Scholar
  64. 64.
    Morris S, Devlin N, Parkin D. Economic analysis in health care. 1st ed. Chichester: Wiley, 2007Google Scholar
  65. 65.
    Gold MR, Siegel JE, Russell LB, et al. Cost-effectiveness in health and medicine. 1st ed. New York: Oxford University Press, 1996Google Scholar
  66. 66.
    Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. 1st ed. Oxford: Oxford University Press, 2007Google Scholar

Copyright information

© Springer International Publishing AG 2011

Authors and Affiliations

  • Tazio Vanni
    • 1
    • 2
  • Jonathan Karnon
    • 3
  • Jason Madan
    • 4
  • Richard G. White
    • 2
  • W. John Edmunds
    • 2
  • Anna M. Foss
    • 2
  • Rosa Legood
    • 1
  1. 1.Health Services Research Unit, Department of Public Health and PolicyLondon School of Hygiene and Tropical MedicineLondonUK
  2. 2.Centre for Mathematical Modelling of Infectious Disease, London School of Hygiene and Tropical MedicineLondonUK
  3. 3.School of Population Health and Clinical Practice, University of AdelaideAdelaideAustralia
  4. 4.Academic Unit of Primary Health Care, University of BristolBristolUK

Personalised recommendations