Skip to main content

Advertisement

SpringerLink
Log in

Calibrating Natural History of Cancer Models in the Presence of Data Incompatibility: Problems and Solutions

  • Current Opinion
  • Published:
PharmacoEconomics Aims and scope Submit manuscript

Abstract

The calibration of cancer natural history models is often challenged by a lack of representative calibration targets, forcing modellers to rely on potentially incompatible datasets. Using a microsimulation colorectal cancer model as an example, the purposes of this paper are to (1) highlight the reasons for uncertainty in calibration targets, (2) illustrate practical and generalisable approaches for dealing with incompatibility in calibration targets, and (3) discuss the importance of future research in the area of incorporating uncertainty in calibration. The low quality of data and differences in populations, outcome definitions, and healthcare systems may result in incompatibility between the model and the data. Acknowledging reasons for data incompatibility allows assessment of the risk of incompatibility before calibrating the model. Only a few approaches are available to address data incompatibility, for instance addressing biases in calibration targets and their adjustment, relaxing the goodness-of-fit metric, and validation of the calibration targets to the data not used in the calibration. However, these approaches lack explicit comparison and validation, and so more research is needed to describe the nature and causes of indirect uncertainty (i.e. uncertainty that cannot be expressed in absolute quantitative forms) and identify methods for managing this uncertainty in healthcare modelling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Vanni T, Karnon J, Madan J, White RG, Edmunds WJ, Foss AM, et al. Calibrating models in economic evaluation: a seven-step approach. Pharmacoeconomics. 2011;29(1):35–49. https://doi.org/10.2165/11584600-000000000-00000.

    Article  PubMed  Google Scholar 

  2. Platt D. A comparison of economic agent-based model calibration methods. J Econ Dyn Control. 2020;113: 103859. https://doi.org/10.1016/j.jedc.2020.103859.

    Article  Google Scholar 

  3. Whyte S, Walsh C, Chilcott J. Bayesian calibration of a natural history model with application to a population model for colorectal cancer. Med Decis Making. 2011;31(4):625–41. https://doi.org/10.1177/0272989x10384738.

    Article  PubMed  Google Scholar 

  4. Stout NK, Knudsen AB, Kong CY, McMahon PM, Gazelle GS. Calibration methods used in cancer simulation models and suggested reporting guidelines. Pharmacoeconomics. 2009;27(7):533–45. https://doi.org/10.2165/11314830-000000000-00000.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Drummond M, Barbieri M, Cook J, Glick HA, Lis J, Malik F, et al. Transferability of economic evaluations across jurisdictions: ISPOR Good Research Practices Task Force report. Value Health. 2009;12(4):409–18. https://doi.org/10.1111/j.1524-4733.2008.00489.x.

    Article  PubMed  Google Scholar 

  6. Corro Ramos I, Hoogendoorn M, Rutten-van Mölken MPMH. How to address uncertainty in health economic discrete-event simulation models: an illustration for chronic obstructive pulmonary disease. Med Decis Making. 2020;40(5):619–32. https://doi.org/10.1177/0272989x20932145.

    Article  PubMed  PubMed Central  Google Scholar 

  7. D’Agostino McGowan L, Grantz KH, Murray E. Quantifying uncertainty in mechanistic models of infectious disease. Am J Epidemiol. 2021;190(7):1377–85. https://doi.org/10.1093/aje/kwab013.

    Article  PubMed  Google Scholar 

  8. Bilcke J, Beutels P, Brisson M, Jit M. Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide. Med Decis Making. 2011;31(4):675–92. https://doi.org/10.1177/0272989x11409240.

    Article  PubMed  Google Scholar 

  9. Degeling K, Ijzerman MJ, Koopman M, Koffijberg H. Accounting for parameter uncertainty in the definition of parametric distributions used to describe individual patient variation in health economic models. BMC Med Res Methodol. 2017;17(1):170. https://doi.org/10.1186/s12874-017-0437-y.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Briggs AH, Weinstein MC, Fenwick EA, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force–6. Value Health. 2012;15(6):835–42. https://doi.org/10.1016/j.jval.2012.04.014.

    Article  PubMed  Google Scholar 

  11. Alarid-Escudero F, MacLehose RF, Peralta Y, Kuntz KM, Enns EA. Nonidentifiability in model calibration and implications for medical decision making. Med Decis Making. 2018;38(7):810–21. https://doi.org/10.1177/0272989x18792283.

    Article  PubMed  PubMed Central  Google Scholar 

  12. Rutter CM, Ozik J, DeYoreo M, Collier N. Microsimulation model calibration using incremental mixture approximate Bayesian computation. Ann Appl Stat. 2019;13(4):2189–212. https://doi.org/10.1214/19-aoas1279.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Kong CY, McMahon PM, Gazelle GS. Calibration of disease simulation model using an engineering approach. Value Health. 2009;12(4):521–9. https://doi.org/10.1111/j.1524-4733.2008.00484.x.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Padilla LMK, Powell M, Kay M, Hullman J. Uncertain about uncertainty: how qualitative expressions of forecaster confidence impact decision-making with uncertainty visualizations. Front Psychol. 2021;11: 579267. https://doi.org/10.3389/fpsyg.2020.579267.

    Article  PubMed  PubMed Central  Google Scholar 

  15. Thomas C, Mandrik O, Whyte S. Development of the Microsimulation Model in Cancer of the Bowel (MiMiC-Bowel), an Individual Patient Simulation Model for Investigation of the Cost-effectiveness of Personalised Screening and Surveillance Strategies. 2020. Report No. https://eprints.whiterose.ac.uk/162743/. 1 April 2020.

  16. Mandrik OTC, Strong M, Whyte S. Calibration and Validation of the Microsimulation Model in Cancer of the Bowel (MiMiC-Bowel), an Individual Patient Simulation Model for Investigation of the Cost-effectiveness of Personalised Screening and Surveillance Strategies. Sheffield: School of Health and Related Research, University of Sheffield, 2021. https://eprints.whiterose.ac.uk/171343/.

  17. Brenner H, Altenhofen L, Hoffmeister M. Sex, age, and birth cohort effects in colorectal neoplasms: a cohort analysis. Ann Intern Med. 2010;152(11):697–703. https://doi.org/10.7326/0003-4819-152-11-201006010-00002.

    Article  PubMed  Google Scholar 

  18. Brenner H, Altenhofen L, Katalinic A, Lansdorp-Vogelaar I, Hoffmeister M. Sojourn time of preclinical colorectal cancer by sex and age: estimates from the German national screening colonoscopy database. Am J Epidemiol. 2011;174(10):1140–6. https://doi.org/10.1093/aje/kwr188.

    Article  PubMed  Google Scholar 

  19. Brenner H, Altenhofen L, Stock C, Hoffmeister M. Incidence of colorectal adenomas: birth cohort analysis among 4 million participants of screening colonoscopy. Cancer Epidemiol Biomark Prev. 2014;23(9):1920–7. https://doi.org/10.1158/1055-9965.Epi-14-0367.

    Article  Google Scholar 

  20. Brenner H, Jansen L, Ulrich A, Chang-Claude J, Hoffmeister M. Survival of patients with symptom- and screening-detected colorectal cancer. Oncotarget. 2016;7(28):44695–704. https://doi.org/10.18632/oncotarget.9412.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Altobelli E, D’Aloisio F, Angeletti PM. Colorectal cancer screening in countries of European Council outside of the EU-28. World J Gastroenterol. 2016;22(20):4946–57. https://doi.org/10.3748/wjg.v22.i20.4946.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Incidence numbers of Colorectal Cancer for patients diagnosed between 1996 and 2004 in England, by stage. In: Registries UAoC, editor. 2009.

  23. Kim SH, Shin DW, Kim SY, Yang HK, Nam E, Jho HJ, et al. Terminal versus advanced cancer: do the general population and health care professionals share a common language? Cancer Res Treat. 2016;48(2):759–67. https://doi.org/10.4143/crt.2015.124.

    Article  PubMed  Google Scholar 

  24. Mandrik O, Tolma E, Zielonke N, Meheus F, Ordóñez-Reyes C, Severens JL, et al. Systematic reviews as a “lens of evidence”: determinants of participation in breast cancer screening. J Med Screen. 2020. https://doi.org/10.1177/0969141320930743.

    Article  PubMed  PubMed Central  Google Scholar 

  25. Walters S, Maringe C, Butler J, Brierley JD, Rachet B, Coleman MP. Comparability of stage data in cancer registries in six countries: lessons from the International Cancer Benchmarking Partnership. Int J Cancer. 2013;132(3):676–85. https://doi.org/10.1002/ijc.27651.

    Article  CAS  PubMed  Google Scholar 

  26. Atkin W, Wooldrage K, Parkin DM, Kralj-Hans I, MacRae E, Shah U, et al. Long term effects of once-only flexible sigmoidoscopy screening after 17 years of follow-up: the UK Flexible Sigmoidoscopy Screening randomised controlled trial. Lancet. 2017;389(10076):1299–311. https://doi.org/10.1016/s0140-6736(17)30396-3.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Winawer SJ, Zauber AG, Fletcher RH, Stillman JS, O’Brien MJ, Levin B, et al. Guidelines for colonoscopy surveillance after polypectomy: a consensus update by the US Multi-Society Task Force on Colorectal Cancer and the American Cancer Society. Gastroenterology. 2006;130(6):1872–85. https://doi.org/10.1053/j.gastro.2006.03.012.

    Article  PubMed  Google Scholar 

  28. East JE, Atkin WS, Bateman AC, Clark SK, Dolwani S, Ket SN, et al. British Society of Gastroenterology position statement on serrated polyps in the colon and rectum. Gut. 2017;66(7):1181–96. https://doi.org/10.1136/gutjnl-2017-314005.

    Article  CAS  PubMed  Google Scholar 

  29. Torre LA, Siegel RL, Ward EM, Jemal A. Global cancer incidence and mortality rates and trends–an update. Cancer Epidemiol Biomarkers Prev. 2016;25(1):16–27. https://doi.org/10.1158/1055-9965.Epi-15-0578.

    Article  PubMed  Google Scholar 

  30. Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2018;68(6):394–424. https://doi.org/10.3322/caac.21492.

    Article  PubMed  Google Scholar 

  31. Wild CP, Espina C, Bauld L, Bonanni B, Brenner H, Brown K, et al. Cancer prevention Europe. Mol Oncol. 2019;13(3):528–34. https://doi.org/10.1002/1878-0261.12455.

    Article  PubMed  PubMed Central  Google Scholar 

  32. Keum N, Giovannucci E. Global burden of colorectal cancer: emerging trends, risk factors and prevention strategies. Nat Rev Gastroenterol Hepatol. 2019;16(12):713–32. https://doi.org/10.1038/s41575-019-0189-8.

    Article  PubMed  Google Scholar 

  33. Cancer Registration Statistics, England [Internet]. 2005. https://webarchive.nationalarchives.gov.uk/20160307140012/. https://cy.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/datasets/cancerregistrationstatisticscancerregistrationstatisticsengland. Accessed on 16 Feb 2020.

  34. Brown JP, Wooldrage K, Kralj-Hans I, Wright S, Cross AJ, Atkin WS. Effect of once-only flexible sigmoidoscopy screening on the outcomes of subsequent faecal occult blood test screening. J Med Screen. 2019;26(1):11–8. https://doi.org/10.1177/0969141318785654.

    Article  PubMed  Google Scholar 

  35. Siau K, Yew AC, Ishaq S, Jewes S, Shetty S, Brookes M, et al. Colonoscopy conversion after flexible sigmoidoscopy screening: results from the UK Bowel Scope Screening Programme. Colorectal Dis. 2018;20(6):502–8. https://doi.org/10.1111/codi.13982.

    Article  CAS  PubMed  Google Scholar 

  36. Jackson CH, Jit M, Sharples LD, De Angelis D. Calibration of complex models through Bayesian evidence synthesis: a demonstration and tutorial. Med Decis Making. 2015;35(2):148–61. https://doi.org/10.1177/0272989x13493143.

    Article  PubMed  Google Scholar 

  37. Menzies NA, Soeteman DI, Pandya A, Kim JJ. Bayesian methods for calibrating health policy models: a tutorial. Pharmacoeconomics. 2017;35(6):613–24. https://doi.org/10.1007/s40273-017-0494-4.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Bray F, Ren J-S, Masuyer E, Ferlay J. Global estimates of cancer prevalence for 27 sites in the adult population in 2008. Int J Cancer. 2013;132(5):1133–45. https://doi.org/10.1002/ijc.27711.

    Article  CAS  PubMed  Google Scholar 

  39. Bressler B, Paszat LF, Chen Z, Rothwell DM, Vinden C, Rabeneck L. Rates of new or missed colorectal cancers after colonoscopy and their risk factors: a population-based analysis. Gastroenterology. 2007;132(1):96–102. https://doi.org/10.1053/j.gastro.2006.10.027.

    Article  PubMed  Google Scholar 

  40. Gies A, Cuk K, Schrotz-King P, Brenner H. Direct comparison of diagnostic performance of 9 quantitative fecal immunochemical tests for colorectal cancer screening. Gastroenterology. 2018;154(1):93–104. https://doi.org/10.1053/j.gastro.2017.09.018.

    Article  PubMed  Google Scholar 

  41. Quyn AJ, Fraser CG, Stanners G, Carey FA, Rees CJ, Moores B, et al. Scottish Bowel Screening Programme colonoscopy quality—scope for improvement? Colorectal Dis. 2018;20(9):O277–83. https://doi.org/10.1111/codi.14281.

    Article  CAS  PubMed  Google Scholar 

  42. Bretthauer M, Kaminski MF, Loberg M, Zauber AG, Regula J, Kuipers EJ, et al. Population-based colonoscopy screening for colorectal cancer: a randomized clinical trial. JAMA Intern Med. 2016;176(7):894–902. https://doi.org/10.1001/jamainternmed.2016.0960.

    Article  PubMed  PubMed Central  Google Scholar 

  43. van Rijn AF, Dekker E, Kleibeuker JH. Screening the population for colorectal cancer: the background to a number of pilot studies in the Netherlands. Ned Tijdschr Geneeskd. 2006;150(50):2739–44 (Epub 2007/01/18).

    PubMed  Google Scholar 

  44. Martin-Lopez JE, Beltran-Calvo C, Rodriguez-Lopez R, Molina-Lopez T. Comparison of the accuracy of CT colonography and colonoscopy in the diagnosis of colorectal cancer. Colorectal Dis. 2014;16(3):O82–9. https://doi.org/10.1111/codi.12506.

    Article  CAS  PubMed  Google Scholar 

  45. Census. Office for National Statistics. [Internet]. Office for National Statistics. 2005. https://www.ons.gov.uk/search?q=2005+census.

  46. Hernán MA, Hernández-Díaz S, Robins JM. A structural approach to selection bias. Epidemiology. 2004;15(5):615–25. https://doi.org/10.1097/01.ede.0000135174.63482.43.

    Article  PubMed  Google Scholar 

  47. Sauboin CJ, Van Bellinghen L-A, Van De Velde N, Van Vlaenderen I. Potential public health impact of RTS, S malaria candidate vaccine in sub-Saharan Africa: a modelling study. Malar J. 2015;14:524. https://doi.org/10.1186/s12936-015-1046-z.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Ward ZJ, Yeh JM, Bhakta N, Frazier AL, Girardi F, Atun R. Global childhood cancer survival estimates and priority-setting: a simulation-based analysis. Lancet Oncol. 2019;20(7):972–83. https://doi.org/10.1016/S1470-2045(19)30273-6.

    Article  PubMed  Google Scholar 

  49. Turner RM, Lloyd-Jones M, Anumba DOC, Smith GCS, Spiegelhalter DJ, Squires H, et al. Routine antenatal anti-D prophylaxis in women who are Rh(D) negative: meta-analyses adjusted for differences in study design and quality. PLoS ONE. 2012;7(2):e30711-e. https://doi.org/10.1371/journal.pone.0030711.

    Article  CAS  Google Scholar 

  50. König C, Spoden C, Frey A. An optimized Bayesian hierarchical two-parameter logistic model for small-sample item calibration. Appl Psychol Meas. 2020;44(4):311–26. https://doi.org/10.1177/0146621619893786.

    Article  PubMed  Google Scholar 

  51. Karnon J, Vanni T. Calibrating models in economic evaluation: a comparison of alternative measures of goodness of fit, parameter search strategies and convergence criteria. Pharmacoeconomics. 2011;29(1):51–62. https://doi.org/10.2165/11584610-000000000-00000.

    Article  PubMed  Google Scholar 

  52. Kypraios T, Neal P, Prangle D. A tutorial introduction to Bayesian inference for stochastic epidemic models using Approximate Bayesian Computation. Math Biosci. 2017;287:42–53. https://doi.org/10.1016/j.mbs.2016.07.001.

    Article  PubMed  Google Scholar 

  53. Taylor DC, Pawar V, Kruzikas D, Gilmore KE, Pandya A, Iskandar R, et al. Methods of model calibration: observations from a mathematical model of cervical cancer. Pharmacoeconomics. 2010;28(11):995–1000. https://doi.org/10.2165/11538660-000000000-00000.

    Article  PubMed  Google Scholar 

  54. Hemming V, Burgman MA, Hanea AM, McBride MF, Wintle BC. A practical guide to structured expert elicitation using the IDEA protocol. Methods Ecol Evol. 2018;9(1):169–80. https://doi.org/10.1111/2041-210X.12857.

    Article  Google Scholar 

  55. Rudy DR, Zdon MJ. Update on colorectal cancer. Am Fam Physician. 2000;61(6):1759–70 ((73–4) Epub 2000/04/06).

    CAS  PubMed  Google Scholar 

  56. Castro I, Estevez P, Cubiella J, Hernandez V, Gonzalez-Mao C, Rivera C, et al. Diagnostic performance of fecal immunochemical test and sigmoidoscopy for advanced right-sided colorectal neoplasms. Dig Dis Sci. 2015;60(5):1424–32. https://doi.org/10.1007/s10620-014-3434-6.

    Article  CAS  PubMed  Google Scholar 

  57. Brenner H, Niedermaier T, Chen H. Strong subsite-specific variation in detecting advanced adenomas by fecal immunochemical testing for hemoglobin. Int J Cancer. 2017;140(9):2015–22. https://doi.org/10.1002/ijc.30629.

    Article  CAS  PubMed  Google Scholar 

  58. Niedermaier T, Tikk K, Gies A, Bieck S, Brenner H. Sensitivity of fecal immunochemical test for colorectal cancer detection differs according to stage and location. Clin Gastroenterol Hepatol. 2020. https://doi.org/10.1016/j.cgh.2020.01.025.

    Article  PubMed  Google Scholar 

  59. Afzali HH, Karnon J. Exploring structural uncertainty in model-based economic evaluations. Pharmacoeconomics. 2015;33(5):435–43. https://doi.org/10.1007/s40273-015-0256-0.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We are grateful to Prof. Mark Strong, Dr. Paul Tappenden, and Dr. Pete Dodd from the University of Sheffield for providing critical feedback on the manuscript and to Dr. Nick Menzies for providing clarifications on his published work.

Author information

Authors and Affiliations

  1. School of Health and Related Research, Health Economics and Decision Science, University of Sheffield, Regent Court, Sheffield, S1 4DA, UK

    Olena Mandrik, Chloe Thomas, Sophie Whyte & James Chilcott

Authors
  1. Olena Mandrik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chloe Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sophie Whyte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James Chilcott
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olena Mandrik.

Ethics declarations

Funding

Development of MiMiC-Bowel, its calibration and validation, was funded by the English National Screening Committee and Research England (Quality-related research funding to Support Evidence Based Policy Making distributed by the University of Sheffield). The views expressed are those of the authors and not necessarily those of the funding agencies.

Conflict of interest/competing interests

No conflict of interests to declare.

Ethical approval

Ethical approval was not required for this work as it is based only on published data.

Availability of data and material (data transparency)

The detailed description of the model used by the authors is reported in the referenced online reports. The model was populated using publicly available data.

Code availability

Not applicable.

Consent for publication

The authors jointly give the publisher the permission to publish the work.

Consent for participate

Not applicable.

Author contributions

OM wrote the draft manuscript. CT, SW, and JC revised the draft manuscript. OM, CT, and SW co-developed the model used as a worked example.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mandrik, O., Thomas, C., Whyte, S. et al. Calibrating Natural History of Cancer Models in the Presence of Data Incompatibility: Problems and Solutions. PharmacoEconomics 40, 359–366 (2022). https://doi.org/10.1007/s40273-021-01125-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40273-021-01125-3

Search

Navigation