Bulletin of Mathematical Biology

, Volume 79, Issue 4, pp 939–974 | Cite as

Bayesian Calibration, Validation and Uncertainty Quantification for Predictive Modelling of Tumour Growth: A Tutorial

  • Joe CollisEmail author
  • Anthony J. Connor
  • Marcin Paczkowski
  • Pavitra Kannan
  • Joe Pitt-Francis
  • Helen M. Byrne
  • Matthew E. Hubbard
Research Methods Article


In this work, we present a pedagogical tumour growth example, in which we apply calibration and validation techniques to an uncertain, Gompertzian model of tumour spheroid growth. The key contribution of this article is the discussion and application of these methods (that are not commonly employed in the field of cancer modelling) in the context of a simple model, whose deterministic analogue is widely known within the community. In the course of the example, we calibrate the model against experimental data that are subject to measurement errors, and then validate the resulting uncertain model predictions. We then analyse the sensitivity of the model predictions to the underlying measurement model. Finally, we propose an elementary learning approach for tuning a threshold parameter in the validation procedure in order to maximize predictive accuracy of our validated model.


Bayesian calibration Tumour growth Model validation 



J. Collis and M. E. Hubbard acknowledge the support of EPSRC Grant Number EP/K039342/1. This project has received funding from the European Unions Seventh Framework Programme for research, technological development and demonstration under Grant Agreement No. 600841.


  1. Achilleos A, Loizides C, Stylianopoulos T, Mitsis GD (2013) Multi-process dynamic modeling of tumor-specific evolution. In: 13th IEEE conference on bioinformatics and bioengineering, pp. 1–4. doi: 10.1109/BIBE.2013.6701614
  2. Achilleos A, Loizides C, Hadjiandreou M, Stylianopoulos T, Mitsis GD (2014) Multiprocess dynamic modeling of tumor evolution with Bayesian tumor-specific predictions. Ann Biomed Eng 42(5):1095–1111. doi: 10.1007/s10439-014-0975-y CrossRefGoogle Scholar
  3. Aguilar O, Allmaras M, Bangerth W, Tenorio L (2015) Statistics of parameter estimates: a concrete example. SIAM Rev 57(1):131–149MathSciNetCrossRefzbMATHGoogle Scholar
  4. Akaike H (1973) Information theory and an extension of the maximum likelihood principle. In: Petrov BN, Caski F (eds) Proceedings of the second international symposium on information theory. Akademiai Kiado, BudapestGoogle Scholar
  5. Allmaras M, Bangerth W, Linhart JM, Polanco J, Wang F, Wang K, Webster J, Zedler S (2013) Estimating parameters in physical models through Bayesian inversion: a complete example. SIAM Rev 55(1):149–167MathSciNetCrossRefzbMATHGoogle Scholar
  6. Andrieu C, Thoms J (2008) A tutorial on adaptive MCMC. Stat Comput 18(4):343–373. doi: 10.1007/s11222-008-9110-y MathSciNetCrossRefGoogle Scholar
  7. Arlot S, Celisse A (2010) A survey of cross-validation procedures for model selection. Stat Surv 4:40–79. doi: 10.1214/09-SS054 MathSciNetCrossRefzbMATHGoogle Scholar
  8. ASME (2006) ASME V&V 10-2006: guide for verification and validation in computational solid mechanics. American Society of Mechanical Engineers, New YorkGoogle Scholar
  9. ASME (2009) ASME V&V 20-2009: standard for verification and validation in computational fluid dynamics. American Society of Mechanical Engineers, New YorkGoogle Scholar
  10. ASME (2012) ASME V&V 10.1-2012: an illustration of the concepts of verification and validation in computational solid mechanics. American Society of Mechanical Engineers, New YorkGoogle Scholar
  11. Babuška I, Tempone R, Zouraris GE (2004) Galerkin finite element approximations of stochastic elliptic partial differential equations. SIAM J Numer Anal 42(2):800–825MathSciNetCrossRefzbMATHGoogle Scholar
  12. Babuška I, Nobile F, Tempone R (2007) A stochastic collocation method for elliptic partial differential equations with random input data. SIAM J Numer Anal 45(3):1005–1034MathSciNetCrossRefzbMATHGoogle Scholar
  13. Babuška I, Nobile F, Tempone R (2008) A systematic approach to model validation based on Bayesian updates and prediction related rejection criteria. Comput Methods Appl Mech Eng 197:2517–2539CrossRefzbMATHGoogle Scholar
  14. Baldock AL, Rockne RC, Boone AD, Neal ML, Hawkins-Daarud A, Corwin DM, Bridge CA, Guyman LA, Trister AD, Mrugala MM, Rockhill JK, Swanson KR (2013) From patient-specific mathematical neuro-oncology to precision medicine. Front Oncol 3:62CrossRefGoogle Scholar
  15. Bayarri MJ, Berger J, Paulo R, Sacks J, Cafeo JA, Cavendish JC, Lin C, Tu J (2007) A framework for validation of computer models. Technometrics 49:138–154MathSciNetCrossRefGoogle Scholar
  16. Bellman R, Åström KJ (1970) On structural identifiability. Math Biosci 7(3):329–339. ISSN 0025-5564. doi: 10.1016/0025-5564(70)90132-X
  17. Berger JO (1984) The robust Bayesian viewpoint (with discussion). In: Kadane JB (ed) Robustness of Bayesian analyses. North-Holland, Amsterdam, pp 63–144Google Scholar
  18. Chen W, Wong C, Vosburgh E, Levine AJ, Foran DJ, Xu EY (2014) High-throughput image analysis of tumor spheroids: a user-friendly software application to measure the size of spheroids automatically and accurately. J Vis Exp (89). doi: 10.3791/51639.
  19. Chib S, Greenberg E (1995) Understanding the Metropolis–Hastings algorithm. Am Stat 49(4):327–335Google Scholar
  20. Cobelli C, DiStefano JJ (1980) Parameter and structural identifiability concepts and ambiguities: a critical review and analysis. Am J Physiol Regul Integr Comp Physiol 239(1):R7–R24, ISSN 0363-6119Google Scholar
  21. Connor AJ (2016) Calibration, validation and uncertainty quantification. doi: 10.6084/m9.figshare.3406876. Supporting data and code
  22. Gammon K (2012) Mathematical modelling: forecasting cancer. Nature 491:66–67CrossRefGoogle Scholar
  23. Gelman A, Carlin JB, Stern HS, Dunson DB, Vehtari A, Rubin DB (2014a) Bayesian data analysis, 3rd edn. CRC Press, Boca RatonzbMATHGoogle Scholar
  24. Gelman A, Hwang J, Vehtari A (2014b) Understanding predictive information criteria for Bayesian models. Stat Comput 24(6):997–1016. doi: 10.1007/s11222-013-9416-2 MathSciNetCrossRefzbMATHGoogle Scholar
  25. Ghanem R (1999) Stochastic finite elements with multiple random non-Gaussian properties. ASCE J Eng Math 125(1):26–40CrossRefGoogle Scholar
  26. Ghanem R, Red-Horse J (1999) Propogation of probabilistic uncertainity in complex physical systems using a stochastic finite element approach. Phys D 133:137–144MathSciNetCrossRefzbMATHGoogle Scholar
  27. Ghanem RG, Spanos PD (1991) Stochastic finite elements: a spectral approach. Springer, BerlinCrossRefzbMATHGoogle Scholar
  28. Gilks WR, Richardson S, Spiegelhalter DJ (eds) (1996) Practical Markov chain Monte Carlo. Chapman and Hall, LondonzbMATHGoogle Scholar
  29. Gompertz G (1825) On the nature of the function expressive of the law of human mortality, and on the new mode of determining the value of life contingencies. Philos Trans R Soc Lond 115:513–585CrossRefGoogle Scholar
  30. Hastings WK (1970) Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57:97–109MathSciNetCrossRefzbMATHGoogle Scholar
  31. Hawkins-Daarud A, Prudhomme S, van der Zee KG, Oden JT (2013) Bayesian calibration, validation, and uncertainty quantification of diffuse interface models of tumour growth. J Math Biol 67:1457–1485MathSciNetCrossRefzbMATHGoogle Scholar
  32. Higdon D, Kennedy M, Cavendish JC, Cafeo JA, Ryne RD (2005) Combining field data and computer simulations for calibration and prediction. SIAM J Sci Comput 2(26):448–466MathSciNetzbMATHGoogle Scholar
  33. Insua DR, Ruggeri F (eds) (2000) Robust Bayesian analysis, volume 152 of lecture notes in statistics. Springer, New YorkGoogle Scholar
  34. Kaipio J, Somersalo E (2006) Statistical and computational inverse problems. Springer, New YorkzbMATHGoogle Scholar
  35. Kennedy MC, O’Hagan A (2001) Bayesian calibration of computer models. J R Stat Soc Ser B (Stat Methodol) 63(4):425–464. doi: 10.1111/1467-9868.00294 MathSciNetCrossRefzbMATHGoogle Scholar
  36. Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86. doi: 10.1214/aoms/1177729694 MathSciNetCrossRefzbMATHGoogle Scholar
  37. Laird AK (1964) Dynamics of tumor growth. Br J Cancer 18:490–502CrossRefGoogle Scholar
  38. Lopes HF, Tobias JL (2011) Confronting prior convictions: on issues of prior sensitivity and likelihood robustness in Bayesian analysis. Annu Rev Econ 3:107–131. doi: 10.1146/annurev-economics-111809-125134 CrossRefGoogle Scholar
  39. Massey FJ (1951) The Kolmogorov–Smirnov test for goodness of fit. J Am Stat Assoc 46(253):68–78CrossRefzbMATHGoogle Scholar
  40. Metropolis N, Ulam S (1949) The Monte Carlo method. J Am Stat Assoc 44:335–341CrossRefzbMATHGoogle Scholar
  41. Metropolis N, Rosenbluth AW, Rosenbluth MN, Teller AH, Teller E (1953) Equation of state calculations by fast computing machines. J Chem Phys 21:1087–1092CrossRefGoogle Scholar
  42. Miller LH (1956) Table of percentage points of Kolmogorov statistics. J Am Stat Assoc 51(273):111–121CrossRefzbMATHGoogle Scholar
  43. Najm HN (2009) Uncertainty quantification and polynomial chaos techniques in computational fluid dynamics. Annu Rev Fluid Mech 41:35–52MathSciNetCrossRefzbMATHGoogle Scholar
  44. Nobile F, Tempone R, Webster C (2008a) A sparse grid stochastic collocation method for elliptic partial differential equations with random input data. SIAM J Numer Anal 46(5):2309–2345MathSciNetCrossRefzbMATHGoogle Scholar
  45. Nobile F, Tempone R, Webster C (2008b) An anisotropic sparse grid stochastic collocation method for elliptic partial differential equations with random input data. SIAM J Numer Anal 46(5):2411–2442MathSciNetCrossRefzbMATHGoogle Scholar
  46. Norris JR (1998) Markov chains. Number no. 2008 in Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press. ISBN 9780521633963Google Scholar
  47. NRC (2012) Assessing the reliability of complex models: mathematical and statistical foundations of verification, validation, and uncertainty quantification. The National Academies Press. ISBN 9780309256346Google Scholar
  48. Oberkampf WL, Barone MF (2006) Measures of agreement between computation and experiment: validation metrics. J Comput Phys 217(1):5–36. doi: 10.1016/ CrossRefzbMATHGoogle Scholar
  49. Oberkampf WL, Roy CJ (2010) Verification and validation in scientific computing. Cambridge University Press, CambridgeCrossRefzbMATHGoogle Scholar
  50. Oberkampf WL, Trucano TG, Hirsch C (2004) Verification, validation, and predictive capability in computational engineering and physics. Appl Mech Rev 57(5):345–384. doi: 10.1115/1.1767847 CrossRefGoogle Scholar
  51. Oden JT, Moser R, Ghattas O (2010a) Computer predictions with quantified uncertainty, part i. SIAM News 43(9):1–3Google Scholar
  52. Oden JT, Moser R, Ghattas O (2010b) Computer predictions with quantified uncertainty, part ii. SIAM News 43(10):1–3Google Scholar
  53. Pathmanathan P, Gray RA (2013) Ensuring reliability of safety-critical clinical applications of computational cardiac models. Front Physiol. doi: 10.3389/fphys.2013.00358 Google Scholar
  54. Pericchi LR, Peréz ME (1994) Posterior robustness with more than one sampling model. J Stat Plan Inference 40:279–294MathSciNetCrossRefzbMATHGoogle Scholar
  55. Raue A, Kreutz C, Maiwald T, Bachmann J, Schilling M, Klingmüller U, Timmer J (2009) Structural and practical identifiability analysis of partially observed dynamical models by exploiting the profile likelihood. Bioinformatics 25(15):1923–1929. doi: 10.1093/bioinformatics/btp358 CrossRefGoogle Scholar
  56. Roache PJ (2009) Fundamentals of verification and validation, 2nd edn. Hermosa Publishers, New MexicoGoogle Scholar
  57. Savage N (2012) Modelling: computing cancer. Nature 491:62–63CrossRefGoogle Scholar
  58. Schervish MJ (1995) Theory of statistics. Springer series in statistics. Springer, New YorkzbMATHGoogle Scholar
  59. Simpson DP, Rue H, Martins TG, Riebler A, Sørbye SH (2014) Penalising model component complexity: a principled, practical approach to constructing priors. In arXiv preprint arXiv:1403.4630 [stat.ME]
  60. Spiegelhalter DJ, Best NG, Carlin BP, van der Linde A (2002) Bayesian measures of model complexity and fit (with discussion). J R Stat Soc B 64(4):583–639MathSciNetCrossRefzbMATHGoogle Scholar
  61. Tarantola A (2005) Inverse problem theory and methods for model parameter estimation. SIAM, PhiladelphiaCrossRefzbMATHGoogle Scholar
  62. Tatang MA, Pan W, Prinn RG, McRae GJ (1997) An efficient method for parametric uncertainity analysis of numerical geophysical models. J Geophys Res 102(D18):21925–21932CrossRefGoogle Scholar
  63. Watanabe S (2010) Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. J Mach Learn Res 11:3571–3594MathSciNetzbMATHGoogle Scholar
  64. Xiu D, Hesthaven JS (2005) High-order collocation methods for differential equations with random inputs. SIAM J Sci Comput 27(3):1118–1139MathSciNetCrossRefzbMATHGoogle Scholar
  65. Xiu D, Karniadakis GE (2002) Modeling uncertainty in steady state diffusion problems via generalized polynomial chaos. Comput Methods Appl Mech Eng 191:4927–4948MathSciNetCrossRefzbMATHGoogle Scholar
  66. Xiu D, Karniadakis GE (2003) Modeling uncertainty in flow simulations via generalized polynomial chaos. J Comput Phys 187:137–167MathSciNetCrossRefzbMATHGoogle Scholar
  67. Yankeelov TE, Atuegwu N, Hormuth D, WeisJA, Barnes SL, Miga MI, Rericha EC, Quaranta V (2013) Clinically relevant modeling of tumor growth and treatment response. Sci Transl Med 5(187). ISSN 1946-6234. doi: 10.1126/scitranslmed.3005686

Copyright information

© Society for Mathematical Biology 2017

Authors and Affiliations

  1. 1.School of Mathematical SciencesUniversity of NottinghamNottinghamUK
  2. 2.Mathematical InstituteUniversity of OxfordOxfordUK
  3. 3.Gray Institute for Radiation Oncology and BiologyUniversity of OxfordOxfordUK
  4. 4.Department of Computer ScienceUniversity of OxfordOxfordUK

Personalised recommendations