Health Care Management Science

, Volume 22, Issue 1, pp 34–52 | Cite as

Probabilistic sensitivity analysis on Markov models with uncertain transition probabilities: an application in evaluating treatment decisions for type 2 diabetes

  • Yuanhui ZhangEmail author
  • Haipeng Wu
  • Brian T. Denton
  • James R. Wilson
  • Jennifer M. Lobo


Markov models are commonly used for decision-making studies in many application domains; however, there are no widely adopted methods for performing sensitivity analysis on such models with uncertain transition probability matrices (TPMs). This article describes two simulation-based approaches for conducting probabilistic sensitivity analysis on a given discrete-time, finite-horizon, finite-state Markov model using TPMs that are sampled over a specified uncertainty set according to a relevant probability distribution. The first approach assumes no prior knowledge of the probability distribution, and each row of a TPM is independently sampled from the uniform distribution on the row’s uncertainty set. The second approach involves random sampling from the (truncated) multivariate normal distribution of the TPM’s maximum likelihood estimators for its rows subject to the condition that each row has nonnegative elements and sums to one. The two sampling methods are easily implemented and have reasonable computation times. A case study illustrates the application of these methods to a medical decision-making problem involving the evaluation of treatment guidelines for glycemic control of patients with type 2 diabetes, where natural variation in a patient’s glycated hemoglobin (HbA1c) is modeled as a Markov chain, and the associated TPMs are subject to uncertainty.


Robustness and sensitivity analysis Markov model Transition probability matrices Medical decision-making Monte Carlo simulation 



This material is based upon work supported in part by the National Science Foundation through Grant Number CMMI 1462060. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.


  1. 1.
    American Diabetes Association (2015) Standards of medical care in diabetes–2015: summary of revisions. Diab Care 38:S4Google Scholar
  2. 2.
    Anderson T, Goodman L (1957) Statistical inference about Markov chains. Ann Math Stat 28:89–110CrossRefGoogle Scholar
  3. 3.
    Arias E (2007) 2011 United States life tables. Natl Vital Stat Rep 59Google Scholar
  4. 4.
    Bennett WL, Maruthur NM, Singh S, Segal JB, Wilson LM, Chatterjee R, Marinopoulos SS, Puhan MA, Ranasinghe P, Block L, Nicholson WK, Hutfless S, Bass EB, Bolen S (2011) Comparative effectiveness and safety of medications for type 2 diabetes: an update including new drugs and 2-drug combinations. Ann Intern Med 154:602–13CrossRefGoogle Scholar
  5. 5.
    Benson RV (1966) Euclidean geomtery and convexity. McGraw-Hill, New YorkGoogle Scholar
  6. 6.
    Bickel PJ, Doksum KA (2007) Mathematical statistics: Basic ideas and selected topics, 2nd edn. Pearson Prentice Hall, Upper Saddle River, NJGoogle Scholar
  7. 7.
    Billingsley P (1995) Probability and measure, 3rd edn. Wiley, New YorkGoogle Scholar
  8. 8.
    Briggs AH, Ades AE, Price MJ (2003) Probabilistic sensitivity analysis for decision trees with multiple branches: Use of the Dirichlet distribution in a Bayesian framework. Med Decis Making 23:341–50CrossRefGoogle Scholar
  9. 9.
    Brillinger DR (1981) Time series: Data analysis and theory. Holden-Day, San Francisco. Expanded editionGoogle Scholar
  10. 10.
    Centers for Disease Control and Prevention (2013) Age at diagnosis of diabetes among adult incident cases aged 18–79 years,
  11. 11.
    Chen Q, Ayer T, Chhatwal J (2017) Sensitivity analysis in sequential decision models. Med Decis Making 37:243–252CrossRefGoogle Scholar
  12. 12.
    Clarke PM, Gray AM, Briggs A, Farmer AJ, Fenn P, Stevens RJ, Matthews DR, Stratton IM, Holman RR (2004) A model to estimate the lifetime health outcomes of patients with type 2 diabetes: the United Kingdom prospective diabetes study (UKPDS) outcomes model (UKPDS No. 68). Diabetologia 47:1747–1759CrossRefGoogle Scholar
  13. 13.
    Craig A, Sendi PP (2002) Estimation of the transition matrix of a discrete-time Markov chain. Health Econ 11:33–42CrossRefGoogle Scholar
  14. 14.
    Devroye L (1986) Non-uniform random variate generation. Springer-Verlag, New YorkCrossRefGoogle Scholar
  15. 15.
    Dugundji J (1966) Topology. Allyn and Bacon, BostonGoogle Scholar
  16. 16.
    Geyer CJ (1992) Practical Markov chain Monte Carlo. Stat Sci 7:473–483CrossRefGoogle Scholar
  17. 17.
    Geyer CJ (2011) Introduction to Markov chain Monte Carlo. CRC Press, Boca RatonCrossRefGoogle Scholar
  18. 18.
    Goh J, Bayati M, Zenios SA, Singh S, Moore D (2015) Data uncertainty in markov chains: application to cost-effectiveness analyses of medical innovations. Working paper,
  19. 19.
    Gold MR, Siegel JE, Russell LB, Weinstein MC (1996) Cost-effectiveness in health and medicine. Oxford University Press, New YorkGoogle Scholar
  20. 20.
    Gold MR, Stevenson D, Fryback DG (2002) HALYs and QALYs and DALYs, Oh My: Similarities An differences in summary measures in population health. Ann Rev Public Health 23:115–134CrossRefGoogle Scholar
  21. 21.
    Gold RZ (1963) Tests auxiliary to χ 2 tests in a Markov chain. Ann Math Stat 34:56–74CrossRefGoogle Scholar
  22. 22.
    Hörmann W., Leydold J, Derflinger G (2004) Automatic nonuniform random variate generation. Springer-Verlag, BerlinCrossRefGoogle Scholar
  23. 23.
    Horn RA, Johnson CR (2013) Matrix analysis, 2nd edn. Cambridge University Press, New YorkGoogle Scholar
  24. 24.
    Karlin S, Taylor H (1975) A first course in stochastic processes, 2nd edn. Academic Press, New YorkGoogle Scholar
  25. 25.
    Kramer H, Cao G, Dugas L, Luke A, Cooper R, Durazo-Arvizu R (2010) Increasing BMI and waist circumference and prevalence of obesity among adults with type 2 diabetes: the national health and nutrition examination surveys. J Diab Compl 24:368–74CrossRefGoogle Scholar
  26. 26.
    Lada EK, Steiger NM, Wilson JR (2006) Performance evaluation of recent procedures for steady-state simulation analysis. IIE Trans 38:711–727CrossRefGoogle Scholar
  27. 27.
    Lada EK, Wilson JR (2006) A wavelet-based spectral procedure for steady-state simulation analysis. Eur J Oper Res 174:1769–1801CrossRefGoogle Scholar
  28. 28.
    Lada EK, Wilson JR (2008) Sbatch: a spaced batch means procedure for steady-state simulation analysis. J Simul 2:170–185CrossRefGoogle Scholar
  29. 29.
    Lada EK, Wilson JR, Steiger NM, Joines JA (2007) Performance of a wavelet-based spectral procedure for steady-state simulation analysis. INFORMS J Comput 19:150–160CrossRefGoogle Scholar
  30. 30.
    Mandelblatt JS, Stout NK, Schechter CB et al (2016) Collaborative modeling of the benefits and harms associated with different U.S. breast cancer screening strategies. Ann Intern Med 164:215–225CrossRefGoogle Scholar
  31. 31.
    Mannor S, Simester D, Sun P, Tsitsiklis JN (2007) Bias and variance approximation in value function estimates. Manag Sci 53:308–322CrossRefGoogle Scholar
  32. 32.
    Nathan DM, Buse JM, Davidson MB (2009) Medical management of hyperglycemia in type 2 diabetes: A consensus algorithm for the initiation and adjustment of therapy: A consensus statement of the American diabetes association and the European association of the study of diabetes. Diab Care 32:193–203CrossRefGoogle Scholar
  33. 33.
    Neuts MF (1973) Probability. Allyn and Bacon, BostonGoogle Scholar
  34. 34.
    Program NCE (2002) Third report of the national cholesterol education program (NCEP) expert panel on detection, evaluation, and treatment of high blood cholesterol in adults (adult treatment panel III) final report. Circulation 106:3143–3421CrossRefGoogle Scholar
  35. 35.
    Puterman ML (2005) Markov decision processes: discrete stochastic dynamic programming. Wiley, New YorkGoogle Scholar
  36. 36.
    Rockafellar RT (1970) Convex analysis. Princeton University Press, Princeton, NJCrossRefGoogle Scholar
  37. 37.
    Royden HL, Fitzpatrick PM (2010) Real analysis, 4th edn. Prentice Hall, New YorkGoogle Scholar
  38. 38.
    Shiryaev A (1996) Probability, 2nd edn. Springer, New YorkCrossRefGoogle Scholar
  39. 39.
    Smith RL (1984) Efficient Monte-Carlo procedures for generating points uniformly distributed over bounded regions. Oper Res 32:1296–1308CrossRefGoogle Scholar
  40. 40.
    Steiger NM, Lada EK, Wilson JR, Joines JA, Alexopoulos C, Goldsman D (2005) ASAP3: a batch means procedure for steady-state simulation output analysis. ACM Trans Model Comput Simul 15:39–73CrossRefGoogle Scholar
  41. 41.
    Steiger NM, Wilson JR (2002) An improved batch means procedure for simulation output analysis. Manag Sci 48:1569–1586CrossRefGoogle Scholar
  42. 42.
    Tafazzoli A, Steiger NM, Wilson JR (2011) N-Skart: a nonsequential Skewness- and autoregression-adjusted batch-means procedure for simulation analysis. IEEE Trans Autom Control 56:254–264CrossRefGoogle Scholar
  43. 43.
    Tafazzoli A, Wilson J (2011) Skart: a skewness- and autoregression-adjusted batch-means procedure for simulation analysis. IIE Trans 43:110–128CrossRefGoogle Scholar
  44. 44.
    Tafazzoli A, Wilson JR, Lada EK, Steiger NM (2011) Performance of Skart: a skewness- and autoregression-adjusted batch-means procedure for simulation analysis. INFORMS J Comput 23:297–314CrossRefGoogle Scholar
  45. 45.
    The MathWorks Inc. (2013) MATLAB and Simulink version R2013aGoogle Scholar
  46. 46.
    Wright JC, Weinstein MC (1998) Gains in life expectancy from medical interventions — standardizing data on outcomes. England J Med 339:380–386CrossRefGoogle Scholar
  47. 47.
    Yeaw J, Lee WC, Aagren M, Christensen T (2012) Cost of self-monitoring of blood glucose in the United States among patients on an insulin regimen for diabetes. J Manag Care Pharm 18:21–32Google Scholar
  48. 48.
    Zhang Y, McCoy R, Mason JE, Smith SA, Shah ND, Denton BT (2014) Second-line agents for glycemic control for type 2 diabetes: Are newer agents better? Diab Care 37:1338–1345CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  1. 1.Operations Research Graduate ProgramNorth Carolina State UniversityRaleighUSA
  2. 2.Google Inc.Mountain ViewUSA
  3. 3.Department of Industrial and Operations EngineeringUniversity of MichiganAnn ArborUSA
  4. 4.Edward P. Fitts Department of Industrial and Systems EngineeringNorth Carolina State UniversityRaleighUSA
  5. 5.Department of Public Health SciencesUniversity of VirginiaCharlottesvilleUSA

Personalised recommendations