Health Care Management Science

, Volume 22, Issue 4, pp 727–755 | Cite as

A Multi-Fidelity Rollout Algorithm for Dynamic Resource Allocation in Population Disease Management

  • Ting-Yu Ho
  • Shan LiuEmail author
  • Zelda B. Zabinsky


Dynamic resource allocation for prevention, screening, and treatment interventions in population disease management has received much attention in recent years due to excessive healthcare costs. In this paper, our goal is to design a model and an efficient algorithm to optimize sequential intervention policies under resource constraints to improve population health outcomes. We consider a discrete-time finite-horizon budget allocation problem with disease progression within a closed birth-cohort population. To address the computational challenges associated with large-state and multiple-period dynamic decision-making problems, we propose a low-fidelity approximation that preserves the population dynamics under a stationary policy. To improve the healthcare interventions in terms of population health outcomes, we then embed the low-fidelity approximation into a high-fidelity optimization model to efficiently identify a good non-stationary sequential intervention policy. Our approach is illustrated by a numerical example of screening and treatment policy implementation for chronic hepatitis C virus (HCV) infection over a budget planning period. We numerically compare our Multi-Fidelity Rollout Algorithm (MF-RA) to a grid search approach and demonstrate the similarity of sequential policy trends and closeness of overall health outcomes measured by quality-adjusted life-years (QALYs) and the total number of individuals that undergo screening and treatment for different annual budgets and birth-cohorts. We also show how our approach scales well to problems with high dimensionality due to many decision periods by studying time to elimination of HCV.


Screening and treatment interventions Dynamic programming Markov processes Simulation Rollout algorithm Hepatitis C 


Funding Information

This study received no funding.

Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.


  1. 1.
    Alagoz O, Hsu H, Schaefer AJ, Roberts MS (2010) Markov decision processes: A tool for sequential decision making under uncertainty. Med Dec Making 30(4):474–483CrossRefGoogle Scholar
  2. 2.
    Alagoz O, Maillart LM, Schaefer AJ, Roberts MS (2004) The optimal timing of living-donor liver transplantation. Manag Sci 50(10):1420–1430CrossRefGoogle Scholar
  3. 3.
    Alagoz O, Maillart LM, Schaefer AJ, Roberts MS (2007) Determining the acceptance of cadaveric livers using an implicit model of the waiting list. Oper Res 55(1):24–36CrossRefGoogle Scholar
  4. 4.
    Alistar SS, Long EF, Brandeau ML, Beck EJ (2014) HIV epidemic control-a model for optimal allocation of prevention and treatment resources. Health Care Management Science 17(2):162–181CrossRefGoogle Scholar
  5. 5.
    Alter MJ, Kruszon-Moran D, Nainan OV, McQuillan GM, Gao F, Moyer LA, Kaslow RA, Margolis HS (1999) The prevalence of hepatitis C virus infection in the United States, 1988 through 1994. N Engl J Med 341(8):556–562CrossRefGoogle Scholar
  6. 6.
    Ayer T, Alagoz O, Stout NK (2012) OR forum-a POMDP approach to personalize mammography screening decisions. Oper Res 60(5):1019–1034CrossRefGoogle Scholar
  7. 7.
    Beekman AT, Geerlings SW, Deeg DJ, Smit JH, Schoevers RS, de Beurs E, Braam AW, Penninx BW, van Tilburg W (2013) The natural history of late-life depression: a 6-year prospective study in the community. Arch Gen Psychiatr 59(7):605–611CrossRefGoogle Scholar
  8. 8.
    Bennett S (2013) In $15 billion hepatitis race, drugs move quickly from revolutionary to outdated. HepCAsia. Accessed April 1, 2018.
  9. 9.
    Bertsekas DP (1995) Dynamic programming and optimal control, volume 1. Athena scientific BelmontGoogle Scholar
  10. 10.
    Bertsekas DP (2013) Rollout algorithms for discrete optimization: A survey. In: Handbook of Combinatorial Optimization, pp 2989–3013. SpringerGoogle Scholar
  11. 11.
    Bertsekas DP, Castanon DA (1998) Rollout algorithms for stochastic scheduling problems. In: Decision and Control, 1998. Proceedings of the 37th IEEE Conference on, volume 2, pp 2143–2148. IEEEGoogle Scholar
  12. 12.
    Bertsekas DP, Tsitsiklis JN, Wu C (1997) Rollout algorithms for combinatorial optimization. J Heuristics 3(3):245–262CrossRefGoogle Scholar
  13. 13.
    Bertsimas D, Farias VF, Trichakis N (2013) Fairness, efficiency, and flexibility in organ allocation for kidney transplantation. Oper Res 61(1):73–87CrossRefGoogle Scholar
  14. 14.
    Bertsimas DP, Demir R (2002) An approximate dynamic programming approach to multidimensional knapsack problems. Manag Sci 48(4):550–565CrossRefGoogle Scholar
  15. 15.
    Brandeau ML, Sainfort F, Pierskalla WP (2004) Operations research and health care: a handbook of methods and applications. In: International series in operations research and management science. Springer, New YorkGoogle Scholar
  16. 16.
    Buckley GJ, Strom BL (2017) A national strategy for the elimination of viral hepatitis emphasizes prevention, screening, and universal treatment of hepatitis C. Ann Intern Med 166(12):895–896CrossRefGoogle Scholar
  17. 17.
    Centers for Disease Control and Prevention (2018) National Health and Nutrition Examination Survey (NHANES), 1999-2010. Accessed April 1, 2018,
  18. 18.
    Chak E, Talal AH, Sherman KE, Schiff ER, Saab S (2011) Hepatitis C virus infection in USA: An estimate of true prevalence. Liver Int 31(8):1090–1101CrossRefGoogle Scholar
  19. 19.
    Chang HS, Fu MC, Hu J, Marcus SI (2007) Simulation-based algorithms for Markov decision processes. Springer, New YorkCrossRefGoogle Scholar
  20. 20.
    Chen Q, Ayer T, Chhatwal J (2017) Optimal M-switch surveillance policies for liver cancer in hepatitis C-infected populationGoogle Scholar
  21. 21.
    Chhatwal J, Kanwal F, Roberts MS, Dunn MA (2015) Cost-effectiveness and budget impact of hepatitis C virus treatment with sofosbuvir and ledipasvir in the United States. Ann Intern Med 162(6):397–406CrossRefGoogle Scholar
  22. 22.
    Denniston MM, Klevens RM, McQuillan GM, Jiles RB (2012) Awareness of infection, knowledge of hepatitis C, and medical follow-up among individuals testing positive for hepatitis C: National health and nutrition examination survey 2001-2008. Hepatology 55(6):1652–1661CrossRefGoogle Scholar
  23. 23.
    Denton BT (2013) Handbook of healthcare operations management: methods and applications. Springer, New YorkCrossRefGoogle Scholar
  24. 24.
    Deo S, Iravani S, Jiang T, Smilowitz K, Samuelson S (2013) Improving health outcomes through better capacity allocation in a community-based chronic care model. Oper Res 61(6):1277–1294CrossRefGoogle Scholar
  25. 25.
    Deo S, Rajaram K, Rath S, Karmarkar US, Goetz MB (2015) Planning for HIV screening, testing, and care at the veterans health administration. Oper Res 63(2):287–304CrossRefGoogle Scholar
  26. 26.
    Deo S, Sohoni M (2015) Optimal decentralization of early infant diagnosis of HIV in resource-limited settings. Manufacturing & Service Operations Management 17(2):191–207CrossRefGoogle Scholar
  27. 27.
    El-Kamary SS, Jhaveri R, Shardell MD (2011) All-cause, liver-related, and non-liver-related mortality among HCV-infected individuals in the general US population. Clin Infect Dis 53(2):150–157CrossRefGoogle Scholar
  28. 28.
    Erdelyi A, Topaloglu H (2009) Computing protection level policies for dynamic capacity allocation problems by using stochastic approximation methods. IIE Trans 41(6):498–510CrossRefGoogle Scholar
  29. 29.
    Falade-Nwulia O, Suarez-Cuervo C, Nelson DR, Fried MW, Segal JB, Sulkowski MS (2017) Oral direct-acting agent therapy for hepatitis C virus infection: a systematic review. Ann Intern Med 166(9):637–648CrossRefGoogle Scholar
  30. 30.
    Gold MR (1996) Cost-effectiveness in health and medicine. Oxford University Press, New YorkGoogle Scholar
  31. 31.
    Gowda C, Lott S, Grigorian M, Carbonari DM, Saine ME, Trooskin S, Roy JA, Kostman JR, Urick P, Lo Re III V (2018) Absolute insurer denial of direct-acting antiviral therapy for hepatitis C: a national specialty pharmacy cohort study. In: Open Forum Infectious Diseases, (5(6):ofy076). US: Oxford University PressGoogle Scholar
  32. 32.
    He T, Li K, Roberts MS, Spaulding AC, Ayer T, Grefenstette JJ, Chhatwal J (2016) Prevention of hepatitis C by screening and treatment in US prisons. Ann Intern Med 164(2):84–92CrossRefGoogle Scholar
  33. 33.
    Helm JE, Lavieri MS, Van Oyen MP, Stein JD, Musch DC (2015) Dynamic forecasting and control algorithms of glaucoma progression for clinician decision support. Oper Res 63(5):979–999CrossRefGoogle Scholar
  34. 34.
    HHS (2016) About the U.S. opioid epidemic, Accessed April 1, 2018,
  35. 35.
    Huang E, Xu J, Zhang S, Chen CH (2015) Multi-fidelity model integration for engineering design. Procedia Computer Science 44:336–344CrossRefGoogle Scholar
  36. 36.
    Huang H, Zabinsky ZB, Li T, Liu S (2016) Analyzing hepatitis C screening and treatment strategies using probabilistic branch and bound. In: Roeder TMK, Frazier PI, Szechtman R, Zhou E (eds) Proceedings of the 2016 Winter Simulation Conference, Washington D.C., pp 2076–2086Google Scholar
  37. 37.
    Kabiri M, Jazwinski AB, Roberts MS, Schaefer AJ, Chhatwal J (2014) The changing burden of hepatitis C virus infection in the United States: model-based predictions. Ann Intern Med 161(3):170–180CrossRefGoogle Scholar
  38. 38.
    Kazemian P, Helm J, Lavieri M, Stein J, Van Oyen M (2016) Dynamic monitoring and control of irreversible chronic diseases with application to glaucoma. Kelley School of Business Research Paper No. 16-22. Accessed July 23, 2018,
  39. 39.
    Khademi A, Saure DR, Schaefer AJ, Braithwaite RS, Roberts MS (2015) The price of nonabandonment: HIV in resource-limited settings. Manufacturing & Service Operations Management 17(4):554–570CrossRefGoogle Scholar
  40. 40.
    Kramer JR, Kanwal F, Richardson P, Giordano TP, Petersen LA, El-Serag HB (2011) Importance of patient, provider, and facility predictors of hepatitis c virus treatment in veterans: a national study. The American Journal of Gastroenterology 106(3):483–491CrossRefGoogle Scholar
  41. 41.
    Larney S, Kopinski H, Beckwith CG, Zaller ND, Jarlais DD, Hagan H, Rich JD, Bergh BJ, Degenhardt L (2013) Incidence and prevalence of hepatitis C in prisons and other closed settings: Results of a systematic review and meta-analysis. Hepatology 58(4):1215–1224CrossRefGoogle Scholar
  42. 42.
    Lauffenburger JC, Mayer CL, Hawke RL, Brouwer KL, Fried MW, Farley JF (2014) Medication use and medical comorbidity in patients with chronic hepatitis c from a us commercial claims database: high utilization of drugs with interaction potential. European Journal of Gastroenterology & Hepatology 26(10): 1073CrossRefGoogle Scholar
  43. 43.
    Lee CP, Chertow GM, Zenios SA (2008) Optimal initiation and management of dialysis therapy. Oper Res 56(6):1428–1449CrossRefGoogle Scholar
  44. 44.
    Li H, Li Y, Lee LH, Chew EP, Pedrielli G, Chen CH (2015) Multi-objective multi-fidelity optimization with ordinal transformation and optimal sampling. In: Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD (eds) Proceedings of the 2015 Winter Simulation Conference, pp 3737–3748Google Scholar
  45. 45.
    Li Y, Huang H, Zabinsky ZB, Liu S (2017) Optimizing implementation of hepatitis C birth-cohort screening and treatment strategies: model-based projections. MDM Policy & Practice 2(1): 2381468316686795CrossRefGoogle Scholar
  46. 46.
    Linas BP, Barter DM, Morgan JR, Pho MT, Leff JA, Schackman BR, Horsburgh CR, Assoumou SA, Salomon JA, Weinstein MC (2015) The cost-effectiveness of sofosbuvir-based regimens for treatment of hepatitis C virus genotype 2 or 3 infection. Ann Intern Med 162(9):619–629CrossRefGoogle Scholar
  47. 47.
    Liu B, Koziel S, Zhang Q (2016) A multi-fidelity surrogate-model-assisted evolutionary algorithm for computationally expensive optimization problems. J Comput Sci 12:28–37CrossRefGoogle Scholar
  48. 48.
    Liu S, Brandeau ML, Goldhaber-Fiebert JD (2017) Optimizing patient treatment decisions in an era of rapid technological advances: the case of hepatitis C treatment. Health Care Management Science 20(1):16–32CrossRefGoogle Scholar
  49. 49.
    Liu S, Cipriano LE, Holodniy M, Goldhaber-Fiebert JD (2013) Cost-effectiveness analysis of risk-factor guided and birth-cohort screening for chronic hepatitis C infection in the United States. PLoS One 8(3):e58975CrossRefGoogle Scholar
  50. 50.
    Liu S, Cipriano LE, Holodniy M, Owens DK, Goldhaber-Fiebert JD (2012) New protease inhibitors for the treatment of chronic hepatitis C: A cost-effectiveness analysis. Ann Intern Med 156(4):279–290CrossRefGoogle Scholar
  51. 51.
    Liu S, Schwarzinger M, Carrat F, Goldhaber-Fiebert JD (2011) Cost effectiveness of fibrosis assessment prior to treatment for chronic hepatitis C patients. PLoS One 6(12):e26783CrossRefGoogle Scholar
  52. 52.
    Liu S, Watcha D, Holodniy M, Goldhaber-Fiebert JD (2014) Sofosbuvir-based treatment regimens for chronic, genotype 1 hepatitis C virus infection in US incarcerated populations: A cost-effectiveness analysis. Ann Intern Med 161(8):546–553CrossRefGoogle Scholar
  53. 53.
    Maillart LM, Ivy JS, Ransom S, Diehl K (2008) Assessing dynamic breast cancer screening policies. Oper Res 56(6):1411–1427CrossRefGoogle Scholar
  54. 54.
    Martin NK, Pitcher AB, Vickerman P, Vassall A, Hickman M (2011) Optimal control of hepatitis C antiviral treatment programme delivery for prevention amongst a population of injecting drug users. PLoS One 6 (8):e22309CrossRefGoogle Scholar
  55. 55.
    Mason JE, Denton BT, Shah ND, Smith SA (2014) Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. Eur J Oper Res 233(3):727–738CrossRefGoogle Scholar
  56. 56.
    Maxwell MS, Restrepo M, Henderson SG, Topaloglu H (2010) Approximate dynamic programming for ambulance redeployment. INFORMS J Comput 22(2):266–281CrossRefGoogle Scholar
  57. 57.
    Najafzadeh M, Andersson K, Shrank WH, Krumme AA, Matlin OS, Brennan T, Avorn J, Choudhry NK (2015) Cost-effectiveness of novel regimens for the treatment of hepatitis C virus. Ann Intern Med 162(6):407–419CrossRefGoogle Scholar
  58. 58.
    Ozcan YA (2005) Quantitative methods in health care management: techniques and applications. Jossey-Bass, San FranciscoGoogle Scholar
  59. 59.
    Patrick J, Puterman ML, Queyranne M (2008) Dynamic multipriority patient scheduling for a diagnostic resource. Oper Res 56(6):1507–1525CrossRefGoogle Scholar
  60. 60.
    Pollack A (2013) Hepatitis C, a silent killer, meets its match. The New York TimesGoogle Scholar
  61. 61.
    Powell WB (2007) Approximate dynamic programming: solving the curses of dimensionality. Wiley, New YorkCrossRefGoogle Scholar
  62. 62.
    Rauner MS, Gutjahr WJ, Heidenberger K, Wagner J, Pasia J (2010) Dynamic policy modeling for chronic diseases: Metaheuristic-based identification of Pareto-optimal screening strategies. Oper Res 58(5):1269–1286CrossRefGoogle Scholar
  63. 63.
    Rein DB, Smith BD, Wittenborn JS, Lesesne SB, Wagner LD, Roblin DW, Patel N, Ward JW, Weinbaum CM (2012) The cost-effectiveness of birth-cohort screening for hepatitis C antibody in US primary care settings. Ann Intern Med 156(4):263–270CrossRefGoogle Scholar
  64. 64.
    Salomon JA, Weinstein MC, Hammitt JK, Goldie SJ (2003) Cost-effectiveness of treatment for chronic hepatitis C infection in an evolving patient population. J Am Med Assoc 290(2):228– 237CrossRefGoogle Scholar
  65. 65.
    Sandıkçı B, Maillart LM, Schaefer AJ, Roberts MS (2013) Alleviating the patient’s price of privacy through a partially observable waiting list. Manag Sci 59(8):1836–1854CrossRefGoogle Scholar
  66. 66.
    Schaefer AJ, Bailey MD, Shechter SM, Roberts MS (2005) Modeling medical treatment using Markov decision processes. In: Operations Research and Health Care, pp 593–612. SpringerGoogle Scholar
  67. 67.
    Secomandi N (2001) A rollout policy for the vehicle routing problem with stochastic demands. Oper Res 49 (5):796–802CrossRefGoogle Scholar
  68. 68.
    Secomandi N (2003) Analysis of a rollout approach to sequencing problems with stochastic routing applications. J Heuristics 9(4):321–352CrossRefGoogle Scholar
  69. 69.
    Shechter SM, Bailey MD, Schaefer AJ, Roberts MS (2008) The optimal time to initiate HIV therapy under ordered health states. Oper Res 56(1):20–33CrossRefGoogle Scholar
  70. 70.
    Smith BD, Morgan RL, Beckett GA, Falck-Ytter Y, Holtzman D, Ward JW (2012) Hepatitis C virus testing of persons born during 1945–1965: Recommendations from the Centers for Disease Control and Prevention. Ann Intern Med 157(11):817–822CrossRefGoogle Scholar
  71. 71.
    Spaulding AC, Thomas DL (2012) Screening for HCV infection in jails. J Am Med Assoc 307(12):1259–1260CrossRefGoogle Scholar
  72. 72.
    Steimle LN, Denton BT (2017) Markov decision processes for screening and treatment of chronic diseases. In: Markov Decision Processes in Practice, pp 189–222. SpringerGoogle Scholar
  73. 73.
    Stepanova M, Kanwal F, El-Serag HB, Younossi ZM (2011) Insurance status and treatment candidacy of hepatitis c patients: Analysis of population-based data from the United States. Hepatology 53(3):737–745CrossRefGoogle Scholar
  74. 74.
    Trudeau K, Manser K, Behling M, Brown C, Niebler W, Budman S (2017) Natural history of prescription opioid misuse/abuse: a qualitative investigation. The Journal of Pain 18(4):S32–S33CrossRefGoogle Scholar
  75. 75.
    U.S. Census Bureau (2015) 2010 Census Summary File, Accessed April 1, 2018,
  76. 76.
    Moyer VA (2018) Screening for hepatitis C virus infection in adults: U.S. Preventive services task force recommendation statement. Ann Intern Med 159(5):349–357CrossRefGoogle Scholar
  77. 77.
    USPSTF (2018) U.S. preventive services task force: Screening for depression in adults. Accessed April 1, 2018,
  78. 78.
    VA (2017) State of care for veterans with chronic hepatitis C. Accessed April 1, 2018,, [accessed: 13 June 2017]
  79. 79.
    Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129(4):370–380CrossRefGoogle Scholar
  80. 80.
    Winston WL (2003) Introduction to probability models: operations research. Volume Two. Brooks/Cole-Thomson LearningGoogle Scholar
  81. 81.
    Xu J, Zhang S, Huang E, Chen VH, Lee LH, Celik N (2014) Efficient multi-fidelity simulation optimization. In: Tolk A, Diallo SY, Ryzhov IO, Yilmaz L, Buckley S, Miller JA (eds) Proceedings of the 2014 Winter Simulation Conference, pp 3940–3951Google Scholar
  82. 82.
    Zaric GS (2013) Operations research and health care policy. Springer, New YorkCrossRefGoogle Scholar
  83. 83.
    Zenios SA, Chertow GM, Wein LM (2000) Dynamic allocation of kidneys to candidates on the transplant waiting list. Oper Res 48(4):549–569CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Industrial and Systems EngineeringUniversity of WashingtonSeattleUSA

Personalised recommendations