Skip to main content

Advertisement

Log in

A Multi-Fidelity Rollout Algorithm for Dynamic Resource Allocation in Population Disease Management

  • Published:
Health Care Management Science Aims and scope Submit manuscript

Abstract

Dynamic resource allocation for prevention, screening, and treatment interventions in population disease management has received much attention in recent years due to excessive healthcare costs. In this paper, our goal is to design a model and an efficient algorithm to optimize sequential intervention policies under resource constraints to improve population health outcomes. We consider a discrete-time finite-horizon budget allocation problem with disease progression within a closed birth-cohort population. To address the computational challenges associated with large-state and multiple-period dynamic decision-making problems, we propose a low-fidelity approximation that preserves the population dynamics under a stationary policy. To improve the healthcare interventions in terms of population health outcomes, we then embed the low-fidelity approximation into a high-fidelity optimization model to efficiently identify a good non-stationary sequential intervention policy. Our approach is illustrated by a numerical example of screening and treatment policy implementation for chronic hepatitis C virus (HCV) infection over a budget planning period. We numerically compare our Multi-Fidelity Rollout Algorithm (MF-RA) to a grid search approach and demonstrate the similarity of sequential policy trends and closeness of overall health outcomes measured by quality-adjusted life-years (QALYs) and the total number of individuals that undergo screening and treatment for different annual budgets and birth-cohorts. We also show how our approach scales well to problems with high dimensionality due to many decision periods by studying time to elimination of HCV.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Alagoz O, Hsu H, Schaefer AJ, Roberts MS (2010) Markov decision processes: A tool for sequential decision making under uncertainty. Med Dec Making 30(4):474–483

    Google Scholar 

  2. Alagoz O, Maillart LM, Schaefer AJ, Roberts MS (2004) The optimal timing of living-donor liver transplantation. Manag Sci 50(10):1420–1430

    Google Scholar 

  3. Alagoz O, Maillart LM, Schaefer AJ, Roberts MS (2007) Determining the acceptance of cadaveric livers using an implicit model of the waiting list. Oper Res 55(1):24–36

    Google Scholar 

  4. Alistar SS, Long EF, Brandeau ML, Beck EJ (2014) HIV epidemic control-a model for optimal allocation of prevention and treatment resources. Health Care Management Science 17(2):162–181

    Google Scholar 

  5. Alter MJ, Kruszon-Moran D, Nainan OV, McQuillan GM, Gao F, Moyer LA, Kaslow RA, Margolis HS (1999) The prevalence of hepatitis C virus infection in the United States, 1988 through 1994. N Engl J Med 341(8):556–562

    Google Scholar 

  6. Ayer T, Alagoz O, Stout NK (2012) OR forum-a POMDP approach to personalize mammography screening decisions. Oper Res 60(5):1019–1034

    Google Scholar 

  7. Beekman AT, Geerlings SW, Deeg DJ, Smit JH, Schoevers RS, de Beurs E, Braam AW, Penninx BW, van Tilburg W (2013) The natural history of late-life depression: a 6-year prospective study in the community. Arch Gen Psychiatr 59(7):605–611

    Google Scholar 

  8. Bennett S (2013) In $15 billion hepatitis race, drugs move quickly from revolutionary to outdated. HepCAsia. Accessed April 1, 2018. http://hepcasia.com/wp-content/uploads/2013/06/in.pdf

  9. Bertsekas DP (1995) Dynamic programming and optimal control, volume 1. Athena scientific Belmont

  10. Bertsekas DP (2013) Rollout algorithms for discrete optimization: A survey. In: Handbook of Combinatorial Optimization, pp 2989–3013. Springer

  11. Bertsekas DP, Castanon DA (1998) Rollout algorithms for stochastic scheduling problems. In: Decision and Control, 1998. Proceedings of the 37th IEEE Conference on, volume 2, pp 2143–2148. IEEE

  12. Bertsekas DP, Tsitsiklis JN, Wu C (1997) Rollout algorithms for combinatorial optimization. J Heuristics 3(3):245–262

    Google Scholar 

  13. Bertsimas D, Farias VF, Trichakis N (2013) Fairness, efficiency, and flexibility in organ allocation for kidney transplantation. Oper Res 61(1):73–87

    Google Scholar 

  14. Bertsimas DP, Demir R (2002) An approximate dynamic programming approach to multidimensional knapsack problems. Manag Sci 48(4):550–565

    Google Scholar 

  15. Brandeau ML, Sainfort F, Pierskalla WP (2004) Operations research and health care: a handbook of methods and applications. In: International series in operations research and management science. Springer, New York

  16. Buckley GJ, Strom BL (2017) A national strategy for the elimination of viral hepatitis emphasizes prevention, screening, and universal treatment of hepatitis C. Ann Intern Med 166(12):895–896

    Google Scholar 

  17. Centers for Disease Control and Prevention (2018) National Health and Nutrition Examination Survey (NHANES), 1999-2010. Accessed April 1, 2018, http://www.cdc.gov/nchs/nhanes/

  18. Chak E, Talal AH, Sherman KE, Schiff ER, Saab S (2011) Hepatitis C virus infection in USA: An estimate of true prevalence. Liver Int 31(8):1090–1101

    Google Scholar 

  19. Chang HS, Fu MC, Hu J, Marcus SI (2007) Simulation-based algorithms for Markov decision processes. Springer, New York

    Google Scholar 

  20. Chen Q, Ayer T, Chhatwal J (2017) Optimal M-switch surveillance policies for liver cancer in hepatitis C-infected population

  21. Chhatwal J, Kanwal F, Roberts MS, Dunn MA (2015) Cost-effectiveness and budget impact of hepatitis C virus treatment with sofosbuvir and ledipasvir in the United States. Ann Intern Med 162(6):397–406

    Google Scholar 

  22. Denniston MM, Klevens RM, McQuillan GM, Jiles RB (2012) Awareness of infection, knowledge of hepatitis C, and medical follow-up among individuals testing positive for hepatitis C: National health and nutrition examination survey 2001-2008. Hepatology 55(6):1652–1661

    Google Scholar 

  23. Denton BT (2013) Handbook of healthcare operations management: methods and applications. Springer, New York

    Google Scholar 

  24. Deo S, Iravani S, Jiang T, Smilowitz K, Samuelson S (2013) Improving health outcomes through better capacity allocation in a community-based chronic care model. Oper Res 61(6):1277–1294

    Google Scholar 

  25. Deo S, Rajaram K, Rath S, Karmarkar US, Goetz MB (2015) Planning for HIV screening, testing, and care at the veterans health administration. Oper Res 63(2):287–304

    Google Scholar 

  26. Deo S, Sohoni M (2015) Optimal decentralization of early infant diagnosis of HIV in resource-limited settings. Manufacturing & Service Operations Management 17(2):191–207

    Google Scholar 

  27. El-Kamary SS, Jhaveri R, Shardell MD (2011) All-cause, liver-related, and non-liver-related mortality among HCV-infected individuals in the general US population. Clin Infect Dis 53(2):150–157

    Google Scholar 

  28. Erdelyi A, Topaloglu H (2009) Computing protection level policies for dynamic capacity allocation problems by using stochastic approximation methods. IIE Trans 41(6):498–510

    Google Scholar 

  29. Falade-Nwulia O, Suarez-Cuervo C, Nelson DR, Fried MW, Segal JB, Sulkowski MS (2017) Oral direct-acting agent therapy for hepatitis C virus infection: a systematic review. Ann Intern Med 166(9):637–648

    Google Scholar 

  30. Gold MR (1996) Cost-effectiveness in health and medicine. Oxford University Press, New York

    Google Scholar 

  31. Gowda C, Lott S, Grigorian M, Carbonari DM, Saine ME, Trooskin S, Roy JA, Kostman JR, Urick P, Lo Re III V (2018) Absolute insurer denial of direct-acting antiviral therapy for hepatitis C: a national specialty pharmacy cohort study. In: Open Forum Infectious Diseases, (5(6):ofy076). US: Oxford University Press

  32. He T, Li K, Roberts MS, Spaulding AC, Ayer T, Grefenstette JJ, Chhatwal J (2016) Prevention of hepatitis C by screening and treatment in US prisons. Ann Intern Med 164(2):84–92

    Google Scholar 

  33. Helm JE, Lavieri MS, Van Oyen MP, Stein JD, Musch DC (2015) Dynamic forecasting and control algorithms of glaucoma progression for clinician decision support. Oper Res 63(5):979–999

    Google Scholar 

  34. HHS (2016) About the U.S. opioid epidemic, Accessed April 1, 2018, https://www.hhs.gov/opioids/about-the-epidemic/

  35. Huang E, Xu J, Zhang S, Chen CH (2015) Multi-fidelity model integration for engineering design. Procedia Computer Science 44:336–344

    Google Scholar 

  36. Huang H, Zabinsky ZB, Li T, Liu S (2016) Analyzing hepatitis C screening and treatment strategies using probabilistic branch and bound. In: Roeder TMK, Frazier PI, Szechtman R, Zhou E (eds) Proceedings of the 2016 Winter Simulation Conference, Washington D.C., pp 2076–2086

  37. Kabiri M, Jazwinski AB, Roberts MS, Schaefer AJ, Chhatwal J (2014) The changing burden of hepatitis C virus infection in the United States: model-based predictions. Ann Intern Med 161(3):170–180

    Google Scholar 

  38. Kazemian P, Helm J, Lavieri M, Stein J, Van Oyen M (2016) Dynamic monitoring and control of irreversible chronic diseases with application to glaucoma. Kelley School of Business Research Paper No. 16-22. Accessed July 23, 2018, https://ssrn.com/abstract=2733399

  39. Khademi A, Saure DR, Schaefer AJ, Braithwaite RS, Roberts MS (2015) The price of nonabandonment: HIV in resource-limited settings. Manufacturing & Service Operations Management 17(4):554–570

    Google Scholar 

  40. Kramer JR, Kanwal F, Richardson P, Giordano TP, Petersen LA, El-Serag HB (2011) Importance of patient, provider, and facility predictors of hepatitis c virus treatment in veterans: a national study. The American Journal of Gastroenterology 106(3):483–491

    Google Scholar 

  41. Larney S, Kopinski H, Beckwith CG, Zaller ND, Jarlais DD, Hagan H, Rich JD, Bergh BJ, Degenhardt L (2013) Incidence and prevalence of hepatitis C in prisons and other closed settings: Results of a systematic review and meta-analysis. Hepatology 58(4):1215–1224

    Google Scholar 

  42. Lauffenburger JC, Mayer CL, Hawke RL, Brouwer KL, Fried MW, Farley JF (2014) Medication use and medical comorbidity in patients with chronic hepatitis c from a us commercial claims database: high utilization of drugs with interaction potential. European Journal of Gastroenterology & Hepatology 26(10): 1073

    Google Scholar 

  43. Lee CP, Chertow GM, Zenios SA (2008) Optimal initiation and management of dialysis therapy. Oper Res 56(6):1428–1449

    Google Scholar 

  44. Li H, Li Y, Lee LH, Chew EP, Pedrielli G, Chen CH (2015) Multi-objective multi-fidelity optimization with ordinal transformation and optimal sampling. In: Yilmaz L, Chan WKV, Moon I, Roeder TMK, Macal C, Rossetti MD (eds) Proceedings of the 2015 Winter Simulation Conference, pp 3737–3748

  45. Li Y, Huang H, Zabinsky ZB, Liu S (2017) Optimizing implementation of hepatitis C birth-cohort screening and treatment strategies: model-based projections. MDM Policy & Practice 2(1): 2381468316686795

    Google Scholar 

  46. Linas BP, Barter DM, Morgan JR, Pho MT, Leff JA, Schackman BR, Horsburgh CR, Assoumou SA, Salomon JA, Weinstein MC (2015) The cost-effectiveness of sofosbuvir-based regimens for treatment of hepatitis C virus genotype 2 or 3 infection. Ann Intern Med 162(9):619–629

    Google Scholar 

  47. Liu B, Koziel S, Zhang Q (2016) A multi-fidelity surrogate-model-assisted evolutionary algorithm for computationally expensive optimization problems. J Comput Sci 12:28–37

    Google Scholar 

  48. Liu S, Brandeau ML, Goldhaber-Fiebert JD (2017) Optimizing patient treatment decisions in an era of rapid technological advances: the case of hepatitis C treatment. Health Care Management Science 20(1):16–32

    Google Scholar 

  49. Liu S, Cipriano LE, Holodniy M, Goldhaber-Fiebert JD (2013) Cost-effectiveness analysis of risk-factor guided and birth-cohort screening for chronic hepatitis C infection in the United States. PLoS One 8(3):e58975

    Google Scholar 

  50. Liu S, Cipriano LE, Holodniy M, Owens DK, Goldhaber-Fiebert JD (2012) New protease inhibitors for the treatment of chronic hepatitis C: A cost-effectiveness analysis. Ann Intern Med 156(4):279–290

    Google Scholar 

  51. Liu S, Schwarzinger M, Carrat F, Goldhaber-Fiebert JD (2011) Cost effectiveness of fibrosis assessment prior to treatment for chronic hepatitis C patients. PLoS One 6(12):e26783

    Google Scholar 

  52. Liu S, Watcha D, Holodniy M, Goldhaber-Fiebert JD (2014) Sofosbuvir-based treatment regimens for chronic, genotype 1 hepatitis C virus infection in US incarcerated populations: A cost-effectiveness analysis. Ann Intern Med 161(8):546–553

    Google Scholar 

  53. Maillart LM, Ivy JS, Ransom S, Diehl K (2008) Assessing dynamic breast cancer screening policies. Oper Res 56(6):1411–1427

    Google Scholar 

  54. Martin NK, Pitcher AB, Vickerman P, Vassall A, Hickman M (2011) Optimal control of hepatitis C antiviral treatment programme delivery for prevention amongst a population of injecting drug users. PLoS One 6 (8):e22309

    Google Scholar 

  55. Mason JE, Denton BT, Shah ND, Smith SA (2014) Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients. Eur J Oper Res 233(3):727–738

    Google Scholar 

  56. Maxwell MS, Restrepo M, Henderson SG, Topaloglu H (2010) Approximate dynamic programming for ambulance redeployment. INFORMS J Comput 22(2):266–281

    Google Scholar 

  57. Najafzadeh M, Andersson K, Shrank WH, Krumme AA, Matlin OS, Brennan T, Avorn J, Choudhry NK (2015) Cost-effectiveness of novel regimens for the treatment of hepatitis C virus. Ann Intern Med 162(6):407–419

    Google Scholar 

  58. Ozcan YA (2005) Quantitative methods in health care management: techniques and applications. Jossey-Bass, San Francisco

    Google Scholar 

  59. Patrick J, Puterman ML, Queyranne M (2008) Dynamic multipriority patient scheduling for a diagnostic resource. Oper Res 56(6):1507–1525

    Google Scholar 

  60. Pollack A (2013) Hepatitis C, a silent killer, meets its match. The New York Times

  61. Powell WB (2007) Approximate dynamic programming: solving the curses of dimensionality. Wiley, New York

    Google Scholar 

  62. Rauner MS, Gutjahr WJ, Heidenberger K, Wagner J, Pasia J (2010) Dynamic policy modeling for chronic diseases: Metaheuristic-based identification of Pareto-optimal screening strategies. Oper Res 58(5):1269–1286

    Google Scholar 

  63. Rein DB, Smith BD, Wittenborn JS, Lesesne SB, Wagner LD, Roblin DW, Patel N, Ward JW, Weinbaum CM (2012) The cost-effectiveness of birth-cohort screening for hepatitis C antibody in US primary care settings. Ann Intern Med 156(4):263–270

    Google Scholar 

  64. Salomon JA, Weinstein MC, Hammitt JK, Goldie SJ (2003) Cost-effectiveness of treatment for chronic hepatitis C infection in an evolving patient population. J Am Med Assoc 290(2):228– 237

    Google Scholar 

  65. Sandıkçı B, Maillart LM, Schaefer AJ, Roberts MS (2013) Alleviating the patient’s price of privacy through a partially observable waiting list. Manag Sci 59(8):1836–1854

    Google Scholar 

  66. Schaefer AJ, Bailey MD, Shechter SM, Roberts MS (2005) Modeling medical treatment using Markov decision processes. In: Operations Research and Health Care, pp 593–612. Springer

  67. Secomandi N (2001) A rollout policy for the vehicle routing problem with stochastic demands. Oper Res 49 (5):796–802

    Google Scholar 

  68. Secomandi N (2003) Analysis of a rollout approach to sequencing problems with stochastic routing applications. J Heuristics 9(4):321–352

    Google Scholar 

  69. Shechter SM, Bailey MD, Schaefer AJ, Roberts MS (2008) The optimal time to initiate HIV therapy under ordered health states. Oper Res 56(1):20–33

    Google Scholar 

  70. Smith BD, Morgan RL, Beckett GA, Falck-Ytter Y, Holtzman D, Ward JW (2012) Hepatitis C virus testing of persons born during 1945–1965: Recommendations from the Centers for Disease Control and Prevention. Ann Intern Med 157(11):817–822

    Google Scholar 

  71. Spaulding AC, Thomas DL (2012) Screening for HCV infection in jails. J Am Med Assoc 307(12):1259–1260

    Google Scholar 

  72. Steimle LN, Denton BT (2017) Markov decision processes for screening and treatment of chronic diseases. In: Markov Decision Processes in Practice, pp 189–222. Springer

  73. Stepanova M, Kanwal F, El-Serag HB, Younossi ZM (2011) Insurance status and treatment candidacy of hepatitis c patients: Analysis of population-based data from the United States. Hepatology 53(3):737–745

    Google Scholar 

  74. Trudeau K, Manser K, Behling M, Brown C, Niebler W, Budman S (2017) Natural history of prescription opioid misuse/abuse: a qualitative investigation. The Journal of Pain 18(4):S32–S33

    Google Scholar 

  75. U.S. Census Bureau (2015) 2010 Census Summary File, Accessed April 1, 2018, https://www.census.gov/prod/cen2010/doc/sf1.pdf

  76. Moyer VA (2018) Screening for hepatitis C virus infection in adults: U.S. Preventive services task force recommendation statement. Ann Intern Med 159(5):349–357

    Google Scholar 

  77. USPSTF (2018) U.S. preventive services task force: Screening for depression in adults. Accessed April 1, 2018, https://www.uspreventiveservicestaskforce.org/Page/Document/UpdateSummaryFinal/depression-in-adults-screening1

  78. VA (2017) State of care for veterans with chronic hepatitis C. Accessed April 1, 2018, http://www.hepatitis.va.gov/provider/policy/HCV-state-of-care-2010.asp, [accessed: 13 June 2017]

  79. Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129(4):370–380

    Google Scholar 

  80. Winston WL (2003) Introduction to probability models: operations research. Volume Two. Brooks/Cole-Thomson Learning

  81. Xu J, Zhang S, Huang E, Chen VH, Lee LH, Celik N (2014) Efficient multi-fidelity simulation optimization. In: Tolk A, Diallo SY, Ryzhov IO, Yilmaz L, Buckley S, Miller JA (eds) Proceedings of the 2014 Winter Simulation Conference, pp 3940–3951

  82. Zaric GS (2013) Operations research and health care policy. Springer, New York

    Google Scholar 

  83. Zenios SA, Chertow GM, Wein LM (2000) Dynamic allocation of kidneys to candidates on the transplant waiting list. Oper Res 48(4):549–569

    Google Scholar 

Download references

Funding

This study received no funding.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Shan Liu.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Appendices

Appendix A: Computational Complexity and Properties of MF-RA

1.1 A.1 Computational Complexity of MF-RA

There are two primary sources of computation in MF-RA: the progression of the population as in Eq. 2, and the computation of LFM in Eq. 7 involving a matrix inverse.

Evaluating a complete sequential policy for the whole lifetime horizon requires nq population computations, i.e., population is evolved nq times using Eq. 2. An exhaustive search with np discretized policies, i.e., the grid search approach, would evaluate \((n_{p})^{n_{d}}\) possible sequential policies, in total

$$ n_{q}(n_{p})^{n_{d}} $$
(17)

population computations, which is exponential in the number of decision periods nd.

In contrast, MF-RA requires far fewer population computations. In each decision period d ∈ {1,…,nd}, there is a total of np discretized policies that need to be evaluated. For one discretized policy at decision d, we need (nq − (d − 1)nq,dec) population computations to calculate QALYs from that time forward. Also, each discretized policy requires npnq,dec (ndd) population computations in low-fidelity approximation (Step 2 in MF-RA) to iteratively calculate LFM. Note that in the last decision period nd, MF-RA does not need to compute LFM, so only population computations from nd to nq are needed.

As a result, the overall number of population computations in MF-RA is

$$\begin{array}{@{}rcl@{}} &&{}\sum\limits_{d = 1}^{n_{d}}n_{p}\left\{\left[n_{q}-\left( d-1\right)n_{q,dec}\right]+n_{p}n_{q,dec}\left( n_{d}-d\right)\right\} \\ &&{}\leq n_{p}n_{d}n_{q}+{n_{p}^{2}}{n_{d}^{2}}n_{q,dec} . \end{array} $$
(18)

In addition, the low-fidelity approximation conducts a matrix inversion, i.e., (IQ(It))− 1,

$$ \sum\limits_{d = 1}^{n_{d}}{n_{p}^{2}}n_{q,dec}\left( n_{d}-d\right)\leq {n_{p}^{2}}{n_{d}^{2}}n_{q,dec} $$
(19)

times. Using Gauss-Jordan elimination, the computational complexity for each matrix inversion is \(O(|\mathcal {S}_{HS}|^{3})\); however, the matrix is very sparse. To summarize, the number of population computations of MF-RA is quadratic in the number of decision periods nd and the number of matrix inversions is quadratic in nd.

1.2 A.2 Properties of MF-RA

We characterize the structure of the overall QALYs in budget planning years with respect to the general form of a rollout algorithm. We show the proposed MF-RA has the property of sequential QALYs improvement, which indicates that, using the low-fidelity approximation, the policy obtained in the decision period d + 1 is no worse than the performance of the policies obtained in the decision period d.

A rollout algorithm is an iterative method that uses a heuristic to iteratively improve the objective function value of combinatorial optimization problems. Unlike dynamic programming, which calculates a recursive value function with boundary conditions, a rollout algorithm approximates the optimal value function by a designed heuristic approach [10]. To optimize healthcare interventions, we need more effective sequential actions. Thus, different from other applications of rollout algorithms by only adopting the rollout process once, e.g., stochastic scheduling [11], stochastic routing [67, 68], and multidimensional knapsack problem [14], MF-RA adopts two rollout processes by incorporating low-fidelity approximation into the high-fidelity model. The proposed MF-RA, combining discretized policies enumeration and low-fidelity approximation, satisfies the sequential QALYs improvement.

Proposition 1

Sequential QALYs Improvement of MF-RA MF-RA improves overall QALYs in every decision period.

To prove MF-RA improves overall QALYs over time, we need to use two useful properties developed in rollout algorithms: sequential consistency and sequential improvement. In our population health outcome maximization problem, sequential consistency means that, given current population size, MF-RA is able to generate the same population evolution afterward regardless of earlier population size. That is, given \(\mathbf {N}_{1+(d-1)n_{q,dec}}\) at decision period d, if MF-RA generates the sequential policies \(\left \{I^{(t)*}\right \}_{t\in \mathcal {T}_{d}\cup \mathcal {T}_{d + 1}\cup {\cdots } \mathcal {T}_{n_{d}}}\) and associated population evolution \(\left \{\mathbf {N}_{t}\right \}_{t\in \mathcal {T}_{d}\cup \mathcal {T}_{d + 1}\cup {\cdots } \mathcal {T}_{n_{d}}}\), then given \(\mathbf {N}_{1+(d)n_{q,dec}}\) at decision period d + 1, MF-RA will generate the sequential policies \(\left \{I^{(t)*}\right \}_{t\in \mathcal {T}_{d + 1}\cup {\cdots } \cup \mathcal {T}_{n_{d}}}\) and associated population evolution \(\left \{\mathbf {N}_{t}\right \}_{t\in \mathcal {T}_{d + 1}\cup {\cdots } \cup \mathcal {T}_{n_{d}}}\). Sequential improvement in MF-RA means that the overall QALYs improves over time. That is, maximum QALYs calculated at decision period d (Equation (20)) is no better than maximum QALYs calculated at decision period d + 1 (Equation (21)). Note that the low-fidelity approximation (step 2 in MF-RA) is also a rollout algorithm and holds these two properties.

We prove in (a) that the low-fidelity approximation satisfies sequential consistency, as well as MF-RA in (b). Then, according to the theorem in [10], the sequential consistency of MF-RA allows us to claim its sequential improvement.

1.2.1 (a) Sequential Consistency of Low-Fidelity Approximation

Starting from \(\mathbf {N}_{1+(d-1)n_{q,dec}}\), maximizing LFM iteratively generates sequential policies

$$\left\{I_{Ak}^{(t)}\right\}_{t\in \mathcal{T}_{d}\cup \mathcal{T}_{d + 1}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}$$

and the corresponding sequence of population size

$$\left\{\mathbf{N}_{1+(d-1)n_{q,dec}}, \ldots,\mathbf{N}_{1+(n_{d})n_{q,dec}}\right\} $$

(shown in Eqs. 2224). Due to the fact that

$$\mathbf{N}_{1+dn_{q,dec}}=\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}}\mathbf{P}\left( \mathbf{I}_{Ak}^{(t)}\right)$$

if starting from \(\mathbf {N}_{1+dn_{q,dec}}\), then \(\mathbf {N}_{1+(d + 1)n_{q,dec}}\ldots \)\(\mathbf {N}_{1+(d-1)n_{q,dec}}\) is calculated by Eq. (25) and (26).

Therefore, whenever low-fidelity approximation generates the sequence of population size

$$\left\{\mathbf{N}_{1+(d-1)n_{q,dec}},\mathbf{N}_{1+dn_{q,dec}},\ldots,\mathbf{N}_{1+n_{d}n_{q,dec}}\right\} $$

starting from \(\mathbf {N}_{1+(d-1)n_{q,dec}}\), it also generates the sequence of population evolution

$$\left\{\mathbf{N}_{1+dn_{q,dec}},\mathbf{N}_{1+(d + 1)n_{q,dec}},\ldots,\mathbf{N}_{1+n_{d}n_{q,dec}}\right\} $$

starting from \(\mathbf {N}_{1+dn_{q,dec}}\). Thus, the low-fidelity approach is sequentially consistent, which helps us to prove that MF-RA is also sequentially consistent.

$$\begin{array}{@{}rcl@{}} \max\limits_{k\in\{1,\ldots,n_{p}\}}QALY\left( \mathbf{N}_{1}, \left\{I^{(t)*}\right\}_{t\in \mathcal{T}_{1}\cup {\cdots} \cup \mathcal{T}_{d-1}},\left\{I_{k}^{(t)}\right\}_{t\in \mathcal{T}_{d}},\left\{I_{Ak}^{(t)}\right\}_{t\in \mathcal{T}_{d + 1}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\right) \end{array} $$
(20)
$$\begin{array}{@{}rcl@{}} \max\limits_{k\in\{1,\ldots,n_{p}\}}QALY\left( \mathbf{N}_{1}, \left\{I^{(t)*}\right\}_{t\in \mathcal{T}_{1}\cup {\cdots} \cup \mathcal{T}_{d}},\left\{I_{k}^{(t)}\right\}_{t\in \mathcal{T}_{d + 1}},\left\{I_{Ak}^{(t)}\right\}_{t\in \mathcal{T}_{d + 2}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\right) \end{array} $$
(21)
$$\begin{array}{@{}rcl@{}} \mathbf{N}_{1+dn_{q,dec}}& = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}}\mathbf{P}\left( I_{Ak}^{(t)}\right) \end{array} $$
(22)
$$\begin{array}{@{}rcl@{}} \mathbf{N}_{1+(d + 1)n_{q,dec}}& = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}\cup \mathcal{T}_{d + 1}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ &\vdots& \end{array} $$
(23)
$$\begin{array}{@{}rcl@{}} \mathbf{N}_{1+n_{d}n_{q,dec}}& = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}\cup \mathcal{T}_{d + 1}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ \end{array} $$
(24)
$$\begin{array}{@{}rcl@{}} \mathbf{N}_{1+(d + 1)n_{q,dec}}& = &\mathbf{N}_{1+dn_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d + 1}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ & = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\prod\limits_{t\in \mathcal{T}_{d + 1}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\!\\ & = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}\cup \mathcal{T}_{d + 1}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ &\vdots& \end{array} $$
(25)
$$\begin{array}{@{}rcl@{}} \mathbf{N}_{1+n_{d}n_{q,dec}}& = &\mathbf{N}_{1+dn_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d + 1}\cup \mathcal{T}_{d + 2}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ & = &\mathbf{N}_{1+(d-1)n_{q,dec}}\prod\limits_{t\in \mathcal{T}_{d}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ &&\prod\limits_{t\in \mathcal{T}_{d + 1}\cup \mathcal{T}_{d + 2}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\mathbf{P}\left( I_{Ak}^{(t)}\right)\\ & = &\mathbf{N}_{1+(d-1)n_{q,dec}} \prod\limits_{t\in \mathcal{T}_{d}\cup \mathcal{T}_{d + 1}\cup {\cdots} \cup \mathcal{T}_{n_{d}}}\mathbf{P}\left( I_{Ak}^{(t)}\right) \\ \end{array} $$
(26)

1.2.2 (b) Sequential Consistency of MF-RA

With the same population size \(\mathbf {N}_{1+(d-1)n_{q,dec}}\) and the budget constraints, MF-RA enumerates the discretized intervention policies in decision period d in the same discretized rule. In addition, due to the sequential consistency of the low-fidelity approximation, MF-RA examines the same promising sequential policies and thus derives the same best policy I(t)∗, \(t\in \mathcal {T}_{d}\) and proceeds to the next population size \(\mathbf {N}_{1+dn_{q,dec}}\). Therefore, MF-RA generates the same sequence of population evolution \(\left \{\mathbf {N}_{1+(d-1)n_{q}d},\mathbf {N}_{1+dn_{q,dec}},\ldots ,\mathbf {N}_{1+(n_{d}-1)n_{q,dec}}\right \}\) regardless of the population size before decision period d. That is, starting from any population size in the sequential population size generated by MF-RA, MF-RA is able to generate the same sequential population size afterward.

1.2.3 (c) Sequential Improvement of MF-RA

By applying the theorem in [10], we can claim that MF-RA satisfies sequential improvement, and since MF-RA determined the best policy by calculating and comparing accumulating overall QALYs, it is capable of improving overall QALYs when rolling out to the next decision period.

Appendix B: Detail of the Transition Probability Matrix for HCV Progression

We provide the details of the transition matrix for HCV progression. The transition probabilities from group A to other groups are functions of the decision variables

$$I_{A,t}=\left\{I_{A,t}^{H_{A}}, I_{A,t}^{F0_{A}},I_{A,t}^{F1_{A}},I_{A,t}^{F2_{A}}, I_{A,t}^{F3_{A}},I_{A,t}^{F4_{A}}\right\}. $$

The transition probabilities from group B to other groups are functions of the decision variables

$$I_{B,t}=\left\{I_{B,t}^{F0_{B}},I_{B,t}^{F1_{B}}, I_{B,t}^{F2_{B}},I_{B,t}^{F3_{B}},I_{B,t}^{F4_{B}}\right\}. $$

We denote \(\theta _{a,r,waiting}^{ij}\) as the fraction of patient population of age a ∈{40 − 49, 50 − 59, 60 − 69} and gender r ∈{female,male} who are not going through treatment and transit from health status i to health status j, and \(\theta _{a,r,ongoing}^{ij}\) as the fraction of patient population of age a and gender r who are going through treatment and transit from health status i to health status j, i,j ∈{H, F0-F4, R1-R3, DC, HCC, LT, ALT, M}. The complete transition probability matrix is in Equation (27), where Pa,female (IA,t, IB,t) and Pa,male (IA,t, IB,t) are 20 × 20 sub-matrices of transition probabilities for female and male, respectively, a ∈{40 − 49, 50 − 59, 60 − 69}. The transition probabilities are summarized in Fig. 7 and Tables 68.

Fig. 7
figure 7

Transition probability sub-matrix for a ∈{40 − 49,50 − 59,60 − 69} and \(r\in \left \{female,male\right \}\)

Table 6 List of Transition Probabilities from Group A to Other Groups
Table 7 List of Transition Probabilities from Group B to Other Groups
Table 8 List of Transition Probabilities from Groups C and D to Other Groups

Appendix C: Details on Input Data

Table 9 Model parameters values
Table 10 Cohort characteristics

Appendix D: Supplementary Figures and Tables for Experiments

Table 11 Experiment 1: Incremental QALYs over time (MF-RA: multi-fidelity rollout algorithm)
Fig. 8
figure 8

Experiment 3: HCV control over 20 years for 50-59 birth-cohort

Table 12 Experiment 3: Dollars (in billions) spent every two years (every decision period) for 50-59 birth-cohort
Fig. 9
figure 9

Experiment 3: HCV control over 10 years for 60-69 birth-cohort

Table 13 Experiment 3: Dollars (in billions) spent every two years (every decision period) for 60-69 birth-cohort

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ho, TY., Liu, S. & Zabinsky, Z.B. A Multi-Fidelity Rollout Algorithm for Dynamic Resource Allocation in Population Disease Management. Health Care Manag Sci 22, 727–755 (2019). https://doi.org/10.1007/s10729-018-9454-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10729-018-9454-6

Keywords

Navigation