Health Care Management Science

, Volume 20, Issue 2, pp 286–302 | Cite as

Strategic level proton therapy patient admission planning: a Markov decision process modeling approach

  • Ridvan Gedik
  • Shengfan ZhangEmail author
  • Chase Rainwater


A relatively new consideration in proton therapy planning is the requirement that the mix of patients treated from different categories satisfy desired mix percentages. Deviations from these percentages and their impacts on operational capabilities are of particular interest to healthcare planners. In this study, we investigate intelligent ways of admitting patients to a proton therapy facility that maximize the total expected number of treatment sessions (fractions) delivered to patients in a planning period with stochastic patient arrivals and penalize the deviation from the patient mix restrictions. We propose a Markov Decision Process (MDP) model that provides very useful insights in determining the best patient admission policies in the case of an unexpected opening in the facility (i.e., no-shows, appointment cancellations, etc.). In order to overcome the curse of dimensionality for larger and more realistic instances, we propose an aggregate MDP model that is able to approximate optimal patient admission policies using the worded weight aggregation technique. Our models are applicable to healthcare treatment facilities throughout the United States, but are motivated by collaboration with the University of Florida Proton Therapy Institute (UFPTI).


Patient admission policy Proton therapy Markov decision process State aggregation 


  1. 1.
    Adan I, Vissers J (2002) Patient mix optimisation in hospital admission planning: a case study. Int J Oper Prod Manage 22(4):445–461CrossRefGoogle Scholar
  2. 2.
    Agnetis A, Coppi A, Corsini M, Dellino G, Meloni C, Pranzo M (2012) Long term evaluation of operating theater planning policies. Oper Res Health Care 1(4):95–104CrossRefGoogle Scholar
  3. 3.
    Agnetis A, Coppi A, Corsini M, Dellino G, Meloni C, Pranzo M (2014) A decomposition approach for the combined master surgical schedule and surgical case assignment problems. Health Care Manag Sci 17(1):49–59CrossRefGoogle Scholar
  4. 4.
    American Cancer Society (2014) Cancer facts and figures 2014, Accessed 30 June 2014
  5. 5.
    Andre D, Russell SJ (2002) State abstraction for programmable reinforcement learning agents. In: AAAI/IAAI, pp 119–125Google Scholar
  6. 6.
    Averill RF, Muldoon JH, Vertrees JC, Goldfield NI, Mullin RL, Fineran EC, Zhang MZ, Steinbeck B, Grant T (1993) The evolution of casemix measurement using diagnosis related groups (drgs). Wallingford 3M Health Information SystemsGoogle Scholar
  7. 7.
    Bertsimas D, Demir R (2002) An approximate dynamic programming approach to multidimensional knapsack problems. Manag Sci 48(4):550–565CrossRefGoogle Scholar
  8. 8.
    Bowers J, Mould G (2005) Ambulatory care and orthopaedic capacity planning. Health Care Manag Sci 8:41–47CrossRefGoogle Scholar
  9. 9.
    Cayirli T, Veral E, Rosen H (2006) Designing appointment scheduling systems for ambulatory care services. Health Care Manag Sci 9:47–58CrossRefGoogle Scholar
  10. 10.
    Ciavotta M, Dellino G, Meloni C, Pranzo M (2010) A rollout algorithmic approach for complex parallel machine scheduling in healthcare operations. In: Operations research for patient: centered health care delivery: proceedings of the XXXVI international ORAHS conference, vol 5, pp 316–324Google Scholar
  11. 11.
    Ciavotta M, Meloni C, Pranzo M (2009) Scheduling dispensing and counting in secondary pharmaceutical manufacturing. AIChE J 55(5):1161–1170CrossRefGoogle Scholar
  12. 12.
    Conforti D, Guerriero F, Guido R (2008) Optimization models for radiotherapy patient scheduling. 4OR 6:263–278CrossRefGoogle Scholar
  13. 13.
    Conforti D, Guerriero F, Guido R (2010) Non-block scheduling with priority for radiotherapy treatments. Eur J Oper Res 201:289–296CrossRefGoogle Scholar
  14. 14.
    Dean T, Givan R, Leach S (1997) Model reduction techniques for computing approximately optimal solutions for markov decision processes. In: Proceedings of the 13th conference on uncertainty in artificial intelligence. Morgan Kaufmann Publishers Inc, pp 124–131Google Scholar
  15. 15.
    Dearden R, Boutilier C (1997) Abstraction and approximate decision-theoretic planning. Artif Intell 89 (1):219–283CrossRefGoogle Scholar
  16. 16.
    Dietterich TG (2000) Hierarchical reinforcement learning with the maxq value function decomposition. J Artif Intell Res 13(1):227–303Google Scholar
  17. 17.
    Gedik R (2011) Evaluating the capacity of a proton therapy facility. University of Arkansas, FayettevilleGoogle Scholar
  18. 18.
    Gedik R (2014) Large-scale solution approaches for healthcare and supply chain scheduling. Phd thesis, University of ArkansasGoogle Scholar
  19. 19.
    Gocgun Y, Bresnahan BW, Ghate A, Gunn ML (2011) A markov decision process approach to multi-category patient scheduling in a diagnostic facility. Artif Intell Med 53(2):73–81CrossRefGoogle Scholar
  20. 20.
    Goitein M, Jermann M (2003) The relative costs of proton and x-ray radiation therapy. Clin Oncol 15:S37–S50CrossRefGoogle Scholar
  21. 21.
    Heyman DP, Sobel MJ (2003) Stochastic models in operations research: stochastic optimization, vol 2. Courier Dover PublicationsGoogle Scholar
  22. 22.
    Kapadia AS, Vineberg SE, Rossi CD (1985) Predicting course of treatment in a rehabilitation hospital: a Markovian model. Comput OR 12(5):459–469CrossRefGoogle Scholar
  23. 23.
    Kapamara T, Sheibani K, Haas O, Reeves C, Petrovic D (2006) A review of scheduling problems in radiotherapy. In: Proceedings of the 18th international conference on systems engineering (ICSE2006). Coventry University, UK, pp 201–207Google Scholar
  24. 24.
    Kerstiens J, Johnstone PA (2014) Proton therapy expansion under current united states reimbursement models. Int J Radiat Oncol Biol Phys 89(2):235–240CrossRefGoogle Scholar
  25. 25.
    Kolesar P (1970) A markovian model for hospital admission scheduling. Manag Sci 16(6):B384–B396CrossRefGoogle Scholar
  26. 26.
    Li L, Walsh TJ, Littman ML (2006) Towards a unified theory of state abstraction for mdps. In: ISAIMGoogle Scholar
  27. 27.
    Men C (2009) Optimization models for radiation therapy: treatment planning and patient scheduling. University of FloridaGoogle Scholar
  28. 28.
    Men C, Salari E, Romeijn HE, Klein SL, Li Z, Mendenhall NP, Palta JR (2008) Report of the strategic model. Technical report, University of FloridaGoogle Scholar
  29. 29.
    Nunes LGN, de Carvalho SV, Rodrigues RdCM (2009) Markov decision process applied to the control of hospital elective admissions. Artif Intell Med 47(2):159–171CrossRefGoogle Scholar
  30. 30.
    Peeters A, Grutters JP, Pijls-Johannesma M, Reimoser S, Ruysscher DD, Severens JL, Joore MA, Lambin P (2010) How costly is particle therapy? cost analysis of external beam radiotherapy with carbon-ions, protons and photons. Radiother Oncol 95(1):45– 53CrossRefGoogle Scholar
  31. 31.
    Petrovic D, Castro E, Petrovic S, Kapamara T, Urquhart N (2013) Radiotherapy scheduling. In: Uyar AS, Ozcan E (eds) Automated scheduling and planning. Studies in computational intelligence, vol 505. Springer, Berlin, pp 155–189CrossRefGoogle Scholar
  32. 32.
    Petrovic D, Morshed M, Petrovic S (2009) Genetic algorithm based scheduling of radiotherapy treatments for cancer patients. In: Combi C, Shahar Y, Abu-Hanna A (eds) Artificial Intelligence in Medicine. Lecture Notes in Computer Science, vol 5651. Springer, Berlin, pp 101–105Google Scholar
  33. 33.
    Pham DN, Klinkert A (2008) Surgical case scheduling as a generalized job shop scheduling problem. Eur J Oper Res 185:1011–1025CrossRefGoogle Scholar
  34. 34.
    Powell WB (2007) Approximate dynamic programming: solving the curses of dimensionality. WileyGoogle Scholar
  35. 35.
    Powell WB (2009) What you should know about approximate dynamic programming. Nav Res Logist (NRL) 56(3):239–249CrossRefGoogle Scholar
  36. 36.
    Powell WB, George A, Bouzaiene-Ayari B, Simao HP (2003) Approximate dynamic programming for high dimensional resource allocation problems. In: Proceedings of 2005 IEEE International Joint Conference on Neural Networks, 2005 (IJCNN’05), vol 5. IEEE, pp 2989–2994Google Scholar
  37. 37.
    Powell WB, Simao HP, Bouzaiene-Ayari B (2012) Approximate dynamic programming in transportation and logistics: a unified framework. EURO J Transport Log 1(3):237–284CrossRefGoogle Scholar
  38. 38.
    Salari E, Men C, Li Z, Mendenhall N, Palta J, Romeijn H. (2009) Su-ff-t-467: A new patient scheduling approach for proton therapy treatment delivery. Med Phys 36(6):2630–2630CrossRefGoogle Scholar
  39. 39.
    Sauré A, Patrick J, Tyldesley S, Puterman ML (2012) Dynamic multi-appointment patient scheduling for radiation therapy. Eur J Oper Res 223(2):573–584CrossRefGoogle Scholar
  40. 40.
    Schütz H-J, Kolisch R (2012) Approximate dynamic programming for capacity allocation in the service industry. Eur J Oper Res 218(1):239–250CrossRefGoogle Scholar
  41. 41.
    Shelley W, Brundage M, Hayter C, Paszat L, Zhou S, Mackillop W (2000) A shorter fractionation schedule for postlumpectomy breast cancer patients. Int J Radiat Oncol Biol Phys 47:1219– 1228CrossRefGoogle Scholar
  42. 42.
    Sickinger S, Kolisch R (2009) The performance of a generalized Bailey–Welch rule for outpatient appointment scheduling under inpatient and emergency demand. Health Care Manag Sci 12(4):408–419CrossRefGoogle Scholar
  43. 43.
    Sutton RS, Precup D, Singh S (1999) Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif Intell 112(1):181–211CrossRefGoogle Scholar
  44. 44.
    Yamada Y, Ackerman I, Franssen E, Mackenzie RG, Thomas G (1999) Does the dose fractionation schedule influence local control of adjuvant radiotherapy for early stage breast cancer. Int J Radiat Oncol Biol Phys 44:99–104CrossRefGoogle Scholar
  45. 45.
    Zhang NL, Zhang W (2001) Speeding up the convergence of value iteration in partially observable markov decision processes. J Artif Intell Res:29–51Google Scholar
  46. 46.
    Zietman A, Goitein M, Tepper JE (2010) Technology evolution: is it survival of the fittest? J Clin Oncol 28(27):4275– 4279CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Mechanical and Industrial Engineering Department Tagliatela College of EngineeringUniversity of New HavenWest HavenUSA
  2. 2.Department of Industrial EngineeringUniversity of ArkansasFayettevilleUSA

Personalised recommendations