Abstract
While reinforcement learning (RL) has proven to be the approach of choice for tackling many complex problems, it remains challenging to develop and deploy RL agents in real-life scenarios successfully. This paper presents pH-RL (personalization in e-Health with RL), a general RL architecture for personalization to bring RL to health practice. pH-RL allows for various levels of personalization in health applications and allows for online and batch learning. Furthermore, we provide a general-purpose implementation framework that can be integrated with various healthcare applications. We describe a step-by-step guideline for the successful deployment of RL policies in a mobile application. We implemented our open-source RL architecture and integrated it with the MoodBuster mobile application for mental health to provide messages to increase daily adherence to the online therapeutic modules. We then performed a comprehensive study with human participants over a sustained period. Our experimental results show that the developed policies learn to select appropriate actions consistently using only a few days’ worth of data. Furthermore, we empirically demonstrate the stability of the learned policies during the study.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bailoni, T., Dragoni, M., Eccher, C., Guerini, M., Maimone, R.: Healthy lifestyle support: the PerKApp ontology. In: Dragoni, M., Poveda-Villalón, M., Jimenez-Ruiz, E. (eds.) OWLED/ORE -2016. LNCS, vol. 10161, pp. 15–23. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54627-8_2
Bellemare, M.G., et al.: Autonomous navigation of stratospheric balloons using reinforcement learning. Nature 588(7836), 77–82 (2020)
Brockman, G., et al.: OpenAI gym. CoRR abs/1606.01540 (2016). http://arxiv.org/abs/1606.01540
Dulac-Arnold, G., Mankowitz, D., Hester, T.: Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901 (2019)
Grua, E.M., De Sanctis, M., Lago, P.: A reference architecture for personalized and self-adaptive e-health apps. In: Muccini, H., et al. (eds.) ECSA 2020. CCIS, vol. 1269, pp. 195–209. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59155-7_15
Grua, E.M., Hoogendoorn, M.: Exploring clustering techniques for effective reinforcement learning based personalization for health and wellbeing. In: 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 813–820. IEEE (2018)
el Hassouni, A.: alielhassouni/ph-rl: ph-rl, March 2021. https://doi.org/10.5281/zenodo.4628543
el Hassouni, A., Hoogendoorn, M., Eiben, A.E., Van Otterlo, M., Muhonen, V.: End-to-end personalization of digital health interventions using raw sensor data with deep reinforcement learning: a comparative study in digital health interventions for behavior change. In: 2019 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pp. 258–264. IEEE (2019)
Hassouni, A., Hoogendoorn, M., van Otterlo, M., Barbaro, E.: Personalization of health interventions using cluster-based reinforcement learning. In: Miller, T., Oren, N., Sakurai, Y., Noda, I., Savarimuthu, B.T.R., Cao Son, T. (eds.) PRIMA 2018. LNCS (LNAI), vol. 11224, pp. 467–475. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03098-8_31
den Hengst, F., Grua, E.M., el Hassouni, A., Hoogendoorn, M.: Reinforcement learning for personalization: a systematic literature review. Data Sci. (Preprint) 3, 107–147 (2020)
Hoffman, M., et al.: Acme: a research framework for distributed reinforcement learning. arXiv preprint arXiv:2006.00979 (2020)
Komorowski, M., Celi, L.A., Badawi, O., Gordon, A.C., Faisal, A.A.: The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24(11), 1716–1720 (2018)
Mikus, A., Hoogendoorn, M., Rocha, A., Gama, J., Ruwaard, J., Riper, H.: Predicting short term mood developments among depressed patients using adherence and ecological momentary assessment data. Internet Interv. 12, 105–110 (2018)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015). https://doi.org/10.1038/nature14236
Mnih, V., et al.: Playing atari with deep reinforcement learning (2013)
Mol, M., Dozeman, E., Provoost, S., Van Schaik, A., Riper, H., Smit, J.H.: Behind the scenes of online therapeutic feedback in blended therapy for depression: mixed-methods observational study. J. Med. Internet Res. 20(5), e174 (2018)
Provoost, S., et al.: Improving adherence to an online intervention for low mood with a virtual coach: study protocol of a pilot randomized controlled trial. Trials 21(1), 1–12 (2020)
Roggeveen, L., et al.: Transatlantic transferability of a new reinforcement learning model for optimizing haemodynamic treatment for critically ill patients with sepsis. Artif. Intell. Med. 112, 102003 (2020)
Saria, S.: Individualized sepsis treatment using reinforcement learning. Nat. Med. 24(11), 1641–1642 (2018)
Schwebel, F.J., Larimer, M.E.: Using text message reminders in health care services: a narrative literature review. Internet Interv. 13, 82–104 (2018)
Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge (2018)
Vinyals, O., et al.: Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019)
Zhu, F., Guo, J., Xu, Z., Liao, P., Yang, L., Huang, J.: Group-driven reinforcement learning for personalized mHealth intervention. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 590–598. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_67
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
el Hassouni, A. et al. (2022). pH-RL: A Personalization Architecture to Bring Reinforcement Learning to Health Practice. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2021. Lecture Notes in Computer Science(), vol 13163. Springer, Cham. https://doi.org/10.1007/978-3-030-95467-3_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-95467-3_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-95466-6
Online ISBN: 978-3-030-95467-3
eBook Packages: Computer ScienceComputer Science (R0)