Abstract
Epilepsy in childhood is a common neurological disorder in children. Most cases are benign childhood epilepsy, which can be controlled with medication by adaptive adjustment of the dosage of antiepileptic drugs (AEDs). Recently, reinforcement learning-based intelligent dosing has attracted increasing attention. In clinical practice, patients usually take more than one drug at a time, however, conventional reinforcement learning algorithms do not sufficiently address the combination of two or more active drugs. In this paper, we propose the multi-action offline reinforcement learning (MA-ORL) model to solve this problem. Concretely, MA-ORL inherits the basic framework of the actor-critic network. For the patient’s health and safety concerns, MA-ORL abandons the intrusive and high-risk “trial-and-error” interactions with the environment but directly learns from the offline clinical dataset in the initial phase until the model is sufficiently trained and ready to use. Besides, to choose multiple actions simultaneously, we replace the actor’s output in standard reinforcement learning with a 2D matrix indicating the mixed feature representation of all different actions, then multiply it with separate masks to obtain separated actions. In addition to reaching an optimal return (i.e., the reduction of seizure frequency), MA-ORL also emphasizes the accuracy of the recommended dosage. Therefore, we introduce weak supervision to the learning objective to restrict the range of the learning outcome. It guarantees the recommended dosage by MA-ORL is as close as the one prescribed by experienced physicians. We conduct extensive experiments on clinical medical records containing 245 cases of epilepsy in childhood. Experimental results show MA-ORL reaches the highest cumulative return among all baseline models. Moreover, the suggested amount of medication taken a day by MA-ORL is more accurate than any other benchmark.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chang, B.S., Lowenstein, D.H.: Epilepsy. N. Engl. J. Med. 349(13), 1257–1266 (2003)
Fisher, R., et al.: ILAE official report: a practical clinical definition of epilepsy. Epilepsia 55(4), 475–482 (2014)
Epilepsy. World Health Organization (WHO) (2022). https://www.who.int/news-room/fact-sheets/detail/epilepsy
Camfield, P., Camfield, C.: Incidence, prevalence and Aetiology of seizures and epilepsy in children. Epileptic Disord. 17(2), 117–123 (2015)
Stafstrom, C.E., Carmant, L.: Seizures and epilepsy: an overview for neuroscientists. Cold Spring Harb. Perspect. Med. 7(5), 1–19 (2015)
Minardi, C., et al.: Epilepsy in children: from diagnosis to treatment with focus on emergency. J. Clin. Med. 8(1), 39 (2019)
George, J., Kulkarni, C., Sarma, G.: Antiepileptic drugs and quality of life in patients with epilepsy: a tertiary care hospital-based study. Value Health Reg. Issues 6, 1–6 (2015)
Bittigau, P., et al.: Antiepileptic drugs and apoptotic neurodegeneration in the developing brain. Proc. Natl. Acad. Sci. 99(23), 15089–15094 (2002)
Liu, G., Nicole, S., Perkins, A.: Epilepsy: treatment options. Am. Fam. Physician 96(2), 87–96 (2017)
Topol, E.J.: High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25(1), 44–56 (2019)
He, J., Baxter, S.L., Xu, J., Xu, J., Zhou, X., Zhang, K.: The practical implementation of artificial intelligence technologies in medicine. Nat. Med. 25(1), 30–36 (2019)
Esteva, A., et al.: A guide to deep learning in healthcare. Nat. Med. 25(1), 24–29 (2019)
Taghanaki, S.A., Abhishek, K., Cohen, J.P., Cohen-Adad, J., Hamarneh, G.: Deep semantic segmentation of natural and medical images: a review. Artif. Intell. Rev. 54(1), 137–178 (2020)
Wang, L., Zhang, W., He, X., Zha, H.: Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2447–2456 (2018)
Zhao, Y., Kosorok, M.R., Zeng, D.: Reinforcement learning design for cancer clinical trials. Stat. Med. 28(26), 3294–3315 (2009)
Yu, C., Liu, J., Nemati, S., Yin, G.: Reinforcement learning in healthcare: a survey. ACM Comput. Surv. 55(1), 1–36 (2021)
Komorowski, M., Celi, L.A., Badawi, O., Gordon, A.C., Faisal, A.A.: The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24(11), 1716–1720 (2018)
Levine, S., Kumar, A., Tucker, G., Fu, J.: Offline reinforcement learning: tutorial, review, and perspectives on open problems. CoRR abs/2005.01643 (2020)
Fujimoto, S., Meger, D., Precup, D.: Off-policy deep reinforcement learning without exploration. In: Proceedings of the 36th International Conference on Machine Learning (ICML2019), pp. 2052–2062 (2019)
Busoniu, L., Babuska, R., De Schutter, B.: A comprehensive survey of multiagent reinforcement learning. IEEE Trans. Syst. Man, Cybern. Part C (Appl. Rev.) 38(2), 156–172 (2008)
Foerster, N.J., et al.: Stabilising experience replay for deep multi-agent reinforcement learning. In: Proceedings of the 34th International Conference on Machine Learning (ICML2017), pp. 1146–1155 (2017)
Sunehag, P., et al.: Value-decomposition networks for cooperative multi-agent learning. In: Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems (AAMAS2018), pp. 2085–2087 (2017)
Rashid, T., Samvelyan, M., Witt, D.S.C., Farquhar, G., Foerster, N.J., Whiteson, S.: QMIX: monotonic value function Factorisation for deep multi-agent reinforcement learning. In: Proceedings of the 35th International Conference on Machine Learning (ICML2018), pp. 4292–4301 (2018)
Ryan, L., Yi, W., Aviv, T., Jean, H., Pieter, A., Igor, M.: Multi-agent actor-critic for mixed cooperative-competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS2017), pp. 6382–6393 (2017)
Vinyals, O., et al.: StarCraft II: a new challenge for reinforcement learning. CoRR abs/1708.04782 (2017)
Baker, B., Kanitscheider, I., Markov, M.T., Wu, Y., Powel, G.: Emergent tool use from multi-agent Autocurricula. In: Proceedings of the 8th International Conference on Learning Representations (ICLR2020) (2020)
Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34(12), 5586–5609 (2022)
Sebastian R.: An overview of multi-task learning in deep neural networks. CoRR abs/1706.05098 (2017)
Yang, Y., Hospedales, T.M.: Trace norm Regularised deep multi-task learning. In: Proceedings of the 5th International Conference on Learning Representations (ICLR2017), Workshop Track Proceedings (2017)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, 2nd edn. MIT Press, Cambridge (2018)
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., Pineau, J.: An introduction to deep reinforcement learning. Found. Trends Mach. Learn. 11(3–4), 219–354 (2018)
Kingma, D.P., Welling, M.: Auto-encoding Variational Bayes. In: Proceedings of the 2nd International Conference on Learning Representations (ICLR2014) (2014)
Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS2015), pp. 3483–3491 (2015)
Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: a survey of learning methods. ACM Comput. Surv. 50(2), 1–35 (2017)
Yue Y., Le, H.M.: Imitation learning tutorial. ICML Tutorial (2018)
Mnih, V., et al.: Playing Atari with deep reinforcement learning. CoRR abs/1312.5602 (2013)
Lillicrap, P.T., et al.: Continuous control with deep reinforcement learning. In: Proceedings of the 4th International Conference on Learning Representations (ICLR2016) (2016)
Acknowledgments
This work is supported by the Ph.D., Scientific Research Foundation, The Chongqing University of Posts and Telecommunications (No. E012A2022026), National Key Research and Development Program of China (No. 2019YFE0110800), National Natural Science Foundation of China (No. 61972060 and 62027827), Natural Science Foundation of Chongqing (No. cstc2020jcyj-zdxmX0025), Research Institute for Artificial Intelligence of Things (Project No. CD5H) and Research and Innovation Office (Project No. BD4A), The Hong Kong Polytechnic University.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Z. et al. (2023). Weakly-Supervised Multi-action Offline Reinforcement Learning for Intelligent Dosing of Epilepsy in Children. In: Wang, X., et al. Database Systems for Advanced Applications. DASFAA 2023. Lecture Notes in Computer Science, vol 13946. Springer, Cham. https://doi.org/10.1007/978-3-031-30678-5_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-30678-5_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30677-8
Online ISBN: 978-3-031-30678-5
eBook Packages: Computer ScienceComputer Science (R0)