Abstract
As social robots become integral to daily life, effective battery management and personalized user interactions are crucial. We employed Q-learning with the Miro-E robot for balancing self-sustained energy management and personalized user engagement. Based on our approach, we anticipate that the robot will learn when to approach the charging dock and adapt interactions according to individual user preferences. For energy management, the robot underwent iterative training in a simulated environment, where it could opt to either “play” or “go to the charging dock”. The robot also adapts its interaction style to a specific individual, learning which of three actions would be preferred based on feedback it would receive during real-world human-robot interactions. From an initial analysis, we identified a specific point at which the Q values are inverted, indicating the robot’s potential establishment of a battery threshold that triggers its decision to head to the charging dock in the energy management scenario. Moreover, by monitoring the probability of the robot selecting specific behaviours during human-robot interactions over time, we expect to gather evidence that the robot can successfully tailor its interactions to individual users in the realm of personalized engagement.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
See: https://miro-e.com/robot.
References
Cao, J., Harrold, D., Fan, Z., Morstyn, T., Healey, D., Li, K.: Deep reinforcement learning-based energy storage arbitrage with accurate lithium-ion battery degradation model. IEEE Trans. Smart Grid 11(5), 4513–4521 (2020)
Castro-González, A., Amirabdollahian, F., Polani, D., Malfaz, M., Salichs, M.A.: Robot self-preservation and adaptation to user preferences in game play, a preliminary study. In: International Conference on Robotics and Biomimetics, pp. 2491–2498 (2011)
Chaoui, H., Gualous, H., Boulon, L., Kelouwani, S.: Deep reinforcement learning energy management system for multiple battery based electric vehicles. In: 2018 IEEE Vehicle Power and Propulsion Conference (VPPC), pp. 1–6. IEEE (2018)
Chellal, A.A., Lima, J., Gonçalves, J., Megnafi, H.: Battery management system for mobile robots based on an extended Kalman filter approach. In: 2021 29th Mediterranean Conference on Control and Automation, pp. 1131–1136. IEEE (2021)
Deshmukh, A., Aylett, R.: Socially constrained management of power resources for social mobile robots. In: International Conference on Human-Robot Interaction, pp. 119–120 (2012)
Fasola, J., Mataric, M.J.: Using socially assistive human-robot interaction to motivate physical exercise for older adults. Proc. IEEE 100(8), 2512–2526 (2012)
Gockley, R., et al.: Designing robots for long-term social interaction, pp. 1338–1343, September 2005
Kuznetsova, E., Li, Y.F., Ruiz, C., Zio, E., Ault, G., Bell, K.: Reinforcement learning for microgrid energy management. Energy 59, 133–146 (2013)
Liu, C., Conn, K., Sarkar, N., Stone, W.: Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE Trans. Rob. 24(4), 883–896 (2008)
Liu, Y., Zhang, J.: Self-adapting J-type air-based battery thermal management system via model predictive control. Appl. Energy 263, 114640 (2020)
Mitsunaga, N., Smith, C., Kanda, T., Ishiguro, H., Hagita, N.: Adapting robot behavior for human-robot interaction. IEEE Trans. Rob. 24(4), 911–916 (2008)
Natella, D., Vasca, F.: Battery state of health estimation via reinforcement learning. In: 2021 European Control Conference (ECC), pp. 1657–1662. IEEE (2021)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Tapus, A., Ţăpuş, C., Matarić, M.J.: User-robot personality matching and assistive robot behavior adaptation for post-stroke rehabilitation therapy. Intel. Serv. Robot. 1, 169–183 (2008)
Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between Epsilon-Greedy and Softmax. In: Bach, J., Edelkamp, S. (eds.) KI 2011. LNCS (LNAI), vol. 7006, pp. 335–346. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24455-1_33
Watkins, C.J.C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Fujii, K., Holthaus, P., Samani, H., Premachandra, C., Amirabdollahian, F. (2024). Two-Level Reinforcement Learning Framework for Self-sustained Personal Robots. In: Ali, A.A., et al. Social Robotics. ICSR 2023. Lecture Notes in Computer Science(), vol 14453 . Springer, Singapore. https://doi.org/10.1007/978-981-99-8715-3_30
Download citation
DOI: https://doi.org/10.1007/978-981-99-8715-3_30
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8714-6
Online ISBN: 978-981-99-8715-3
eBook Packages: Computer ScienceComputer Science (R0)