Skip to main content
SpringerLink
Log in

A Sarsa reinforcement learning hybrid ensemble method for robotic battery power forecasting

一种用于机器人电池电量预测的 Sarsa 强化学习混合集成方法

  • Published:
Journal of Central South University Aims and scope Submit manuscript

Abstract

Building a rail transit workshop with efficient data interconnection has become an inevitable trend in the transformation and development of the current rail transit equipment industry. More and more diversified mobile transport robots have become a priority in the process of digital transformation of smart factories. Accurate prediction of robot battery power can guide the control center to adopt scientific and reasonable instructions in advance to ensure efficient and stable operation of the logistics transportation chain. In this study, we propose a hybrid ensemble method of multiple learners based on state-action-reward-state-action (Sarsa) reinforcement learning algorithm. Maximal overlap discrete wavelet transform (MODWT) is used to preprocess the originally measured robot power supply voltage data. This significantly reduces the non-stationarity and volatility of time series data. Gated recurrent unit (GRU), deep belief network (DBN), and long short-term memory (LSTM), are utilized for the prediction modeling of subseries after decomposition. Finally, the Sarsa reinforcement learning ensemble strategy is used to weight the three basic predictors above. The performance of the Sarsa hybrid model is verified on three real mobile robot power data sets. Experimental results elaborate that the transportation robot battery power hybrid forecasting model is competitive in robustness, accuracy, and adaptability.

摘要

建设数据高效互联的轨道交通车间已成为当前轨道交通装备行业转型发展的必然趋势. 越来越多样化的移动运输机器人设备成为智能工厂数字化转型过程中的关键. 准确预测机器人的电池电量可以指导控制中心提前采取科学合理的指令, 确保物流运输链高效稳定运行. 在本研究中, 我们提出了一种基于状态-动作-奖励-状态-动作(Sarsa)强化学习算法的多学习器混合集成方法. 首先, 采用最大重叠离散小波变换(MODWT)对所测量的机器人原始电源电压数据进行预处理, 可以显著降低时间序列数据的非平稳性和波动性. 其次, 利用门控循环单元(GRU)、 深度置信网络(DBN)和长短期记忆(LSTM)对分解后得到的子序列进行预测建模. 最后, 使用 Sarsa 强化学习集成策略对上述三个基础预测器进行加权组合. 所提出的 Sarsa 混合集成模型的性能在三个真实移动机器人功率数据集上得到验证. 实验结果表明, 运输机器人电池动力混合预测模型在鲁棒性、准确性和适应性方面具有竞争力.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. JONES J L, SEIGER B A, FLYNN A M. Mobile robots: Inspiration to implementation [M]. AK Peters/CRC Press, 1998.

  2. WAKCHAURE M, PATLE B K, MAHINDRAKAR A K. Application of AI techniques and robotics in agriculture: A review [J]. Artificial Intelligence in the Life Sciences, 2023, 3: 100057. DOI: https://doi.org/10.1016/j.ailsci.2023.100057.

    Article  Google Scholar 

  3. HERCIK R, BYRTUS R, JAROS R, et al. Implementation of autonomous mobile robot in SmartFactory [J]. Applied Sciences, 2022, 12(17): 8912. DOI: https://doi.org/10.3390/app12178912.

    Article  Google Scholar 

  4. DAIM T U, YOON B S, LINDENBERG J, et al. Strategic roadmapping of robotics technologies for the power industry: A multicriteria technology assessment [J]. Technological Forecasting and Social Change, 2018, 131: 49–66. DOI: https://doi.org/10.1016/j.techfore.2017.06.006.

    Article  Google Scholar 

  5. ZHANG Han-ye, LIN Wei-ming, CHEN Ai-xia. Path planning for the mobile robot: A review [J]. Symmetry, 2018, 10(10): 450. DOI: https://doi.org/10.3390/sym10100450.

    Article  Google Scholar 

  6. GURUJI A K, AGARWAL H, PARSEDIYA D K. Time-efficient A* algorithm for robot path planning [J]. Procedia Technology, 2016, 23: 144–149. DOI: https://doi.org/10.1016/j.protcy.2016.03.010.

    Article  Google Scholar 

  7. ALEXOPOULOS C, GRIFFIN P M. Path planning for a mobile robot [J]. IEEE Transactions on Systems, Man, and Cybernetics, 1992, 22(2): 318–322. DOI: https://doi.org/10.1109/21.148404.

    Article  Google Scholar 

  8. GUL F, RAHIMAN W, ALHADY S S N. A comprehensive study for robot navigation techniques [J]. Cogent Engineering, 2019, 6(1): 1632046. DOI: https://doi.org/10.1080/23311916.2019.1632046.

    Article  Google Scholar 

  9. RAVANKAR A, RAVANKAR A, KOBAYASHI Y, et al. Path smoothing techniques in robot navigation: State-of-the-art, current and future challenges [J]. Sensors, 2018, 18(9): 3170. DOI: https://doi.org/10.3390/s18093170.

    Article  Google Scholar 

  10. BORENSTEIN J, EVERETT H R, FENG L, et al. Mobile robot positioning: Sensors and techniques [J]. Journal of Robotic Systems, 1997, 14(4): 231–249. DOI: https://doi.org/10.1002/(sici)1097-4563(199704)14:4<231:aid-rob2>3.0.co;2-r.

    Article  Google Scholar 

  11. YU Jing, JIANG Wen-song, LUO Zai, et al. Application of a vision-based single target on robot positioning system [J]. Sensors, 2021, 21(5): 1829. DOI: https://doi.org/10.3390/s21051829.

    Article  Google Scholar 

  12. ZHANG Wei, CHENG Hong-tai, HAO Li-na, et al. An obstacle avoidance algorithm for robot manipulators based on decision-making force [J]. Robotics and Computer-Integrated Manufacturing, 2021, 71: 102114. DOI: https://doi.org/10.1016/j.rcim.2020.102114.

    Article  Google Scholar 

  13. PADOY N, HAGER G D. Human-machine collaborative surgery using learned models [C]// 2011 IEEE International Conference on Robotics and Automation. Shanghai, China: IEEE, 2011: 5285–5292. DOI: https://doi.org/10.1109/ICRA.2011.5980250.

    Chapter  Google Scholar 

  14. HAESEVOETS T, de CREMER D, DIERCKX K, et al. Human-machine collaboration in managerial decision making [J]. Computers in Human Behavior, 2021, 119: 106730. DOI: https://doi.org/10.1016/j.chb.2021.106730.

    Article  Google Scholar 

  15. THUROW K, CHEN Chao, JUNGINGER S, et al. Transportation robot battery power forecasting based on bidirectional deep-learning method [J]. Transportation Safety and Environment, 2019, 1(3): 205–211. DOI: https://doi.org/10.1093/tse/tdz016.

    Article  Google Scholar 

  16. CHEN Jing-dong, RO P I. Human intention-oriented variable admittance control with power envelope regulation in physical human-robot interaction [J]. Mechatronics, 2022, 84: 102802. DOI: https://doi.org/10.1016/j.mechatronics.2022.102802.

    Article  Google Scholar 

  17. FAROOQ M U, EIZAD A, BAE H K. Power solutions for autonomous mobile robots: A survey [J]. Robotics and Autonomous Systems, 2023, 159: 104285. DOI: https://doi.org/10.1016/j.robot.2022.104285.

    Article  Google Scholar 

  18. PARASURAMAN R, KERSHAW K, PAGALA P, et al. Model based on-line energy prediction system for semi-autonomous mobile robots [C]// 2014 5th International Conference on Intelligent Systems, Modelling and Simulation. Langkawi, Malaysia: IEEE, 2015: 411–416. DOI: https://doi.org/10.1109/ISMS.2014.76.

    Google Scholar 

  19. ALHASSAN A B, ZHANG Xiao-dong, SHEN Hai-ming, et al. Power transmission line inspection robots: A review, trends and challenges for future research [J]. International Journal of Electrical Power & Energy Systems, 2020, 118: 105862. DOI: https://doi.org/10.1016/j.ijepes.2020.105862.

    Article  Google Scholar 

  20. QUANN M, OJEDA L, SMITH W, et al. Off-road ground robot path energy cost prediction through probabilistic spatial mapping [J]. Journal of Field Robotics, 2020, 37(3): 421–439. DOI: https://doi.org/10.1002/rob.21927.

    Article  Google Scholar 

  21. HAMZA A, AYANIAN N. Forecasting battery state of charge for robot missions [C]// Proceedings of the Symposium on Applied Computing. New York: ACM, 2017: 249–255. DOI: https://doi.org/10.1145/3019612.3019705.

    Chapter  Google Scholar 

  22. LÜ Xue-qin, DENG Rui-yu, CHEN Chao, et al. Performance optimization of fuel cell hybrid power robot based on power demand prediction and model evaluation [J]. Applied Energy, 2022, 316: 119087. DOI: https://doi.org/10.1016/j.apenergy.2022.119087.

    Article  Google Scholar 

  23. SHEN W X. State of available capacity estimation for lead-acid batteries in electric vehicles using neural network [J]. Energy Conversion and Management, 2007, 48(2): 433–442. DOI: https://doi.org/10.1016/j.enconman.2006.06.023.

    Article  Google Scholar 

  24. PENTZER J, BRENNAN S, REICHARD K. On-line estimation of vehicle motion and power model parameters for skid-steer robot energy use prediction [C]// 2014 American Control Conference. IEEE, 2014: 2786–2791.

  25. HOU Lin-fei, ZHANG Liang, KIM J. Energy modeling and power measurement for mobile robots [J]. Energies, 2018, 12(1): 27. DOI: https://doi.org/10.3390/en12010027.

    Article  Google Scholar 

  26. SABAREESH S U, ARAVIND K S N, CHOWDARY K B, et al. LSTM based 24 hours ahead forecasting of solar PV system for standalone household system [J]. Procedia Computer Science, 2023, 218: 1304–1313. DOI: https://doi.org/10.1016/j.procs.2023.01.109.

    Article  Google Scholar 

  27. PARASURAMAN R, MIN B C, ÖGREN P. Rapid prediction of network quality in mobile robots [J]. Ad Hoc Networks, 2023, 138: 103014. DOI: https://doi.org/10.1016/j.adhoc.2022.103014.

    Article  Google Scholar 

  28. RIBEIRO J, RUI Li-ma, ECKHARDT T, et al. Robotic process automation and artificial intelligence in industry 4.0-A literature review [J]. Procedia Computer Science, 2021, 181: 51–58. DOI: https://doi.org/10.1016/j.procs.2021.01.104.

    Article  Google Scholar 

  29. LIU Hui, STOLL N, JUNGINGER S, et al. A new approach to battery power tracking and predicting for mobile robot transportation using wavelet decomposition and ANFIS networks [C]//2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014). Bali, Indonesia: IEEE, 2015: 253–258. DOI: https://doi.org/10.1109/ROBIO.2014.7090339.

    Google Scholar 

  30. LEUTBECHER M, PALMER T N. Ensemble forecasting [J]. Journal of Computational Physics, 2008, 227(7): 3515–3539. DOI: https://doi.org/10.1016/j.jcp.2007.02.014.

    Article  MathSciNet  Google Scholar 

  31. PENG Fei, ZHENG Li, DUAN Zhu, et al. Multi-objective multi-learner robot trajectory prediction method for IoT mobile robot systems [J]. Electronics, 2022, 11(13): 2094. DOI: https://doi.org/10.3390/electronics11132094.

    Article  Google Scholar 

  32. HAMED Y, SHAFIE A, MUSTAFFA Z B, et al. An application of k-nearest neighbor interpolation on calibrating corrosion measurements collected by two non-destructive techniques [C]// 2015 IEEE 3rd International Conference on Smart Instrumentation, Measurement and Applications (ICSIMA). Kuala Lumpur, Malaysia: IEEE, 2016: 1–5. DOI: https://doi.org/10.1109/ICSIMA.2015.7559030.

    Google Scholar 

  33. WARSITO B, SUBANAR S, ABDURAKHMAN A. Wavelet decomposition for time series: Determining input model by using mRMR criterion [J]. Hacettepe Journal of Mathematics and Statistics, 2015, 44(1): 229–238. DOI: https://doi.org/10.15672/hjms.2014117462.

    MathSciNet  Google Scholar 

  34. ZHU Li, WANG Yan-xin, FAN Qi-bin. MODWT-ARMA model for time series prediction [J]. Applied Mathematical Modelling, 2014, 38(5–6): 1859–1865. DOI: https://doi.org/10.1016/j.apm.2013.10.002.

    Article  MathSciNet  Google Scholar 

  35. LI Ming-yang, CHEN Wan-zhong, ZHANG Tao. Application of MODWT and log-normal distribution model for automatic epilepsy identification [J]. Biocybernetics and Biomedical Engineering, 2017, 37(4): 679–689. DOI: https://doi.org/10.1016/j.bbe.2017.08.003.

    Article  Google Scholar 

  36. YAMAK P T, LI Yu-jian, GADOSEY P K. A comparison between ARIMA, LSTM, and GRU for time series forecasting [C]//Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence. New York: ACM, 2019: 49–55. DOI: https://doi.org/10.1145/3377713.3377722.

    Chapter  Google Scholar 

  37. LI Ming-wei, XU Dong-yang, GENG Jing, et al. A hybrid approach for forecasting ship motion using CNN-GRU-AM and GCWOA [J]. Applied Soft Computing, 2022, 114: 108084. DOI: https://doi.org/10.1016/j.asoc.2021.108084.

    Article  Google Scholar 

  38. KANG Ke, SUN Hong-bin, ZHANG Cheng-kang, et al. Short-term electrical load forecasting method based on stacked auto-encoding and GRU neural network [J]. Evolutionary Intelligence, 2019, 12(3): 385–394. DOI: https://doi.org/10.1007/s12065-018-00196-0.

    Article  Google Scholar 

  39. KUMAR S, HUSSAIN L, BANARJEE S, et al. Energy load forecasting using deep learning approach-LSTM and GRU in spark cluster [C]// 2018 Fifth International Conference on Emerging Applications of Information Technology (EAIT). Kolkata, India: IEEE, 2018: 1–4. DOI: https://doi.org/10.1109/EAIT.2018.8470406.

    Google Scholar 

  40. WU Li-zhen, KONG Chun, HAO Xiao-hong, et al. A short-term load forecasting method based on GRU-CNN hybrid neural network model [J]. Mathematical Problems in Engineering, 2020, 2020: 1–10. DOI: https://doi.org/10.1155/2020/1428104.

    Google Scholar 

  41. SHEN Fu-rao, CHAO Jing, ZHAO Jin-xi. Forecasting exchange rate using deep belief networks and conjugate gradient method [J]. Neurocomputing, 2015, 167: 243–253. DOI: https://doi.org/10.1016/j.neucom.2015.04.071.

    Article  Google Scholar 

  42. REN Yong-pan, MAO Jing-li, LIU Yong, et al. A novel dbn model for time series forecasting [J]. IAENG International Journal of Computer Science, 2017, 44(1): 79–86.

    Google Scholar 

  43. LIU Hui, YANG Rui, WANG Tian-tian, et al. A hybrid neural network model for short-term wind speed forecasting based on decomposition, multi-learner ensemble, and adaptive multiple error corrections [J]. Renewable Energy, 2021, 165: 573–594. DOI: https://doi.org/10.1016/j.renene.2020.11.002.

    Article  Google Scholar 

  44. LIU Hui, YANG Rui, DUAN Zhu. Wind speed forecasting using a new multi-factor fusion and multi-resolution ensemble model with real-time decomposition and adaptive error correction [J]. Energy Conversion and Management, 2020, 217: 112995. DOI: https://doi.org/10.1016/j.enconman.2020.112995.

    Article  Google Scholar 

  45. YANG Rui, LIU Hui, NIKITAS N, et al. Short-term wind speed forecasting using deep reinforcement learning with improved multiple error correction approach [J]. Energy, 2022, 239: 122128. DOI: https://doi.org/10.1016/j.energy.2021.122128.

    Article  Google Scholar 

  46. LIU Hui, YANG Rui, DUAN Zhu, et al. A hybrid neural network model for marine dissolved oxygen concentrations time-series forecasting based on multi-factor analysis and a multi-model ensemble [J]. Engineering, 2021, 7(12): 1751–1765. DOI: https://doi.org/10.1016/j.eng.2020.10.023.

    Article  Google Scholar 

  47. JAMEI M, ALI M, MALIK A, et al. Development of a TVF-EMD-based multi-decomposition technique integrated with Encoder-Decoder-Bidirectional-LSTM for monthly rainfall forecasting [J]. Journal of Hydrology, 2023, 617: 129105. DOI: https://doi.org/10.1016/j.jhydrol.2023.129105.

    Article  Google Scholar 

  48. MAO Wei-fang, ZHU Hui-ming, WU Hao, et al. Forecasting and trading credit default swap indices using a deep learning model integrating Merton and LSTMs [J]. Expert Systems with Applications, 2023, 213: 119012. DOI: https://doi.org/10.1016/j.eswa.2022.119012.

    Article  Google Scholar 

  49. LIU Hui, YANG Rui. A spatial multi-resolution multi-objective data-driven ensemble model for multi-step air quality index forecasting based on real-time decomposition [J]. Comput Ind, 2021, 125: 103387. DOI: https://doi.org/10.1016/j.compind.2020.103387.

    Article  Google Scholar 

  50. SIAMI-NAMINI S, NAMIN A. Forecasting economics and financial time series: ARIMA vs LSTM [J]. arXiv preprint arXiv:180306386, 2018.

  51. WIERING M A, van HASSELT H. Ensemble algorithms in reinforcement learning [J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 2008, 38(4): 930–936. DOI: https://doi.org/10.1109/TSMCB.2008.920231.

    Article  Google Scholar 

  52. ZHAO Dong-bin, WANG Hai-tao, KUN Shao, et al. Deep reinforcement learning with experience replay based on SARSA [C]//2016 IEEE Symposium Series on Computational Intelligence (SSCI). Athens: IEEE, 2017: 1–6. DOI: https://doi.org/10.1109/SSCI.2016.7849837.

    Google Scholar 

  53. FAUßER S, SCHWENKER F. Neural network ensembles in reinforcement learning [J]. Neural Processing Letters, 2015, 41(1): 55–69. DOI: https://doi.org/10.1007/s11063-013-9334-5.

    Article  Google Scholar 

  54. XU Zhi-xiong, CAO Lei, CHEN Xi-liang, et al. Deep reinforcement learning with sarsa and Q-learning: A hybrid approach [J]. IEICE Transactions on Information and Systems, 2018, E101. D(9): 2315–2322. DOI: https://doi.org/10.1587/transinf.2017edp7278.

    Article  Google Scholar 

  55. CANESE L, CARDARILLI G C, DI NUNZIO L, et al. Multi-agent reinforcement learning: A review of challenges and applications [J]. Applied Sciences, 2021, 11(11): 4948. DOI: https://doi.org/10.3390/app11114948.

    Article  Google Scholar 

  56. GO C K, LAO B, YOSHIMOTO J, et al. A reinforcement learning approach to the shepherding task using SARSA [C]//2016 International Joint Conference on Neural Networks (IJCNN). Vancouver, BC, Canada: IEEE, 2016: 3833–3836. DOI: https://doi.org/10.1109/IJCNN.2016.7727694.

    Chapter  Google Scholar 

  57. CHAI T, DRAXLER R. Root mean square error (RMSE) or mean absolute error (MAE)? - Arguments against avoiding RMSE in the literature [J]. Geoscientific Model Development Discussions, 2014, 7(1): 1525–1534.

    Google Scholar 

  58. de MYTTENAERE A, GOLDEN B, le GRAND B, et al. Mean absolute percentage error for regression models [J]. Neurocomputing, 2016, 192: 38–48. DOI: https://doi.org/10.1016/j.neucom.2015.12.114.

    Article  Google Scholar 

  59. YANG Rui, LIU Hui, LI Yan-fei. Quantifying uncertainty of marine water quality forecasts for environmental management using a dynamic multi-factor analysis and multiresolution ensemble approach [J]. Chemosphere, 2023, 331: 138831. DOI: https://doi.org/10.1016/j.chemosphere.2023.138831.

    Article  Google Scholar 

  60. YANG Rui, LIU Hui, LI Yan-fei. An ensemble self-learning framework combined with dynamic model selection and divide-conquer strategies for carbon emissions trading price forecasting [J]. Chaos, Solitons & Fractals, 2023, 173: 113692. DOI: https://doi.org/10.1016/j.chaos.2023.113692.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, Tsinghua University, Beijing, 100084, China

    Fei Peng  (彭飞) & Li Zheng  (郑力)

  2. CRRC Academy Co., Ltd., Beijing, 100070, China

    Fei Peng  (彭飞)

  3. Institute of Artificial Intelligence & Robotics (IAIR), Key Laboratory of Traffic Safety on Track of Ministry of Education, School of Traffic and Transportation Engineering, Central South University, Changsha, 410075, China

    Hui Liu  (刘辉)

Authors
  1. Fei Peng  (彭飞)
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hui Liu  (刘辉)
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Li Zheng  (郑力)
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

PENG Fei provided the methodology and wrote the first draft of the manuscript. LIU Hui conducted the validation and edited the draft of manuscript. ZHENG Li conducted the conceptualization and edited the draft of manuscript.

Corresponding author

Correspondence to Li Zheng  (郑力).

Ethics declarations

PENG Fei, LIU Hui, and ZHENG Li declare that they have no conflict of interest.

Additional information

Foundation item: Project(Z211100002121140) supported by the Beijing New Star Program of Science and Technology, China; Project (72188101) supported by the National Natural Science Foundation of China

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Peng, F., Liu, H. & Zheng, L. A Sarsa reinforcement learning hybrid ensemble method for robotic battery power forecasting. J. Cent. South Univ. 30, 3867–3880 (2023). https://doi.org/10.1007/s11771-023-5451-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11771-023-5451-0

Key words

关键词

Search

Navigation