Skip to main content
SpringerLink
Log in

Deep reinforcement learning-based microservice selection in mobile edge computing

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

In mobile edge computing environment, due to resources constraints of edge devices, when user locations continue changing, the network will be delayed or interrupted, which affects the quality of user’s service access. Previous studies have shown that deploying multiple microservice instances with the same function on multiple edge servers through container technology can solve this problem. However, how to choose the optimal microservice instance from multiple servers in a cloud-edge hybrid environment needs to be further investigated. This paper studies the selection of microservices problem based on the dynamic and heterogeneous characters of the cloud-edge collaborative environment, which is defined as a microservice selection and scheduling optimization problem (MSSP) to minimize users’ service access delay. To cope with the complexity of cloud-edge collaborative environment and improve learning efficiency, MSSP is regarded as a Markov decision-making process, a Deep Deterministic Policy Gradient algorithm for microservice selection called MS_DDPG is then proposed to solve this problem, and the microservice selection strategy experience pool is established in MS_DDPG. Performance evaluations of MS_DDPG based on a real dataset and some synthetic dataset have been conducted, and the results show that MS_DDPG outperforms the other three baseline algorithms. In terms of average access delay, MS_DDPG is reduced by 23.82%. We also validate the performance of MS_DDPG by increasing the number of user requests, and the results also show that MS_DDPG obtains better performance in scalability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Zhang, Y.: Mobile Edge Computing, vol. 9. Springer, Cham (2022)

    Google Scholar 

  2. Tran, T.X., Hajisami, A., Pandey, P., Pompili, D.: Collaborative mobile edge computing in 5g networks: new paradigms, scenarios, and challenges. IEEE Commun. Mag. 55(4), 54–61 (2017). https://doi.org/10.1109/MCOM.2017.1600863

    Article  Google Scholar 

  3. Chen, L., Kuang, X., Deng, D., Zhu, F., Xia, J., Fan, L.: Multi-cap assisted intelligent mobile edge computing networks for internet of things. IEEE Access 8, 137235–137243 (2020). https://doi.org/10.1109/ACCESS.2020.3009686

    Article  Google Scholar 

  4. Tang, B., Kang, L.: Eicache: a learning-based intelligent caching strategy in mobile edge computing. Peer-to-Peer Netw. Appl. 15(2), 934–949 (2022). https://doi.org/10.1007/s12083-021-01266-4

    Article  Google Scholar 

  5. Bagchi, S., Abdelzaher, T.F., Govindan, R., Shenoy, P.J., Atrey, A., Ghosh, P., Xu, R.: New frontiers in iot: networking, systems, reliability, and security challenges. IEEE Internet Things J. 7(12), 11330–11346 (2020). https://doi.org/10.1109/JIOT.2020.3007690

    Article  Google Scholar 

  6. Li, X., He, J., Vijayakumar, P., Zhang, X., Chang, V.: A verifiable privacy-preserving machine learning prediction scheme for edge-enhanced hcpss. IEEE Trans. Ind. Inform. 18(8), 5494–5503 (2022). https://doi.org/10.1109/TII.2021.3110808

    Article  Google Scholar 

  7. Liang, W., Tang, M., Long, J., Peng, X., Xu, J., Li, K.: A secure fabric blockchain-based data transmission technique for industrial internet-of-things. IEEE Trans. Ind. Inform. 15(6), 3582–3592 (2019). https://doi.org/10.1109/TII.2019.2907092

    Article  Google Scholar 

  8. Xing, L.: Reliability in internet of things: current status and future perspectives. IEEE Internet Things J. 7(8), 6704–6721 (2020). https://doi.org/10.1109/JIOT.2020.2993216

    Article  Google Scholar 

  9. Whaiduzzaman, M., Mahi, M.J.N., Barros, A., Khalil, M.I., Fidge, C.J., Buyya, R.: BFIM: performance measurement of a blockchain based hierarchical tree layered fog-iot microservice architecture. IEEE Access 9, 106655–106674 (2021). https://doi.org/10.1109/ACCESS.2021.3100072

    Article  Google Scholar 

  10. Aksakalli, I.K., Çelik, T., Can, A.B., Tekinerdogan, B.: Deployment and communication patterns in microservice architectures: a systematic literature review. J. Syst. Softw. 180, 111014 (2021). https://doi.org/10.1016/j.jss.2021.111014

    Article  Google Scholar 

  11. Guo, F., Tang, B., Tang, M.: Joint optimization of delay and cost for microservice composition in mobile edge computing. World Wide Web (2022). https://doi.org/10.1007/s11280-022-01017-2

    Article  Google Scholar 

  12. Hannousse, A., Yahiouche, S.: Securing microservices and microservice architectures: a systematic mapping study. Comput. Sci. Rev. 41, 100415 (2021). https://doi.org/10.1016/j.cosrev.2021.100415

    Article  Google Scholar 

  13. Henning, S., Hasselbring, W.: Theodolite: scalability benchmarking of distributed stream processing engines in microservice architectures. Big Data Res. 25, 100209 (2021). https://doi.org/10.1016/j.bdr.2021.100209

    Article  Google Scholar 

  14. Chen, L., Xu, Y., Lu, Z., Wu, J., Gai, K., Hung, P.C.K., Qiu, M.: Iot microservice deployment in edge-cloud hybrid environment using reinforcement learning. IEEE Internet Things J. 8(16), 12610–12622 (2021). https://doi.org/10.1109/JIOT.2020.3014970

    Article  Google Scholar 

  15. Guo, F., Tang, B., Tang, M., Zhao, H., Liang, W.: Microservice selection in edge-cloud collaborative environment: a deep reinforcement learning approach. In: 8th IEEE International Conference on Cyber Security and Cloud Computing, CSCloud 2021/7th IEEE International Conference on Edge Computing and Scalable Cloud, EdgeCom 2021, Washington, DC, USA, June 26–28, 2021, pp. 24–29. IEEE (2021). https://doi.org/10.1109/CSCloud-EdgeCom52276.2021.00015

  16. Tang, B., Fedak, G.: Wukastore: scalable, configurable and reliable data storage on hybrid volunteered cloud and desktop systems. IEEE Trans. Big Data 8(1), 85–98 (2022). https://doi.org/10.1109/TBDATA.2017.2758791

    Article  Google Scholar 

  17. Fan, G., Chen, L., Yu, H., Qi, W.: Multi-objective optimization of container-based microservice scheduling in edge computing. Comput. Sci. Inf. Syst. 18(1), 23–42 (2021). https://doi.org/10.2298/CSIS200229041F

    Article  Google Scholar 

  18. Wang, S., Guo, Y., Zhang, N., Yang, P., Zhou, A., Shen, X.: Delay-aware microservice coordination in mobile edge computing: a reinforcement learning approach. IEEE Trans. Mob. Comput. 20(3), 939–951 (2021). https://doi.org/10.1109/TMC.2019.2957804

    Article  Google Scholar 

  19. Zou, G., Qin, Z., Deng, S., Li, K., Gan, Y., Zhang, B.: Towards the optimality of service instance selection in mobile edge computing. Knowl. Based Syst. 217, 106831 (2021). https://doi.org/10.1016/j.knosys.2021.106831

    Article  Google Scholar 

  20. Tang, M., Liang, W., Yang, Y., Xie, J.: A factorization machine-based qos prediction approach for mobile service selection. IEEE Access 7, 32961–32970 (2019). https://doi.org/10.1109/ACCESS.2019.2902272

    Article  Google Scholar 

  21. Zheng, Z., Xiaoli, L., Tang, M., Xie, F., Lyu, M.R.: Web service qos prediction via collaborative filtering: a survey. IEEE Trans. Serv. Comput. (2020). https://doi.org/10.1109/TSC.2020.2995571

    Article  Google Scholar 

  22. Hwang, S., Hsu, C., Lee, C.: Service selection for web services with probabilistic qos. IEEE Trans. Serv. Comput. 8(3), 467–480 (2015). https://doi.org/10.1109/TSC.2014.2338851

    Article  Google Scholar 

  23. Zhang, H., Yang, N., Xu, Z., Tang, B., Ma, H.: Microservice based video cloud platform with performance-aware service path selection. In: 2018 IEEE International Conference on Web Services, ICWS 2018, San Francisco, CA, USA, July 2–7, 2018, pp. 306–309. IEEE (2018). https://doi.org/10.1109/ICWS.2018.00048

  24. Wu, Q., Zhou, M., Zhu, Q., Xia, Y.: VCG auction-based dynamic pricing for multigranularity service composition. IEEE Trans. Autom. Sci. Eng. 15(2), 796–805 (2018). https://doi.org/10.1109/TASE.2017.2695123

    Article  Google Scholar 

  25. Samanta, A., Tang, J.: Dyme: dynamic microservice scheduling in edge computing enabled iot. IEEE Internet Things J. 7(7), 6164–6174 (2020). https://doi.org/10.1109/JIOT.2020.2981958

    Article  Google Scholar 

  26. Lin, M., Xi, J., Bai, W., Wu, J.: Ant colony algorithm for multi-objective optimization of container-based microservice scheduling in cloud. IEEE Access 7, 83088–83100 (2019). https://doi.org/10.1109/ACCESS.2019.2924414

    Article  Google Scholar 

  27. Zhao, H., Deng, S., Liu, Z., Yin, J., Dustdar, S.: Distributed redundancy scheduling for microservice-based applications at the edge. CoRR abs/1911.03600 (2019)

  28. Wang, S., Ding, Z., Jiang, C.: Elastic scheduling for microservice applications in clouds. IEEE Trans. Parallel Distrib. Syst. 32(1), 98–115 (2021). https://doi.org/10.1109/TPDS.2020.3011979

    Article  Google Scholar 

  29. Freire, A.F.A.A., Sampaio, A.F., Carvalho, L.H.L., Medeiros, O., Mendonça, N.C.: Migrating production monolithic systems to microservices using aspect oriented programming. Softw. Pract. Exp. 51(6), 1280–1307 (2021). https://doi.org/10.1002/spe.2956

    Article  Google Scholar 

  30. Rui, L., Zhang, M., Gao, Z., Qiu, X., Wang, Z., Xiong, A.: Service migration in multi-access edge computing: a joint state adaptation and reinforcement learning mechanism. J. Netw. Comput. Appl. 183–184, 103058 (2021). https://doi.org/10.1016/j.jnca.2021.103058

    Article  Google Scholar 

  31. Liang, Z., Liu, Y., Lok, T., Huang, K.: Multi-cell mobile edge computing: joint service migration and resource allocation. IEEE Trans. Wirel. Commun. 20(9), 5898–5912 (2021). https://doi.org/10.1109/TWC.2021.3070974

    Article  Google Scholar 

  32. Ray, K., Banerjee, A., Narendra, N.C.: Proactive microservice placement and migration for mobile edge computing. In: 5th IEEE/ACM Symposium on Edge Computing, SEC 2020, San Jose, CA, USA, November 12–14, 2020, pp. 28–41. IEEE (2020). https://doi.org/10.1109/SEC50012.2020.00010

  33. Li, C., Ma, S., Lu, T.: Microservice migration using strangler fig pattern: A case study on the green button system. In: International Computer Symposium, ICS 2020, Tainan, Taiwan, December 17–19, 2020, pp. 519–524. IEEE (2020). https://doi.org/10.1109/ICS51289.2020.00107

  34. Chen, Y., Deng, S., Ma, H., Yin, J.: Deploying data-intensive applications with multiple services components on edge. Mob. Netw. Appl. 25(2), 426–441 (2020). https://doi.org/10.1007/s11036-019-01245-3

    Article  Google Scholar 

  35. Lai, P., He, Q., Abdelrazek, M., Chen, F., Hosking, J.G., Grundy, J.C., Yang, Y.: Optimal edge user allocation in edge computing with variable sized vector bin packing. CoRR abs/1904.05553 (2019)

  36. Tang, L., Tang, B., Zhang, L., Guo, F., He, H.: Joint optimization of network selection and task offloading for vehicular edge computing. J. Cloud Comput. 10(1), 23 (2021). https://doi.org/10.1186/s13677-021-00240-y

    Article  Google Scholar 

  37. Yang, X., Fei, Z., Zheng, J., Zhang, N., Anpalagan, A.: Joint multi-user computation offloading and data caching for hybrid mobile cloud/edge computing. IEEE Trans. Veh. Technol. 68(11), 11018–11030 (2019). https://doi.org/10.1109/TVT.2019.2942334

    Article  Google Scholar 

  38. Gai, K., Qiu, M.: Optimal resource allocation using reinforcement learning for iot content-centric services. Appl. Soft Comput. 70, 12–21 (2018). https://doi.org/10.1016/j.asoc.2018.03.056

    Article  Google Scholar 

  39. Gai, K., Qiu, M.: Reinforcement learning-based content-centric services in mobile sensing. IEEE Netw. 32(4), 34–39 (2018). https://doi.org/10.1109/MNET.2018.1700407

    Article  Google Scholar 

  40. Cui, G., He, Q., Chen, F., Jin, H., Yang, Y.: Trading off between user coverage and network robustness for edge server placement. IEEE Trans. Cloud Comput. (2020). https://doi.org/10.1109/TCC.2020.3008440

    Article  Google Scholar 

  41. Lai, P., He, Q., Grundy, J., Chen, F., Abdelrazek, M., Hosking, J.G., Yang, Y.: Cost-effective app user allocation in an edge computing environment. IEEE Trans. Cloud Comput. (2020). https://doi.org/10.1109/TCC.2020.3001570

    Article  Google Scholar 

  42. Guo, J., Chang, Z., Wang, S., Ding, H., Feng, Y., Mao, L., Bao, Y.: Who limits the resource efficiency of my datacenter: an analysis of alibaba datacenter traces. In: Proceedings of the International Symposium on Quality of Service, IWQoS 2019, Phoenix, AZ, USA, June 24–25, 2019, pp. 39:1–39:10. ACM (2019). https://doi.org/10.1145/3326285.3329074

Download references

Acknowledgements

The authors would like to thank all the reviewers for their helpful comments. A preliminary version of this paper was presented at the 8th IEEE International Conference on Cyber Security and Cloud Computing (CSCloud 2021), and part of data has been published [15]. This paper is substantially extended with new content being added.

Funding

This work is supported by National Key R &D Program of China (No. 2018YFB1402800), National Natural Science Foundation of China (No. 61872138 and 61602169), and the Natural Science Foundation of Hunan Province (No. 2021JJ30278).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Hunan University of Science and Technology, Xiangtan, 411201, China

    Feiyan Guo, Bing Tang & Wei Liang

  2. Hunan Key Laboratory for Service Computing and Novel Software Technology, Hunan University of Science and Technology, Xiangtan, 411201, China

    Feiyan Guo, Bing Tang & Wei Liang

  3. Hunan Electrical College of Technology, Xiangtan, 411101, China

    Feiyan Guo

  4. School of Information Science and Technology, Guangdong University of Foreign Studies, Guangzhou, 510006, China

    Mingdong Tang

Authors
  1. Feiyan Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bing Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingdong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wei Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization: BT; Methodology: FG; Formal analysis and investigation: FG; Validation: BT, MT; Writing—original draft preparation: FG; Writing—review and editing: BT, WL; Funding acquisition: BT; Resources: MT; Supervision: WL.

Corresponding author

Correspondence to Bing Tang.

Ethics declarations

Conflict of interest

The authors declared that they have no conflict of interest to this work.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, F., Tang, B., Tang, M. et al. Deep reinforcement learning-based microservice selection in mobile edge computing. Cluster Comput 26, 1319–1335 (2023). https://doi.org/10.1007/s10586-022-03661-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-022-03661-9

Keywords

Search

Navigation