Skip to main content
SpringerLink
Log in

DIMA: Distributed cooperative microservice caching for internet of things in edge computing by deep reinforcement learning

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

The ubiquitous Internet of Things (IoTs) devices spawn growing mobile services of applications with computationally-intensive and latency-sensitive features, which increases the data traffic sharply. Driven by container technology, microservice is emerged with flexibility and scalability by decomposing one service into several independent lightweight parts. To improve the quality of service (QoS) and alleviate the burden of the core network, caching microservices at the edge of networks empowered by the mobile edge computing (MEC) paradigm is envisioned as a promising approach. However, considering the stochastic retrieval requests of IoT devices and time-varying network topology, it brings challenges for IoT devices to decide the caching node selection and microservice replacement independently without complete information of dynamic environments. In light of this, a MEC-enabled di stributed cooperative m icroservice ca ching scheme, named DIMA, is proposed in this paper. Specifically, the microservice caching problem is modeled as a Markov decision process (MDP) to optimize the fetching delay and hit ratio. Moreover, a distributed double dueling deep Q-network (D3QN) based algorithm is proposed, by integrating double DQN and dueling DQN, to solve the formulated MDP, where each IoT device performs actions independently in a decentralized mode. Finally, extensive experimental results are demonstrated that the DIMA is well-performed and more effective than existing baseline schemes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Bi, R., Liu, Q., Ren, J., Tan, G.: Utility aware offloading for mobile-edge computing. Tsinghua Sci. Technol. 26(2), 239–250 (2020)

    Article  Google Scholar 

  2. Bi, S., Huang, L., Zhang, Y.-J.A.: Joint optimization of service caching placement and computation offloading in mobile edge computing systems. IEEE Trans. Wirel. Commun. 19(7), 4947–4963 (2020)

    Article  Google Scholar 

  3. Chen, H., Zhang, Y., Cao, Y., Xie, J.: Security issues and defensive approaches in deep learning frameworks. Tsinghua Sci. Technol. 26(6), 894–905 (2021)

    Article  Google Scholar 

  4. Chen, L., Song, L., Chakareski, J., Jie, X.: Collaborative content placement among wireless edge caching stations with time-to-live cache. IEEE Trans. Multimed. 22(2), 432–444 (2019)

    Article  Google Scholar 

  5. Chen, S., Yao, Z., Jiang, X., Yang, J., Hanzo, L.: Multi-agent deep reinforcement learning-based cooperative edge caching for ultra-dense next-generation networks. IEEE Trans. Commun 69(4), 2441–2456 (2020)

    Article  Google Scholar 

  6. Hasselt, H., Guez, A., Silver, D.: Deep reinforcement learning with double q-learning. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence AAAI’16, pp 2094–2100. AAAI Press (2016)

  7. He, Q., Cui, G., Zhang, X., Chen, F., Deng, S., Jin, H., Li, Y., Yang, Y.: A game-theoretical approach for user allocation in edge computing environment. IEEE Trans. Parall. Distrib. Syst. 31(3), 515–529 (2019)

    Article  Google Scholar 

  8. Hong, Z., Chen, W., Huang, H., Guo, S., Zheng, Z.: Multi-hop cooperative computation offloading for industrial iot–edge–cloud computing environments. IEEE Trans. Parall. Distrib. Syst. 30(12), 2759–2774 (2019)

    Article  Google Scholar 

  9. Hu, S., Guanghui, L.: Dynamic request scheduling optimization in mobile edge computing for iot applications. IEEE Internet Things J. 7(2), 1426–1437 (2019)

    Article  Google Scholar 

  10. Jaramillo, D., Nguyen, D.V, Smart, R.: Leveraging microservices architecture by using docker technology. In: SoutheastCon 2016, pp 1–5. IEEE (2016)

  11. Kim, J., Kim, T., Hashemi, M., Brinton, C.G, Love, D.J: Joint optimization of signal design and resource allocation in wireless d2d edge computing. In: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp 2086–2095. IEEE (2020)

  12. Lei, L., Xu, H., Xiong, X., Zheng, K., Xiang, W.: Joint computation offloading and multiuser scheduling using approximate dynamic programming in nb-iot edge computing systemxc. IEEE Internet Things J. 6(3), 5345–5362 (2019)

    Article  Google Scholar 

  13. Li, S., Li Da, X., Zhao, S.: 5g internet of things A survey. J. Industr. Inform. Integr. 10, 1–9 (2018)

    Google Scholar 

  14. Liu, Y., Zeng, Z., Liu, X., Zhu, X., Bhuiyan, M.Z.A.: A novel load balancing and low response delay framework for edge-cloud network based on sdn. IEEE Internet Things J. 7(7), 5922–5933 (2019)

    Article  Google Scholar 

  15. Mabrouki, J., Azrour, M., Dhiba, D., Farhaoui, Y., Hajjaji, S.E.: Iot-based data logger for weather monitoring using arduino-based wireless sensor networks with remote graphical application and alerts. Big Data Mining Analytics 4(1), 25–32 (2021)

    Article  Google Scholar 

  16. Malek, Y.N., Najib, M., Bakhouya, M., Essaaidi, M.: Multivariate deep learning approach for electric vehicle speed forecasting. Big Data Mining Analytics 4(1), 56–64 (2021)

    Article  Google Scholar 

  17. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning. arXiv:1312.5602 (2013)

  18. Ning, Z., Dong, P., Wang, X., Xiping, H., Guo, L., Bin, H., Yi, G., Qiu, T., Kwok, RYK: Mobile edge computing enabled 5g health monitoring for internet of medical things A decentralized game theoretic approach. IEEE J. Select. Areas Commun. 39(2), 463–478 (2021)

    Article  Google Scholar 

  19. Prabadevi, B, Deepa, N, Pham, Q-V, Nguyen, DC., Praveen Kumar Reddy, M, Thippa Reddy, G, Pathirana, P.N., Dobre, O.: Toward blockchain for edge-of-things A new paradigm, opportunities, and future directions. IEEE Internet Things Magaz. 4(2), 102–108 (2021)

    Article  Google Scholar 

  20. Pingyang, W., Li, J., Shi, L., Ding, M., Cai, K., Yang, F.: Dynamic content update for wireless edge caching via deep reinforcement learning. IEEE Commun. Lett. 23(10), 1773–1777 (2019)

    Article  Google Scholar 

  21. Qi, H., Birman, K., Renesse, R.V., Lloyd, W., Kumar, S., Li, HC: An analysis of facebook photo caching. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP ’13, pp 167–181 (2013)

  22. Rafique, W., Qi, L., Yaqoob, I., Imran, M., Rasool, R.U., Dou, W.: Complementing iot services through software defined networking and edge computing: A comprehensive survey. IEEE Commun. Surv. Tutor. 22(3), 1761–1804 (2020)

    Article  Google Scholar 

  23. Ren, Z., Liu, Y., Shi, T., Xie, L., Zhou, Y., Zhai, J., Zhang, Y., Zhang, Y., Chen, W.: Aiperf: Automated machine learning as an ai-hpc benchmark. Big Data Mining Analytics 4(3), 208–220 (2021)

    Article  Google Scholar 

  24. Samanta, A., Tang, J.: Dyme: Dynamic microservice scheduling in edge computing enabled iot. IEEE Internet Things J. 7(7), 6164–6174 (2020)

    Article  Google Scholar 

  25. Shi, W., Cao, J., Zhang, Q., Li, Y., Xu, L.: Edge computing: Vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)

    Article  Google Scholar 

  26. Song, C., Xu, W., Wu, T., Yu, S., Zeng, P., Zhang, N.: Qoe-driven edge caching in vehicle networks based on deep reinforcement learning. IEEE Trans. Veh. Technol. 70(6), 5286–5295 (2021)

    Article  Google Scholar 

  27. Tang, L., Huang, Q., Lloyd, W., Kumar, S., Li, K.: {RIPQ}: Advanced photo caching on flash for facebook. In: 13th {USENIX} Conference on File and Storage Technologies ({FAST} 15), pp 373–386. USENIX Association (2015)

  28. Tong, Z., Ye, F., Yan, M., Liu, H., Basodi, S.: A survey on algorithms for intelligent computing and smart city applications. Big Data Mining Analytics 4(3), 155–172 (2021)

    Article  Google Scholar 

  29. Wang, Y., He, Q., Ye, D., Yang, Y.: Formulating criticality-based cost-effective fault tolerance strategies for multi-tenant service-based systems. IEEE Trans. Softw. Eng. 44(3), 291–307 (2017)

    Article  Google Scholar 

  30. Wang, S., Guo, Y., Zhang, N., Yang, P., Zhou, A., Shen, X.S.: Delay-aware microservice coordination in mobile edge computing: A reinforcement learning approach. IEEE Trans. Mobile Comput. 20(3), 939–951 (2021)

    Article  Google Scholar 

  31. Wang, X., Li, R., Wang, C., Li, X., Taleb, T., Leung, Victor CM: Attention-weighted federated deep reinforcement learning for device-to-device assisted heterogeneous collaborative edge caching. IEEE J. Select. Areas Commun. 39(1), 154–169 (2020)

    Article  Google Scholar 

  32. Wang, Z., Schaul, T., Hessel, M., Hasselt, H., Lanctot, M., Freitas, N.: Dueling network architectures for deep reinforcement learning. In: International Conference on Machine Learning, pp 1995–2003. PMLR (2016)

  33. Wei, D., Ning, H., Shi, F., Wan, Y., Xu, J., Yang, S., Zhu, L.: Dataflow management in the internet of things: Sensing, control, and security. Tsinghua Sci. Technol. 26(6), 918–930 (2021)

    Article  Google Scholar 

  34. Xia, X., Chen, F., He, Q., Grundy, J.C, Abdelrazek, M., Jin, H.: Cost-effective app data distribution in edge computing. IEEE Trans. Parall. Distrib. Syst 32(1), 31–44 (2020)

    Article  Google Scholar 

  35. Xie, Z., Chen, W.: Storage-efficient edge caching with asynchronous user requests. IEEE Trans. Cognit. Commun. Netw. 6(1), 229–241 (2019)

    Article  MathSciNet  Google Scholar 

  36. Xiong, X., Zheng, K., Lei, L., Hou, L u: Resource allocation based on deep reinforcement learning in iot edge computing. IEEE J. Select. Areas Commun. 38(6), 1133–1146 (2020)

    Article  Google Scholar 

  37. Xu, X., Xihua, L., Xu, Z., Fei, D., Xuyun, Z., Lianyong, Q.: Trust-oriented iot service placement for smart cities in edge computing. IEEE Internet Things J. 7(5), 4084–4091 (2019)

    Article  Google Scholar 

  38. Xu, Z., Wang, S., Liu, S., Dai, H., Xia, Q., Liang, W., Wu, G.: Learning for exception: Dynamic service caching in 5g-enabled mecs with bursty user demands. In: 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pp 1079–1089. IEEE (2020)

  39. Zhang, K., Cao, J., Liu, H., Maharjan, S., Zhang, Y.: Deep reinforcement learning for social-aware edge computing and caching in urban informatics. IEEE Trans. Industr. Informat. 16(8), 5467–5477 (2019)

    Article  Google Scholar 

  40. Zhang, T., Fang, X., Liu, Y., Li, G.Y., Xu, W.: D2d-enabled mobile user edge caching: A multi-winner auction approach. IEEE Trans. Veh. Technol. 68(12), 12314–12328 (2019)

    Article  Google Scholar 

  41. Zhang, W., Chen, X., Jiang, J.: A multi-objective optimization method of initial virtual machine fault-tolerant placement for star topological data centers of cloud systems. Tsinghua Sci. Technol. 26(1), 95–111 (2020)

    Article  Google Scholar 

  42. Zhang, Y., Meng, L., Xue, X., Zhou, Z., Tomiyama, H.: Qoe-constrained concurrent request optimization through collaboration of edge servers. IEEE Internet Things J. 6(6), 9951–9962 (2019)

    Article  Google Scholar 

  43. Zhong, C., Gursoy, M, Velipasalar, S.: Deep reinforcement learning-based edge caching in wireless networks. IEEE Trans. Cogn. Commun. Netw. 6 (1), 48–61 (2020)

    Article  Google Scholar 

  44. Zhou, X., Liang, W., She, J., Yan, Z., Wang, K.I.-K.: Two-layer federated learning with heterogeneous model aggregation for 6g supported internet of vehicles. IEEE Trans. Veh. Technol. 70(6), 5308–5317 (2021)

    Article  Google Scholar 

  45. Zhou, X., Liang, W., Shimizu, S., Ma, J., Jin, Q.: Siamese neural network based few-shot learning for anomaly detection in industrial cyber-physical systems. IEEE Trans. Industr. Inform. 17(8), 5790–5798 (2020)

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by the Financial and Science Technology Plan Project of Xinjiang Production and Construction Corps under grant no. 2020DB005. This research is also supported by the National Natural Science Foundation of China under grant no.41975183 and 62173023.

Author information

Authors and Affiliations

  1. School of Computer and Software, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Hao Tian & Xiaolong Xu

  2. Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology, Nanjing, China

    Xiaolong Xu

  3. Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology (CICAEET), Nanjing University of Information Science and Technology and Engineering, Nanjing, 210044, China

    Xiaolong Xu

  4. Provincial Key Laboratory for Computer Information Processing Technology, Soochow University, Suzhou, China

    Xiaolong Xu

  5. State Key Laboratory Novel Software Technology, Nanjing University, Nanjing, China

    Xiaolong Xu

  6. State Key Laboratory of Complex Product Intelligent Manufacturing System Technology, Beijing Institute of Electronic System Engineering, Beijing, China

    Tingyu Lin

  7. School of Automation, Nanjing University of Information Science and Technology, Nanjing, 210044, China

    Yong Cheng

  8. Jiangsu Hydraulic Research Institute, Nanjing, 210017, China

    Cheng Qian

  9. School of Automation Science and Electrical Engineering, Beihang University, Beijing, China

    Lei Ren

  10. Depatment of Computer and Electronics Systems Engineering, Hankuk University of Foreign Studies, Yongin-si, Gyeonggi-do, 17035, Korea

    Muhammad Bilal

Authors
  1. Hao Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaolong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tingyu Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Cheng Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lei Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Muhammad Bilal
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaolong Xu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Resource Management at the Edge for Future Web, Mobile and IoT Applications Guest Editors: Qiang He, Fang Dong, Chenshu Wu, and Yun Yang

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, H., Xu, X., Lin, T. et al. DIMA: Distributed cooperative microservice caching for internet of things in edge computing by deep reinforcement learning. World Wide Web 25, 1769–1792 (2022). https://doi.org/10.1007/s11280-021-00939-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00939-7

Keywords

Search

Navigation