Skip to main content

Advertisement

SpringerLink
Log in

Reinforcement learning based distributed resource allocation technique in device-to-device (D2D) communication

  • Original Paper
  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Device to Device communication (D2D) is a promising and one of the technologies that is envisioned to be significant one in the future wireless communication systems. The fifth-generation and beyond wireless networks deal with a numerous number of devices connected to each other that demands for higher data rates at a very low latency. To meet these demands in the future, in this paper, we have considered optimizing the system throughput and fairness of an underlay D2D communication system by a device-centric joint spectrum and power allocation schemes. Due to the unknown and dynamic nature of the channel parameters, we propose a distributed reinforcement learning (RL)-based iterative resource allocation scheme that performs more efficiently compared to the traditional optimization methods. RL enables the system to learn about the environment on its own by the trial-and-error method without any prior knowledge or information about the same. We expand the observation space by integrating a set of states with the required amount of system specific variables thereby improving the accuracy of the learning algorithm. The graphical simulation results demonstrate the improvement of the throughput, energy efficiency (EE) and spectrum efficiency (SE) and fairness of the system. The competence of the proposed method in terms of the sum reward is also shown in comparison with the conventional distributed and random allocation methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Dogra, A., Jha, R. K., & Jain, S. (2021). A survey on beyond 5G network with the advent of 6G: Architecture and emerging technologies. IEEE Access, 9, 67512–67547. https://doi.org/10.1109/ACCESS.2020.3031234

    Article  Google Scholar 

  2. Xu, Y., Gui, G., Gacanin, H., & Adachi, F. (2021). A survey on resource allocation for 5G heterogeneous networks: current research, future trends, and challenges. IEEE Communications Surveys and Tutorials, 23(2), 668–695. https://doi.org/10.1109/COMST.2021.3059896

    Article  Google Scholar 

  3. Pedhadiya, M. K., Jha, R. K., & Bhatt, H. G. (2019). Device to device communication: A survey. Journal of Network and Computer Applications. https://doi.org/10.1016/j.jnca.2018.10.012

    Article  Google Scholar 

  4. Chakraborty, C., & Rodrigues, J. J. C. P. (2020). A comprehensive review on device-to-device communication paradigm: trends, challenges and applications. Wireless Personal Communications. https://doi.org/10.1007/s11277-020-07358-3

    Article  Google Scholar 

  5. Y. Wang, W. Ning, S. Zhang, H. Yu, H. Cen, and S. Wang, “Architecture and key terminal technologies of 5G-based internet of vehicles,” Computers and Electrical Engineering, vol. 95, Oct. 2021, https://doi.org/10.1016/j.compeleceng.2021.107430.

  6. Adnan, M. H., & Zukarnain, Z. A. (2020). Device-to-device communication in 5G environment: Issues, solutions, and challenges. Symmetry, 12(11), 1–22. https://doi.org/10.3390/sym12111762

    Article  Google Scholar 

  7. Kumar, N., Swain, S. N., Siva Ram Murthy, C. (2018). A novel distributed Q-learning based resource reservation framework for facilitating D2D content access requests in LTE-A networks. IEEE Transactions on Network and Service Management, 15(2), 718–731. https://doi.org/10.1109/TNSM.2018.2807594.

  8. Awan, A. Y., Ali, M., Naeem, M., Qamar, F., & Sial, M. N. (2020). Joint network admission control, mode assignment, and power allocation in energy harvesting aided D2D communication. IEEE Transactions on Industrial Informatics. https://doi.org/10.1109/TII.2019.2922667

    Article  Google Scholar 

  9. Siddiqui, M. U. A., Qamar, F., Ahmed, F., Nguyen, Q. N., & Hassan, R. (2021). Interference management in 5G and beyond network: Requirements, challenges and future directions. IEEE Access, 9, 68932–68965. https://doi.org/10.1109/ACCESS.2021.3073543

    Article  Google Scholar 

  10. Zheng, Z., Wang, L., Zhu, F., & Liu, L. (2021). Potential technologies and applications based on deep learning in the 6G networks. Computers and Electrical Engineering, 95. https://doi.org/10.1016/j.compeleceng.2021.107373.

  11. Jayakumar, S., & Nandakumar, S. (2021). A review on resource allocation techniques in D2D communication for 5G and B5G technology. Peer-to-Peer Networking and Applications, 14(1), 243–269. https://doi.org/10.1007/s12083-020-00962-x.

  12. Lai, W. K., Wang, Y. C., Lin, H. C., & Li, J. W. (2020). Efficient resource allocation and power control for LTE-A D2D communication with pure D2D Model. IEEE Transactions on Vehicular Technology, 69(3), 3202–3216. https://doi.org/10.1109/TVT.2020.2964286

    Article  Google Scholar 

  13. Sharma, S., & Singh, B. (2019). Weighted cooperative reinforcement learning-based energy-efficient autonomous resource selection strategy for underlay D2D communication. IET Communications, 13(14), 2078–2087. https://doi.org/10.1049/iet-com.2018.6028

    Article  Google Scholar 

  14. Liu, Z., Li, X., Yuan, Y., & Guan, X. (2020). Power control of D2D communication based on quality of service assurance under imperfect channel information. Peer-to-Peer Networking and Applications, 13, 1327–1339. https://doi.org/10.1007/s12083-019-00864-7

    Article  Google Scholar 

  15. Pandey, K., Arya, R., & Kumar, S. (2021). Lagrange’s multiplier based resource management for energy efficient D2D communication in 5G networks. International Journal of Systems Assurance Engineering and Management. https://doi.org/10.1007/s13198-020-01045-z

    Article  Google Scholar 

  16. Wu, W., Liu, R., Yang, Q., & Quek, T. Q. S. (2022). Learning-based robust resource allocation for D2D underlaying cellular network. IEEE Transactions on Wireless Communications, 21(8), 6731–6745. https://doi.org/10.1109/TWC.2022.3152260

  17. Pandey, K., & Arya, R. (2021). Lyapunov optimization machine learning resource allocation approach for uplink underlaid D2D communication in 5G networks. In: IET Communications, no. July, pp. 1–9, 2021, https://doi.org/10.1049/cmu2.12264.

  18. Vali Mohamad, N.M., Ambastha, P., Gautam, S., Jain, R., Subramaniyam, H., & Muthukaruppan, L. (2021). Dynamic Sectorization and parallel processing for device-to-device (D2D) resource allocation in 5G and B5G cellular network. Peer-to-Peer Networking and Applications, 14, (1), 296–304. https://doi.org/10.1007/s12083-020-00949-8.

  19. Zia, K., Javed, N., Sial, M. N., Ahmed, S., Pirzada, A. A., & Pervez, F. (2019). A distributed multi-agent RL-based autonomous spectrum allocation scheme in D2D enabled multi-tier HetNets. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2890210

    Article  Google Scholar 

  20. Liu, Z., Han, X., Xie, Y., Yuan, Y., & Chan, K. Y. (2021). Energy-efficiency maximization in D2D-enabled vehicular communications with consideration of dynamic channel information and fairness. Peer-to-Peer Networking and Applications, 14, (1), 164–176. https://doi.org/10.1007/s12083-020-00970-x.

  21. Li, Z., & Guo, C. (2020). Multi-agent deep reinforcement learning based spectrum allocation for D2D underlay communications. IEEE Transactions on Vehicular Technology, 69(2), 1828–1840. https://doi.org/10.1109/TVT.2019.2961405

    Article  Google Scholar 

  22. Tan, J., Liang, Y. C., Zhang, L., & Feng, G. (2021). Deep reinforcement learning for joint channel selection and power control in D2D networks. IEEE Transactions on Wireless Communications, 20(2), 1363–1378. https://doi.org/10.1109/TWC.2020.3032991

    Article  Google Scholar 

  23. Zhang, T., Zhu, K., & Wang, J. (2021). Energy-efficient mode selection and resource allocation for D2D-enabled heterogeneous networks: a deep reinforcement learning approach. IEEE Transactions on Wireless Communications, 20(2), 1175–1187. https://doi.org/10.1109/TWC.2020.3031436

    Article  Google Scholar 

  24. Wang, L., Liu, J., Chen, M., Gui, G., & Sari, H. (2018). Optimization-based access assignment scheme for physical-layer security in D2D communications underlaying a cellular network. IEEE Transactions on Vehicular Technology, 67(7), 5766–5777. https://doi.org/10.1109/TVT.2017.2789022

    Article  Google Scholar 

  25. Al-Abiad, M. S., Hassan, M. Z., & Hossain, M. J. (2022). A joint reinforcement-learning enabled caching and cross-layer network code in F-RAN With D2D communications. IEEE Transactions on Communications, 70(7), 4400–4416. https://doi.org/10.1109/TCOMM.2022.3168058

    Article  Google Scholar 

  26. Yang, Y., Feng, L., Zhang, C., Ou, Q., & Li, W. (2020). Resource allocation for virtual reality content sharing based on 5G D2D multicast communication. EURASIP Journal on Wireless Communications and Networking, 1, 2020. https://doi.org/10.1186/s13638-020-01690-9

    Article  Google Scholar 

  27. Hussain, F., Hassan, M. Y., Hossen, M. S., & Choudhury, S. (2018). System capacity maximization with efficient resource allocation algorithms in D2D communication. IEEE Access. https://doi.org/10.1109/ACCESS.2018.2839190

    Article  Google Scholar 

  28. Xu, J., & Guo, C. (2019). Resource allocation for real-time D2D communications underlaying cellular networks. IEEE Transactions on Mobile Computing. https://doi.org/10.1109/TMC.2018.2849743

    Article  Google Scholar 

  29. Ahmad, M., et al. (2020). Device-centric communication in IoT: An energy efficiency perspective. Transactions on Emerging Telecommunications Technologies. https://doi.org/10.1002/ett.3750

    Article  Google Scholar 

  30. Kaur, J., Khan, M. A., Iftikhar, M., Imran, M., & Emad Ul Haq, Q. (2021). Machine Learning Techniques for 5G and beyond. IEEE Access, 9, 23472–23488. https://doi.org/10.1109/ACCESS.2021.3051557.

  31. Fourati, H., Maaloul, R., & Chaari, L. (2021). A survey of 5G network systems: challenges and machine learning approaches, vol. 12, no. 2. Springer. https://doi.org/10.1007/s13042-020-01178-4.

  32. Morocho-Cayamcela, M. E., Lee, H., & Lim, W. (2019). Machine learning for 5G/B5G mobile and wireless communications: Potential, limitations, and future directions. IEEE Access, 7, 137184–137206. https://doi.org/10.1109/ACCESS.2019.2942390

    Article  Google Scholar 

  33. Jagannath, J., Polosky, N., Jagannath, A., Restuccia, F., & Melodia, T. (2019). Machine learning for wireless communications in the Internet of Things: A comprehensive survey. Ad Hoc Networks. https://doi.org/10.1016/j.adhoc.2019.101913

    Article  Google Scholar 

  34. Park, H., & Lim, Y. (2020). Reinforcement learning for energy optimization with 5g communications in vehicular social networks. Sensors (Switzerland), 20(8). https://doi.org/10.3390/s20082361.

  35. Qian, Y., Jun, Wu., Wang, R., Zhu, F., & Zhang, W. (2019). Survey on Reinforcement Learning Applications in Communication Networks. Journal of Communications and Information Networks, 4(2), 30–39.

    Article  Google Scholar 

  36. Elsherief, M., Elwekeil, M., & Abd-Elnaby, M. (2019). Resource and power allocation for achieving rate fairness in D2D communications overlaying cellular networks. Wireless Networks. https://doi.org/10.1007/s11276-018-01935-y

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electronics Engineering, Vellore Institute of Technology, Vellore, India

    Steffi Jayakumar & S. Nandakumar

Authors
  1. Steffi Jayakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Nandakumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Nandakumar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jayakumar, S., Nandakumar, S. Reinforcement learning based distributed resource allocation technique in device-to-device (D2D) communication. Wireless Netw 29, 1843–1858 (2023). https://doi.org/10.1007/s11276-023-03230-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-023-03230-x

Keywords

Search

Navigation