Skip to main content
SpringerLink
Log in

Reinforcement Learning Based Q-Routing: Performance Evaluation on Cognitive Radio Network Topologies

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

The present work explores the scope of cognitive radio networks (CRNs) to support the spectrum demand on future diverse application specific wireless services supported by long-range multi-hop relay-based transmission. The essential requirement is dynamic route establishment where the traditional non-adaptive routing algorithms are not sufficient to support it in dynamic wireless networks. To address this issue, reinforcement learning (RL) based self-adaptable Q-routing in the CRN with different network topologies is explored with an objective to obtain the best performance in the secondary throughput. The traditional Q-routing policy offers the worst-case time-complexity of \({\mathcal {O}}(ed)\) to find the optimal route. The excessive time complexity for the route establishment reduces the actual secondary data transmission time, this also increases the power consumption during transmission. To reduce the time complexity of Q-routing, the proposed work invokes a strict upper bound on the network topology, which reduces the time complexity to \({\mathcal {O}}(N^2)\) and increases the secondary network throughput. The further study also reports that the complexity can be reduced to \({\mathcal {O}}(N)\) if the distributed relays follow a tree topology in the network configuration. A large set of simulation results show that the tree structure-based Q-routing in CR enhances the network throughput by \(\sim 6.83\%\), \(\sim 7.21\%\) over the other network topologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of data and material

Not shareable currently.

References

  1. Hassan, A. M., & Awad, A. I. (2018). Urban transition in the era of the internet of things: Social implications and privacy challenges. IEEE Access, 6, 36428–36440.

    Article  Google Scholar 

  2. Ma, Y., Wu, Y., Ge, J., & LI, J. (2018). An architecture for accountable anonymous access in the internet-of-things network. IEEE Access, 6, 14451–14461.

    Article  Google Scholar 

  3. Reeves, D. (2018). How to create a smart city: Future-proofed cities that foster growth and innovation. IEEE Electrification Magazine, 6, 34–41.

    Article  Google Scholar 

  4. Salameh, H. A. B., Almajali, S., Ayyash, M., & Elgala, H. (2018). Spectrum assignment in cognitive radio networks for internet-of-things delay-sensitive applications under jamming attacks. IEEE Internet of Things Journal, 5, 1904–1913.

    Article  Google Scholar 

  5. Zaheer, K., Othman, M., Rehmani, M. H., & Perumal, T. (2018). A survey of decision-theoretic models for cognitive internet of things (CIoT). IEEE Access, 6, 22489–22512.

    Article  Google Scholar 

  6. Han, R., Gao, Y., Wu, C., & Lu, D. (2018). An effective multi-objective optimization algorithm for spectrum allocations in the cognitive-radio-based internet of things. IEEE Access, 6, 12858–12867.

    Article  Google Scholar 

  7. Kakalou, I., Psannis, K. E., Krawiec, P., & Badea, R. (2017). Cognitive radio network and network service chaining toward 5G: Challenges and requirements. IEEE Communications Magazine, 55, 145–151.

    Article  Google Scholar 

  8. Katta, S., & Prasad, M. S. G. (2020). Performance analysis of cognitive radio under spectrum sharing using CTMC queuing model. Journal of Critical Reviews, 7(5), 555–558.

    Google Scholar 

  9. Paul, A., Banerjee, A., & Maity, S. P. (2019). Throughput maximisation in cognitive radio networks with residual bandwidth. IET Communications, 13(10), 1327–1335.

    Article  Google Scholar 

  10. Lee, S., Duong, T. Q., da Costa, D. B., Ha, D. B., & Nguyen, S. Q. (2018). Underlay cognitive radio networks with cooperative non-orthogonal multiple access. IET Communications, 12(3), 359–366.

    Article  Google Scholar 

  11. Ganesh, D., Kumar, T. P., & Kumar, M. S. (2021). Optimised levenshtein centroid cross-layer defence for multi-hop cognitive radio networks. IET Communications, 15(2), 245–256.

    Article  MathSciNet  Google Scholar 

  12. Zhang, X., Zhang, X., Han, L., & Xing, R. (2018). Utilization-oriented spectrum allocation in an underlay cognitive radio network. IEEE Access, 6, 12905–12912.

    Article  Google Scholar 

  13. Yan, Z., Chen, S., Zhang, X., & Liu, H. L. (2018) Outage performance analysis of wireless energy harvesting relay-assisted random underlay cognitive networks. IEEE Internet of Things Journal, pp. 1.

  14. Tian, R., Wang, Z., & Tan, X. (2018) A new leakage-based precoding scheme in iot oriented cognitive MIMO-OFDM systems. IEEE Access, pp. 1.

  15. Khakzad, H., Taherpour, A., Shakeri, R., & Khattab, T. (2017). Dynamic interference-limited relay sharing in cognitive radio networks by using hierarchical modulation. IET Communications, 11(12), 1903–1912.

    Article  Google Scholar 

  16. Huang, X., Yu, R., Kang, J., Xia, Z., & Zhang, Y. (2018). Software defined networking for energy harvesting internet of things. IEEE Internet of Things Journal, 5, 1389–1399.

    Article  Google Scholar 

  17. NavnathDattatraya, K., & Rao, K. R. (2019). Maximising network lifetime and energy efficiency of wireless sensor network using group search ant lion with levy flight. IET Communications, 14(6), 914–922.

    Article  Google Scholar 

  18. Banerjee, A., Paul, A., & Maity, S. P. (2018). Joint power allocation and route selection for outage minimization in multihop cognitive radio networks with energy harvesting. IEEE Transactions on Cognitive Communications and Networking, 4, 82–92.

    Article  Google Scholar 

  19. Huang, M., Liu, Y., Zhang, N., Xiong, N. N., Liu, A., Zeng, Z., & Song, H. (2018). A services routing based caching scheme for cloud assisted crns. IEEE Access, 6, 15787–15805.

    Article  Google Scholar 

  20. Sonti, S. R., & Prasad, M. S. G. (2019). Enhanced fuzzy c-means clustering based cooperative spectrum sensing combined with multi-objective resource allocation approach for delay-aware crns. IET Communications, 14(4), 619–626.

    Article  Google Scholar 

  21. Parida, R. K., Mishra, R. K., Sahoo, N. K., Muduli, A., Panda, D. C., & Mishra, R. K. (2020). A hybrid multi-port antenna system for cognitive radio. Progress in Electromagnetics Research, 106, 1–16.

    Article  Google Scholar 

  22. Tang, F., & Li, J. (2017). Joint rate adaptation, channel assignment and routing to maximize social welfare in multi-hop cognitive radio networks. IEEE Transactions on Wireless Communications, 16, 2097–2110.

    Article  Google Scholar 

  23. Paul, A., & Maity, S. P. (2018). On outage minimization in cognitive radio networks through routing and power control. Wireless Personal Communications, 98, 251–269.

    Article  Google Scholar 

  24. Anusha, M., & Vemuru, S. (2020). An effective mac protocol for multi-radio multi-channel environment of cognitive radio wireless mesh network (CRWMN). In: Proceedings of the First International Conference on Sustainable Technologies for Computational Intelligence, pp. 21–35, Springer, New York.

  25. Ch, S., Ramesh, K., et al. (2020). Fuzzy guided integrative factors-based spectrum decision-making in cognitive radio networks. International Journal of Intelligent Unmanned Systems.

  26. Paul, A., & Maity, S. P. (2016). Kernel fuzzy c-means clustering on energy detection based cooperative spectrum sensing. Digital Communications and Networks, 2(4), 196–205.

    Article  Google Scholar 

  27. Paul, A., Kunarapu, P., Banerjee, A., & Maity, S. P. (2019). Spectrum sensing in cognitive vehicular networks for uniform mobility model. IET Communications, 13(19), 3127–3134.

    Article  Google Scholar 

  28. Anumandla, K. K., Sabat, S. L., Peesapati, R., AV, P., Dabbakuti, J. K., & Rout, R. Optimal spectrum and power allocation using evolutionary algorithms for cognitive radio networks. Internet Technology Letters, p. e207.

  29. Nayak, D. K., Muduli, A., Hussain, M. T., Mirza, A. A., Gummadipudi, J. R., & Kumar, N. S. (2020). Channel allocation in cognitive radio networks using energy detection technique. Materials Today: Proceedings, 33, 934–938.

    Google Scholar 

  30. Reddy, S. S., & Prasad, M. S. G. (2021). Improved whale optimization algorithm and convolutional neural network based cooperative spectrum sensing in cognitive radio networks. Information Security Journal: A Global Perspective, 30(3), 160–172.

    Google Scholar 

  31. Paul, A., & Maity, S. P. (2020). Outage analysis in cognitive radio networks with energy harvesting and q-routing. IEEE Transactions on Vehicular Technology, 69(6), 6755–6765.

    Article  Google Scholar 

  32. Paul, A., & Maity, S. P. (2017). Optimal cluster power for joint spectrum sensing and secondary data transmission in cognitive radio networks. In: Proceedings of the IEEE International Conference on Advanced Networks and Telecommunications Systems (ANTS), pp. 1–6, IEEE.

  33. Du, Y., Xue, L., Xu, Y., & Liu, Z. (2019). An apprenticeship learning scheme based on expert demonstrations for cross-layer routing design in cognitive radio networks. AEU-International Journal of Electronics and Communications, 107, 221–230.

    Google Scholar 

  34. Lavanya, S., & Bhagyaveni, M. A. (2017). Design of sop based cross-layered opportunistic routing protocol for cr ad-hoc networks. Wireless Personal Communications, 96(4), 6543–6556.

    Article  Google Scholar 

  35. Saad, M. (2014). Joint optimal routing and power allocation for spectral efficiency in multihop wireless networks. IEEE Transactions on Wireless Communications, 13, 2530–2539.

    Article  Google Scholar 

  36. Saleem, Y., Yau, K. L. A., Mohamad, H., Ramli, N., Rehmani, M. H., & Ni, Q. (2017). Clustering and reinforcement-learning-based routing for cognitive radio networks. IEEE Wireless Communications, 24, 146–151.

    Article  Google Scholar 

  37. Syed, A. R., Yau, K. L. A., Qadir, J., Mohamad, H., Ramli, N., & Keoh, S. L. (2016). Route selection for multi-hop cognitive radio networks using reinforcement learning: An experimental study. IEEE Access, 4, 6304–6324.

    Article  Google Scholar 

  38. Wang, J., Yue, H., Hai, L., & Fang, Y. (2017). Spectrum-aware anypath routing in multi-hop cognitive radio networks. IEEE Transactions on Mobile Computing, 16, 1176–1187.

    Article  Google Scholar 

  39. Mansourkiaie, F., Ismail, L. S., Elfouly, T. M., & Ahmed, M. H. (2017). Maximizing lifetime in wireless sensor network for structural health monitoring with and without energy harvesting. IEEE Access, 5, 2383–2395.

    Article  Google Scholar 

  40. Babaee, R., & Beaulieu, N. C. (2011). Power-optimized routing with bandwidth guarantee in multihop relaying networks. In: Proceedings of the IEEE International Conference on Communications (ICC), pp. 1–6, IEEE.

  41. Zhang, L., Cai, Z., Li, P., Wang, L., & Wang, X. (2017). Spectrum-availability based routing for cognitive sensor networks. IEEE Access, 5, 4448–4457.

    Article  Google Scholar 

  42. Bahbahani, M. S., & Alsusa, E. (2018). A cooperative clustering protocol with duty cycling for energy harvesting enabled wireless sensor networks. IEEE Transactions on Wireless Communications, 17, 101–111.

    Article  Google Scholar 

  43. Vivekanand, C. V., & Bagan, K. B. (2020). Secure distance based improved leach routing to prevent puea in cognitive radio network. Wireless Personal Communications, 113(4), 1823–1837.

    Article  Google Scholar 

  44. Koenig, S., & Simmons, R. G. (1992). Complexity analysis of real-time reinforcement learning applied to finding shortest paths in deterministic domains. Technical report

  45. Salih, Q. M., Rahman, M. A., Al-Turjman, F., & Azmi, Z. R. M. (2020). Smart routing management framework exploiting dynamic data resources of cross-layer design and machine learning approaches for mobile cognitive radio networks: A survey. IEEE Access, 8, 67835–67867.

    Article  Google Scholar 

  46. Elangovan, K., & Subashini, S. (2018). Particle bee optimized convolution neural network for managing security using cross-layer design in cognitive radio network. Journal of Ambient Intelligence and Humanized Computing, pp. 1–9.

  47. Gawas, M. A., & Govekar, S. (2021). State-of-art and open issues of cross-layer design and qos routing in internet of vehicles. Wireless Personal Communications, 116(3), 2261–2297.

    Article  Google Scholar 

  48. Singhal, C., & Rajesh, A. (2020). Review on cross-layer design for cognitive ad-hoc and sensor network. IET Communications, 14(6), 897–909.

    Article  Google Scholar 

  49. Awang, A., Husain, K., Kamel, N., & Aissa, S. (2017). Routing in vehicular ad-hoc networks: a survey on single-and cross-layer design techniques, and perspectives. IEEE Access, 5, 9497–9517.

    Article  Google Scholar 

  50. Chen, J., Ping, S., Jia, J., Deng, Y., Dohler, M., & Aghvami, H. (2016). Cross-layer optimization for spectrum aggregation-based cognitive radio ad-hoc networks. In: Proceedings of the IEEE Global Communications Conference (GLOBECOM), pp. 1–6.

Download references

Funding

No funding is available for this work.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Koneru Lakshmaiah Education Foundation, Hyderabad, Telangana, 500075, India

    Anal Paul

  2. Department of Information Technology, Indian Institute of Engineering Science and Technology, Shibpur, 711103, India

    Santi Prasad Maity

Authors
  1. Anal Paul
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Santi Prasad Maity
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Santi Prasad Maity.

Ethics declarations

Conflicts of interest

Not applicable.

Code availability

Not shareable currently.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Paul, A., Maity, S.P. Reinforcement Learning Based Q-Routing: Performance Evaluation on Cognitive Radio Network Topologies. Wireless Pers Commun 125, 1425–1441 (2022). https://doi.org/10.1007/s11277-022-09612-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-022-09612-2

Keywords

Search

Navigation