Skip to main content
SpringerLink
Log in

A Reinforcement Learning Based Routing in Cognitive Radio Networks for Primary Users with Multi-stage Periodicity

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Designing an efficient routing protocol for cognitive radio networks is critical due to the dynamic behavior of the primary users. Based on empirical studies, the primary users activity on the licensed channels has periodicity comprised of several stages, and that the model of primary users activity may change during different stages. This paper has identified two main challenges facing designers: how to transmit packets via a stable route, and how to ensure imposing of minimal interference on the primary users. To address these, we propose a routing protocol which is based on a generalized version of Q-learning and which exploits the said model of primary users behavior. We divide the infinite time horizon into cycles (corresponding to periods), then break each cycle into several sub-cycles (corresponding to stages), making an assumption that the statistical model parameters of the primary users activity will not change during a sub-cycle. The extensive simulations confirm that our proposed routing approach outperforms existing schemes in terms of throughput and minimal interference. The paper also verifies that imposing of significant interference on the primary users and degradation of QoS of secondary users stem from lack of attention to the multi-stage periodic behavior of primary users.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. FCC, E. (2003). Docket no 03-222 notice of proposed rule making and order.

  2. Mitola, J. (2000). Cognitive radio—An integrated agent architecture for software defined radio.

  3. Haykin, S. (2005). Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications, 23(2), 201–220.

    Article  Google Scholar 

  4. Akyildiz, I. F., Lee, W. Y., & Chowdhury, K. R. (2009). Crahns: Cognitive radio ad hoc networks. AD Hoc Networks, 7(5), 810–836.

    Article  Google Scholar 

  5. Al-Rawi, H., & Yau, A. (2013). Routing in distributed cognitive radio networks: A survey. Wireless Personal Communications, 69(4), 1983–2020.

    Article  Google Scholar 

  6. Caleffi, M., & Cacciapuoti, A. S. (2016). Optimal database access for tv white space. IEEE Transactions on Communications, 64(1), 83–93.

    Article  Google Scholar 

  7. Cacciapuoti, A. S., Caleffi, M., & Paura, L. (2016). Optimal strategy design for enabling the coexistence of heterogeneous networks in tv white space. IEEE Transactions on Vehicular Technology, 65(9), 7361–7373.

    Article  Google Scholar 

  8. Wellens, M., RiihijäRvi, J., & MäHöNen, P. (2009). Empirical time and frequency domain models of spectrum use. Physical Communication, 2(1), 10–32.

    Article  Google Scholar 

  9. Macone, D., Oddi, G., & Pietrabissa, A. (2013). Mq-routing: Mobility-, gps-and energy-aware routing protocol in manets for disaster relief scenarios. Ad Hoc Networks, 11(3), 861–878.

    Article  Google Scholar 

  10. Shiang, H. P., & Van der Schaar, M. (2009). Distributed resource management in multihop cognitive radio networks for delay-sensitive transmission. IEEE Transactions on Vehicular Technology, 58(2), 941–953.

    Article  Google Scholar 

  11. Xia, B., Wahab, M. H., Yang, Y., Fan, Z., & Sooriyabandara, M. (2009). Reinforcement learning based spectrum-aware routing in multi-hop cognitive radio networks. In 2009. CROWNCOM’09. 4th international conference on cognitive radio oriented wireless networks and communications (pp. 1–5). IEEE.

  12. Zheng, K., Li, H., Qiu, R. C., & Gong, S. (2012). Multi-objective reinforcement learning based routing in cognitive radio networks: Walking in a random maze. In 2012 international conference on computing, networking and communications (ICNC) (pp. 359–363). IEEE.

  13. Al-Rawi, H., Yau, A., Mohamad, H., Ramli, N., & Hashim W., et al. (2014). A reinforcement learning-based routing scheme for cognitive radio ad hoc networks. In 2014 7th IFIP wireless and mobile networking conference (WMNC) (pp. 1–8). IEEE.

  14. Wang, W., Kwasinski, A., & Han, Z. (2014). Routing based on layered stochastic games for multi-hop cognitive radio networks. In 2014 IEEE wireless communications and networking conference (WCNC) (pp. 63–68). IEEE.

  15. Saleem, Y., Yau, A., Mohamad, H., Ramli, N., & Rehmani, M. H. (2015). Smart: A spectrum-aware cluster-based routing scheme for distributed cognitive radio networks. Computer Networks, 91, 196–224.

    Article  Google Scholar 

  16. Geirhofer, S., Tong, L., & Sadler, B. M. (2008). Cognitive medium access: Constraining interference based on experimental models. IEEE Journal on Selected Areas in Communications, 26(1), 95–105.

    Article  Google Scholar 

  17. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction (Vol. 1). Cambridge: MIT Press.

    Google Scholar 

  18. Xing, X., Jing, T., Huo, Y., Li, H., & Cheng, X. (2013). Channel quality prediction based on Bayesian inference in cognitive radio networks. In INFOCOM, 2013 proceedings IEEE (pp. 1465–1473). IEEE.

  19. Varga, A. (2010). Omnet++ user manual.

  20. Bayhan, S., & Alagöz, F. (2012). Distributed channel selection in crahns: A non-selfish scheme for mitigating spectrum fragmentation. Ad Hoc Networks, 10(5), 774–788.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mathematics and Computer Science, Amirkabir University of Technology, Tehran, Iran

    Mahsa Soheil Shamaee & Mohammad Ebrahim Shiri

  2. Department of Computer Engineering and Information Technology, Amirkabir University of Technology, Tehran, Iran

    Masoud Sabaei

Authors
  1. Mahsa Soheil Shamaee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Ebrahim Shiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Masoud Sabaei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohammad Ebrahim Shiri.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Soheil Shamaee, M., Shiri, M.E. & Sabaei, M. A Reinforcement Learning Based Routing in Cognitive Radio Networks for Primary Users with Multi-stage Periodicity. Wireless Pers Commun 101, 465–490 (2018). https://doi.org/10.1007/s11277-018-5700-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-018-5700-y

Keywords

Search

Navigation