Advertisement

Artificial Intelligence Review

, Volume 51, Issue 3, pp 493–506 | Cite as

A survey of dynamic spectrum allocation based on reinforcement learning algorithms in cognitive radio networks

  • Yonghua WangEmail author
  • Zifeng Ye
  • Pin Wan
  • Jiajun Zhao
Article

Abstract

Cognitive radio is an emerging technology that is considered to be an evolution for software device radio in which cognition and decision-making components are included. The main function of cognitive radio is to exploit “spectrum holes” or “white spaces” to address the challenge of the low utilization of radio resources. Dynamic spectrum allocation, whose significant functions are to ensure that cognitive users access the available frequency and bandwidth to communicate in an opportunistic manner and to minimize the interference between primary and secondary users, is a key mechanism in cognitive radio networks. Reinforcement learning, which rapidly analyzes the amount of data in a model-free manner, dramatically facilitates the performance of dynamic spectrum allocation in real application scenarios. This paper presents a survey on the state-of-the-art spectrum allocation algorithms based on reinforcement learning techniques in cognitive radio networks. The advantages and disadvantages of each algorithm are analyzed in their specific practical application scenarios. Finally, we discuss open issues in dynamic spectrum allocation that can be topics of future research.

Keywords

Reinforcement learning Spectrum allocation Cognitive radio networks 

Notes

Acknowledgements

This work was supported in part by special funds from the central finance to support the development of local universities under No. 400170044, the project supported by the State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences under Grant No. 20180106, the science and technology program of Guangdong Province under Grant No. 2016B090918031, the degree and graduate education reform project of Guangdong Province under Grant No. 2016JGXM_MS_26, the foundation of key laboratory of machine intelligence and advanced computing of the Ministry of Education under Grant No. MSC-201706A and the higher education quality projects of Guangdong Province and Guangdong University of Technology.

References

  1. Ahmad A, Ahmad S, Rehmani MH, Hassan NU (2015) A survey on radio resource allocation in cognitive radio sensor networks. IEEE Commun Surv Tutor 17(2):888–917CrossRefGoogle Scholar
  2. Ahmed E, Gani A, Abolfazli S, Yao LJ, Khan SU (2014) Channel assignment algorithms in cognitive radio networks: taxonomy, open issues, and challenges. IEEE Commun Surv Tutor 18(1):795–823CrossRefGoogle Scholar
  3. Al-Rawi HA, Ng MA, Yau KLA (2015) Application of reinforcement learning to routing in distributed wireless networks: a review. Artif Intell Rev 43(3):381–416CrossRefGoogle Scholar
  4. Alsarhan A, Agarwal A (2011) Profit optimization in multi-service cognitive mesh network using machine learning. EURASIP J Wirel Commun Netw 1:36CrossRefGoogle Scholar
  5. Anifantis E, Karyotis V, Papavassiliou S (2012) A markov random field framework for channel assignment in cognitive radio networks. In: 2012 IEEE international conference on pervasive computing and communications workshops (PERCOM workshops). IEEE, pp 770–775Google Scholar
  6. Berthold U, Fu F, Van M der Schaar, Jondral FK (2008) Detection of spectral resources in cognitive radios using reinforcement learning. In: New frontiers in dynamic spectrum access networks, pp 1–5Google Scholar
  7. Bkassiny M, Li Y, Jayaweera SK (2013) A survey on machine-learning techniques in cognitive radios. IEEE Commun Surv Tutor 15(3):1136–1159CrossRefGoogle Scholar
  8. Busoniu L, Babuska R, De Schutter B (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Trans Syst Man Cybern Part C Appl Rev 38(2):156–172CrossRefGoogle Scholar
  9. Chen S, Huang Y, Namuduri K (2011) A factor graph based dynamic spectrum allocation approach for cognitive network. In: Wireless communications and networking conference (WCNC), 2011 IEEE. IEEE, pp 850–855Google Scholar
  10. Cheng X, Jiang M (2011) Cognitive radio spectrum assignment based on artificial bee colony algorithm. In: 2011 IEEE 13th international conference on communication technology (ICCT). IEEE, pp 161–164Google Scholar
  11. Faganello LR, Kunst R, Both CB, Granville LZ (2013) Improving reinforcement learning algorithms for dynamic spectrum allocation in cognitive sensor networks. In: Wireless communications and networking conference, pp 35–40Google Scholar
  12. Feng Z, Liang L, Tan L, Zhang P (2009) Q-learning based heterogenous network self-optimization for reconfigurable network with CPC assistance. Sci China (Ser F) 52(12):2360–2368zbMATHGoogle Scholar
  13. Han G, Xiao L, Poor HV (2017) Two-dimensional anti-jamming communication based on deep reinforcement learning. In: IEEE international conference on acoustics, speech and signal processing, pp 2087–2091Google Scholar
  14. Hossain E, Bhargava V (2007) Cognitive wireless communication networks. Springer, New YorkCrossRefGoogle Scholar
  15. Iii JM (2000) Cognitive radio: an integrated agent architecture for software defined radio, Ph.D. dissertation. ResearchGate 6(4):13–18Google Scholar
  16. Le HST, Ly HD (2008) Opportunistic spectrum access using fuzzy logic for cognitive radio networks. In: Second international conference on communications and electronics, ICCE 2008. IEEE, pp 240–245Google Scholar
  17. Levorato M, Firouzabadi S, Goldsmith A (2012) A learning framework for cognitive interference networks with partial and noisy observations. IEEE Trans Wirel Commun 11(9):3101–3111CrossRefGoogle Scholar
  18. Li H, Zhu G, Liang Z, Chen Y (2010) A survey on distributed opportunity spectrum access in cognitive network. In: International conference on wireless communications networking and mobile computing, pp 1–4Google Scholar
  19. Li Y, Feng Z, Chen S, Chen Y, Xu D, Zhang P, Zhang Q (2011) Radio resource management for public femtocell networks. EURASIP J Wirel Commun Netw 1:181CrossRefGoogle Scholar
  20. Lilith N, Dogancay K (2005) Distributed reduced-state sarsa algorithm for dynamic channel allocation in cellular networks featuring traffic mobility. IEEE Int Conf Commun 2:860–865Google Scholar
  21. Lv C, Wang J, Yu F, Dai H (2013) A Q-learning-based dynamic spectrum allocation algorithm. ICCSEE-13Google Scholar
  22. Marinho J, Monteiro E (2012) Cognitive radio: survey on communication protocols, spectrum decision issues, and future research directions. Wirel Netw 18(2):147–164CrossRefGoogle Scholar
  23. Mitola J, Maguire GQ (1999) Cognitive radio: making software radios more personal. IEEE Pers Commun 6(4):13–18CrossRefGoogle Scholar
  24. Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. Comput SciGoogle Scholar
  25. Nie J, Haykin S (1997) A Q-learning-based dynamic channel assignment technique for mobile communication systems. IEEE Trans Veh Technol 48(5):1676–1687Google Scholar
  26. Qadir J (2016) Artificial intelligence based cognitive routing for cognitive radio networks. Artif Intell Rev 45(1):25–96CrossRefGoogle Scholar
  27. Ru M, Yin S, Qu Z (2017) Power and spectrum allocation in d2d networks based on coloring and chaos genetic algorithm. Procedia Comput Sci 107:183–189CrossRefGoogle Scholar
  28. Salameh HAB (2011) Throughput-oriented channel assignment for opportunistic spectrum access networks. Math Comput Modell 53(11–12):2108–2118CrossRefGoogle Scholar
  29. Shu T, Krunz M (2010) Exploiting microscopic spectrum opportunities in cognitive radio networks via coordinated channel access. IEEE Trans Mob Comput 9(11):1522–1534CrossRefGoogle Scholar
  30. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, vol 1. MIT Press, CambridgezbMATHGoogle Scholar
  31. Tanwongvarl C, Chantaraskul S (2015) Performance comparison of learning techniques for intelligent channel assignment in cognitive wireless sensor networks. In: Seventh international conference on ubiquitous and future networks, pp 503–507Google Scholar
  32. Teng Y, Zhang Y, Niu F, Dai C (2010) Reinforcement learning based auction algorithm for dynamic spectrum access in cognitive radio networks. In: Vehicular technology conference fall, pp 1–5Google Scholar
  33. Teng Y, Yu FR, Han K, Wei Y, Zhang Y (2013) Reinforcement-learning-based double auction design for dynamic spectrum access in cognitive radio networks. Wirel Pers Commun 69(2):771–791CrossRefGoogle Scholar
  34. Tragos EZ, Zeadally S, Fragkiadakis AG, Siris VA (2013) Spectrum assignment in cognitive radio networks: a comprehensive survey. IEEE Commun Surv Tutor 15(3):1108–1135CrossRefGoogle Scholar
  35. Wang W, Kwasinski A, Niyato D, Han Z (2016) A survey on applications of model-free strategy learning in cognitive wireless networks. IEEE Commun Surv Tutor 18(3):1717–1757CrossRefGoogle Scholar
  36. Xiao L, Li Y, Dai C, Dai H, Poor HV (2017) Reinforcement learning-based NOMA power allocation in the presence of smart jamming. IEEE Trans Veh Technol 67:3377–3389CrossRefGoogle Scholar
  37. Yang M, Grace D (2011) Cognitive radio with reinforcement learning applied to multicast downlink transmission with power adjustment. Wirel Pers Commun 57(1):73–87CrossRefGoogle Scholar
  38. Yang R, Ye F, et al (2010) Non-cooperative spectrum allocation based on game theory in cognitive radio networks. In: 2010 IEEE fifth international conference on bio-inspired computing: theories and applications (BIC-TA). IEEE, pp 1134–1137Google Scholar
  39. Yau KLA, Komisarczuk P, Teal PD (2012) Reinforcement learning for context awareness and intelligence in wireless networks: review, new features and open issues. J Netw Comput Appl 35(1):253–267CrossRefGoogle Scholar
  40. Yi L, Hong J (2012) Q-learning for dynamic channel assignment in cognitive wireless local area network with fibre-connected distributed antennas. J China Univer Posts Telecommun 19(4):80–85CrossRefGoogle Scholar
  41. Yu L, Liu C, Liu Z, Hu W (2010) Heuristic spectrum assignment algorithm in distributed cognitive networks. In: 2010 6th International conference on wireless communications networking and mobile computing (WiCOM). IEEE, pp 1–5Google Scholar
  42. Zhang Y, Lee C, Niyato D, Wang P (2013) Auction approaches for resource allocation in wireless systems: a survey. IEEE Commun Surv Tutor 15(3):1020–1041CrossRefGoogle Scholar
  43. Zhao C, Zou M, Shen B, Kim B, Kwak K (2008) Cooperative spectrum allocation in centralized cognitive networks using bipartite matching. In: Global telecommunications conference, IEEE GLOBECOM 2008. IEEE, pp 1–6Google Scholar
  44. Zhao Q, Sadler BM (2007) A survey of dynamic spectrum access. IEEE Signal Process Mag 24(3):79–89CrossRefGoogle Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  1. 1.School of AutomationGuangdong University of TechnologyGuangzhouChina
  2. 2.State Key Laboratory of Management and Control for Complex Systems, Institute of AutomationChinese Academy of SciencesBeijingChina

Personalised recommendations