Skip to main content
SpringerLink
Log in

Spectrum access in cognitive IoT using reinforcement learning

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

With the advent of fifth generation technologies for wireless networks and the expansion of the use of the Internet of things, the demands in using spectrum transmission have increased significantly, resulting in a shortage of available spectrum resources to meet these needs. The optimal utilization of the spectrum resources plays an important role to overcome this shortage problem. The Cognitive \(IoT\) (\(CIoT\)) is considered as promising technology to enhance spectrum utilization by accessing the vacant 4G/5G spectrum licensed to a primary user (\(PU\)). The choice between single channel and multiple channels spectrum access is critical in achieving higher data transmission and throughput. In single channel access, the \(CIoT\) waits on the same channel until its availability for usage, while in multiple channels access, \(CIoT\) can switch channels whenever it faces occupied channel, which improves the transmission quality and the achieved throughput. In this paper, a proposed proactive multiple channels spectrum access approach is introduced to enhance the spectrum access of \(CIoT\) through multiple available interfaces, wherein \(CIoT\) utilizes past channel states to predict the forthcoming spectrum availability. The proactive approach uses Reinforcement Learning \((RL)\) algorithm to select the available channels and Bayesian algorithm to predict how long the channel will be unoccupied. The available channels are arranged in descending order of their estimated idle probabilities to enable \(CIoT\) find sufficient idle channels quickly. The \(CIoT\) can use multiple channels simultaneously as long as there are enough free channels for transmission to reduce the spectrum handoffs and the transmission interruptions due to collisions. The sensing accuracy is adapted by achieving a high targeted probability of detection to guarantee primary users protection against harmful interference and lower probability of false alarm to increase the spectrum utilization. The simulation results demonstrate the effectiveness of the proposed approach and show an interesting performance compared with the single channel access model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Mainetti, L., Patrono, L., Vilei, A.: Evolution of wireless sensor networks towards the Internet of Things: a survey. In: Proceedings of the IEEE International Conference on Software, Telecommunications and Computer Networks, Split, Croatia, Sept 2011, pp. 15–17

  2. Khalil, N., Abid, M.R., Benhaddou, D., Gerndt, M.: Wireless sensors networks for Internet of Things. In: Proceedings of the IEEE 9th International Conference on Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), Singapore, pp. 1–6. (2014)

  3. Lin, Y., Yang, J., Lv, Z., Wei, W., Song, H.: A self-assessment stereo capture model applicable to the Internet of Things. Sensors 15(8), 20925–20944 (2015)

    Article  Google Scholar 

  4. Yang, J., He, S., Lin, Y., Lv, Z.: Multimedia cloud transmission and storage system based on Internet of Things. Multimedia Tools Appl. 76(17), 17735–17750 (2015)

    Article  Google Scholar 

  5. Maw, H.A., Xiao, H., Christianson, B., Malcolm, J.A.: BTG-AC: break-the-glass access control model for medical data in wireless sensor networks. IEEE J. Biomed. Health Inf. 20(3), 763–774 (2016)

    Article  Google Scholar 

  6. Yuan, B., Fu, C., Chen, D.: Building a large scale wireless sensor network for the industrial environment. In: Proceedings of the IEEE International Conference on Embedded and Real-Time Computing Systems and Applications (RTCSA), Daegu, South Korea, Aug 2016

  7. Chen, M., Yang, J., Hao, Y., Mao, S., Hwang, K.: A 5G cognitive system for healthcare. Big Data Cogn. Comput. 1(1), 2–16 (2017)

    Article  Google Scholar 

  8. Chen, M., Hao, Y., Hwang, K., Wang, L., Wang, L.: Disease prediction by machine learning over big data from healthcare communities. IEEE Access 5, 8869–8879 (2017)

    Article  Google Scholar 

  9. Tervonen, J., Mikhaylov, K., Pieskä, S., Jämsä, J., Heikkilä, M.: Cognitive Internet-of-Things solutions enabled by wireless sensor and actuator networks. In: Proceedings of the IEEE Conference on Cognitive Infocommunications (CogInfoCom), Italy, pp. 97–102. (2014)

  10. Wu, Q., Ding, G., Xu, Y., Feng, S., Du, Z., Wang, J., Long, K.: Cognitive Internet of Things: a new paradigm beyond connection. IEEE Internet Things J. 1(2), 129–143 (2014)

    Article  Google Scholar 

  11. Shah, M.A., Zhang, S., Maple, C.: Cognitive radio networks for Internet of Things: applications, challenges and future. In: Proceedings of the International Conference on Automation and Computing, London, pp. 1–6. (2013)

  12. Haustein, T., Stanczak, S., Wolisz, A., Jondral, F., Schotten, H., Kraemer, R., Mück, M., Mennenga, H., Bender, P.: Cognitive wireless communications—a paradigm shift in dealing with radio resources as a prerequisite for the wireless network of the future—an overview on the topic of cognitive wireless technologies. Frequenz 70(7–8), 281–288 (2016)

    Google Scholar 

  13. Otermat, D.T., Kostanic, I., Otero, C.E.: Analysis of the FM radio spectrum for secondary licensing of low-power short-range cognitive Internet of Things devices. IEEE Access 4, 6681–6691 (2016)

    Article  Google Scholar 

  14. Somov, A., Dupont, C., Giaffreda, R.: Supporting smart-city mobility with cognitive Internet of Things. In: Proceedings of the Future Network & Mobile Summit, Lisbon, Portugal, Oct 2013, pp. 1–10

  15. Nitti, M., Murroni, M., Fadda, M., Atzori, L.: Exploiting social Internet of Things features in cognitive radio. IEEE Access 34, 9204–9212 (2016)

    Article  Google Scholar 

  16. Zhu, J., Song, Y., Jiang, D., Song, H.: Multi-armed bandit channel access scheme with cognitive radio technology in wireless sensor networks for the Internet of Things. IEEE Access 4, 4609–4617 (2016)

    Article  Google Scholar 

  17. Mitola, J., Maguire, G.Q.: Cognitive radio: making software radios more personal. IEEE Pers. Commun. 6(4), 13–18 (1999)

    Article  Google Scholar 

  18. Tsiropoulos, G.I., Dobre, O.A., Hossam Ahmed, M., Baddour, K.E.: Radio resource allocation techniques for efficient spectrum access in cognitive radio networks. IEEE Commun. Surv. Tutor. 18(1), 824–847 (2016)

    Article  Google Scholar 

  19. Liu, X., Jia, M., Zhang, X., Lu, W.: A novel multichannel internet of things based on dynamic spectrum sharing in 5g communication. IEEE Internet Things J. 6(4), 5962–5970 (2019)

    Article  Google Scholar 

  20. Wang, C., Wang, L., Adachi, F.: Modeling and analysis for reactive-decision spectrum handoff in cognitive radio networks. In: 2010 IEEE Global Telecommunications Conference GLOBECOM 2010, Miami, FL, pp. 1–6. (2010). https://doi.org/10.1109/GLOCOM.2010.5683644

  21. Song, Y., Xie, J.: Common hopping based proactive spectrum handoff in cognitive radio ad hoc networks. In: 2010 IEEE Global Telecommunications Conference GLOBECOM 2010, Miami, FL, pp. 1–5. (2010). https://doi.org/10.1109/GLOCOM.2010.5683840

  22. Gosavi, A.: Reinforcement learning: a tutorial survey and recent advances. INFORMS J. Comput. 21(2), 178–192 (2009)

    Article  MathSciNet  Google Scholar 

  23. Raj, V., Dias, I., Tholeti, T., Kalyani, S.: Spectrum access in cognitive radio using a two-stage reinforcement learning approach. IEEE J. Sel. Top. Signal Process. 12(1), 20–34 (2018)

    Article  Google Scholar 

  24. Yang, J., Zhao, H.: Enhanced throughput of cognitive radio networks by imperfect spectrum prediction. IEEE Commun. Lett. 19(10), 1738–1741 (2015)

    Article  Google Scholar 

  25. Lu, D., Huang, X., Zhang, W., Fan, J.: Interference-aware spectrum handover for cognitive radio networks. Wirel. Commun. Mob. Comput. 14(11), 1099–1112 (2014)

    Article  Google Scholar 

  26. Oksanen, J., Koivunen, V.: An order optimal policy for exploiting idle spectrum in cognitive radio networks. IEEE Trans. Signal Process. 63(5), 1214–1227 (2015)

    Article  MathSciNet  Google Scholar 

  27. Lertsinsrubtavee, A., Malouch, N.: Hybrid spectrum sharing through adaptive spectrum handoff and selection. IEEE Trans. Mob. Comput. 15(11), 2781–2793 (2016)

    Article  Google Scholar 

  28. Shi, Q., Shao, W., Fang, B., Zhang, Y., Zhang, Y.: Reinforcement learning-based spectrum handoff scheme with measured PDR in cognitive radio networks. Electron. Lett. 55(25), 1368–1370 (2019)

    Article  Google Scholar 

  29. Syed, A., Yau, K., Mohamad, H., Ramli, N., Hashim, W.: Channel selection in multi-hop cognitive radio network using reinforcement learning: an experimental study. In: IET, International Conference on Frontiers of Communications, Networks and Applications (ICFCNA 2014 - Malaysia), Nov 2014

  30. Notsu, A., Honda, K., Ichihashi, H., Komori, Y.: Simple reinforcement learning for small-memory agent. In: 2011 10th International Conference on Machine Learning and Applications and Workshops, vol. 1, IEEE, pp. 458–461, Dec 2011

  31. Hossen, M.A., Yoo, S.: Q-learning based multi-objective clustering algorithm for cognitive radio ad hoc networks. IEEE Access 7, 181959–181971 (2019)

    Article  Google Scholar 

  32. Lundn, J., Kulkarni, S.R., Koivunen, V., Poor, H.V.: Multiagent reinforcement learning based spectrum sensing policies for cognitive radio networks. IEEE J. Sel. Top. Signal Process. 7(5), 858–868 (2013)

    Article  Google Scholar 

  33. Wu, Y., Hu, F., Kumar, S., Zhu, Y., Talari, A., Rahnavard, N., Matyjas, J.D.: A learning-based QOE-driven spectrum handoff scheme for multimedia transmissions over cognitive radio networks. IEEE J. Sel. Areas Commun. 32(11), 2134–2148 (2014)

    Article  Google Scholar 

  34. Li, F., Lam, K.-Y., Sheng, Z., Zhang, X., Zhao, K., Wang, L.: Q-learning-based dynamic spectrum access in cognitive industrial Internet of Things. Mob. Netw. Appl. 23, 1636–1644 (2018)

    Article  Google Scholar 

  35. Zhu, J., Song, Y., Jiang, D., Song, H.: A new deep-Q-learning-based transmission scheduling mechanism for the cognitive Internet of Things. IEEE Internet Things J. 5(4), 2375–2385 (2018)

    Article  Google Scholar 

  36. Yang, H., Zhong, W.-D., Chen, C., Alphones, A., Xie, X.: Deep-reinforcement-learning-based energy-efficient resource management for social and cognitive Internet of Things. IEEE Internet Things J. 7(6), 5677–5689 (2020)

    Article  Google Scholar 

  37. Xu, Y.-H., Tian, Y.-B., Searyoh, P.K., Yu, G., Yong, Y.-T.: Deep reinforcement learning-based resource allocation strategy for energy harvesting-powered cognitive machine-to-machine networks. Comput. Commun. 160, 706–717 (2020)

    Article  Google Scholar 

  38. Liu, X., Sun, C., Zhou, M., Wu, C., Peng, B., Li, P.: Reinforcement learning-based multislot double-threshold spectrum sensing with Bayesian fusion for industrial big spectrum data. IEEE Trans. Ind. Inf. (2020). https://doi.org/10.1109/TII.2020.2987421

    Article  Google Scholar 

  39. Ning, W., Huang, X., Yang, K., Wu, F., Leng, S.: Reinforcement learning enabled cooperative spectrum sensing in cognitive radio networks. J. Commun. Netw. 22(1), 12–22 (2020)

    Article  Google Scholar 

  40. Sarikhani, R., Keynia, F.: Cooperative spectrum sensing meets machine learning: deep reinforcement learning approach. IEEE Commun. Lett. 24(7), 1459–1462 (2020)

    Article  Google Scholar 

  41. Li, D., Jiang, X., Cao, W., Xie, H., Liu, Y., Yang, J.: A POMDP approach to channel sensing and data transmission for opportunistic spectrum access. In: 2019 IEEE 3rd Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Chongqing, China, pp. 164–169. (2019). https://doi.org/10.1109/IMCEC46724.2019.8984108

  42. Das, A., Ghosh, S.C., Das, N., Barman, A.D.: Q-learning based co-operative spectrum mobility in cognitive radio networks. In: 2017 IEEE 42nd Conference on Local Computer Networks (LCN), Singapore, pp. 502–505. (2017). https://doi.org/10.1109/LCN.2017.80

  43. Balaji, V.: Reinforcement learning based decision fusion scheme for cooperative spectrum sensing in cognitive radios. In: 2018 International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), Chennai, pp. 1–5. (2018). https://doi.org/10.1109/WiSPNET.2018.8538528

  44. Li, Y., Zhang, W., Wang, C., Sun, J., Liu, Y.: Deep reinforcement learning for dynamic spectrum sensing and aggregation in multi-channel wireless networks. IEEE Trans. Cogn. Commun. Netw. 6(2), 464–475 (2020)

    Article  Google Scholar 

  45. Salameh, H.B., Shtyyat, S., Jararweh, Y.: Adaptive variable-size virtual clustering for control channel assignment in dynamic access networks: design and simulations. Simul. Model. Pract. Theory 106, 102197 (2021)

    Article  Google Scholar 

  46. Liu, X., Zhang, X., Ding, H., Peng, B.: Intelligent clustering cooperative spectrum sensing based on Bayesian learning for cognitive radio network. Ad Hoc Netw. 94, 101968 (2019)

    Article  Google Scholar 

  47. Jang, S.-J., Han, C.-H., Lee, K.-E., Yoo, S.-J.: Reinforcement learning-based dynamic band and channel selection in cognitive radio ad-hoc networks. EURASIP J. Wirel. Commun. Netw. 2019, 131 (2019)

    Article  Google Scholar 

Download references

Funding

There is no funding for this research.

Author information

Authors and Affiliations

  1. National Research Center, Information Systems Engineering, Cairo, Egypt

    Walid K. Ghamry

  2. Computer Engineering - Faculty of Computer Science and Information Technology, Albaha University, Al Baha, Saudi Arabia

    Walid K. Ghamry

  3. Higher Technological Institute, Cairo, Egypt

    Suzan Shukry

Authors
  1. Walid K. Ghamry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suzan Shukry
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Walid Ghamry proposed the protocol and made all the analysis. Suzan Shukry conducted and performed the simulations. Walid Ghamry read and approved the final manuscript.

Corresponding author

Correspondence to Walid K. Ghamry.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghamry, W.K., Shukry, S. Spectrum access in cognitive IoT using reinforcement learning. Cluster Comput 24, 2909–2925 (2021). https://doi.org/10.1007/s10586-021-03306-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-021-03306-3

Keywords

Search

Navigation