Skip to main content

Advertisement

SpringerLink
Log in

Uplink NOMA-based long-term throughput maximization scheme for cognitive radio networks: an actor–critic reinforcement learning approach

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Non-orthogonal multiple access (NOMA) is one of the promising techniques for spectrum efficiency in wireless networks. In this paper, we consider an uplink NOMA cognitive system, where the secondary users (SUs) can jointly transmit data to the cognitive base station (CBS) over the same spectrum resources. Thereafter, successive interference cancellation is applied at the CBS to retrieve signals transmitted by the SUs. In addition, the energy-constrained problem in wireless networks is taken into account. Therefore, we assume that the SUs are powered by a wireless energy harvester to prolong their operations; meanwhile, the CBS is equipped with a traditional electrical supply. Herein, we propose an actor–critic reinforcement learning approach to maximize the long-term throughput of the cognitive network. In particular, by interacting and learning directly from the environment over several time slots, the CBS can optimally assign the amount of transmission energy for each SU according to the remaining energy of the SUs and the availability of the primary channel. As a consequence, the simulation results verify that the proposed scheme outperforms other conventional approaches (such as Myopic NOMA and OMA), so the system reward is always maximized in the current time slot, in terms of overall throughput and energy efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Khan, F. A., Ratnarajah, T., & Sellathurai, M. (2010). Multiuser diversity analysis in spectrum sharing cognitive radio networks. In 2010 Proceedings of the fifth international conference on cognitive radio oriented wireless networks and communications, Cannes (pp. 1–5).

  2. Wang, C., et al. (2014). Cellular architecture and key technologies for 5G wireless communication networks. IEEE Communications Magazine, 52(2), 122–130.

    Google Scholar 

  3. Hong, X., Wang, J., Wang, C., & Shi, J. (2014). Cognitive radio in 5G: A perspective on energy-spectral efficiency trade-off. IEEE Communications Magazine, 52(7), 46–53.

    Google Scholar 

  4. Wang, B., & Liu, K. J. R. (2011). Advances in cognitive radio networks: A survey. IEEE Journal of Selected Topics in Signal Processing, 5(1), 5–23.

    Google Scholar 

  5. Akyildiz, I. F., Lee, W., Vuran, M. C., & Mohanty, S. (2008). A survey on spectrum management in cognitive radio networks. IEEE Communications Magazine, 46(4), 40–48.

    Google Scholar 

  6. Mitola, J, I. I. I., & Maguire, G. Q, Jr. (1999). Cognitive radio: Making software radios more personal. IEEE Personal Communications, 6(4), 13–18.

    Google Scholar 

  7. Haykin, S. (2005). Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications, 23(2), 201–220.

    Google Scholar 

  8. Hossain, E., Niyato, D., & Han, Z. (2009). Dynamic spectrum access and management in cognitive radio networks. Cambridge: Cambridge University Press.

    Google Scholar 

  9. Lv, L., Chen, J., Ni, A., Ding, Q. Z., & Jiang, H. (2018). Cognitive non-orthogonal multiple access with cooperative relaying: A new wireless frontier for 5G spectrum sharing. IEEE Communications Magazine, 56(4), 188–195.

    Google Scholar 

  10. Goldsmith, A., Jafar, S. A., Maric, I., & Srinivasa, S. (2009). Breaking spectrum gridlock with cognitive radios: An information theoretic perspective. Proceedings of the IEEE, 97(5), 894–914.

    Google Scholar 

  11. Giang, H. T. H., Hoan, T. N. K., Thanh, P. D., & Koo, I. (2019). A POMDP-based long-term transmission rate maximization for cognitive radio networks with wireless-powered ambient backscatter. International Journal of Communication Systems,. https://doi.org/10.1002/dac.3993.

    Article  Google Scholar 

  12. Ding, Z., et al. (2017). Application of non-orthogonal multiple access in LTE and 5G networks. IEEE Communications Magazine, 55(2), 185–191.

    Google Scholar 

  13. Dai, L., Wang, B., Yuan, Y., Han, S., Chih-Lin, I., & Wang, Z. (2015). Non-orthogonal multiple access for 5G: Solutions, challenges, opportunities, and future research trends. IEEE Communications Magazine, 53(9), 74–81.

    Google Scholar 

  14. Wei, Z., Yuan, J., Ng, D. W. K., Elkashlan, M., & Ding, Z. (2016). A survey of downlink non-orthogonal multiple access for 5G wireless communication networks. ZTE Communications, 14(4), 17–26.

    Google Scholar 

  15. Saito, Y., Kishiyama,Y., Benjebbour, A., Nakamura, T., Li, A., & Higuchi, K. (2013). Non-orthogonal multiple access (NOMA) for cellular future radio access. In 2013 IEEE 77th vehicular technology conference (VTC Spring), Dresden (pp. 1–5).

  16. Ding, Z., Peng, M., & Poor, H. V. (2015). Cooperative non-orthogonal multiple access in 5G systems. IEEE Communications Letters, 19(8), 1462–1465.

    Google Scholar 

  17. Wan, D., Wen, M., Ji, F., Yu, H., & Chen, F. (2018). Non-orthogonal multiple access for cooperative communications: Challenges, opportunities, and trends. IEEE Wireless Communications, 25(2), 109–117.

    Google Scholar 

  18. Tabassum, H., Hossain, E., & Hossain, J. (2017). Modeling and analysis of up-link non-orthogonal multiple access in large-scale cellular networks using Poisson cluster processes. IEEE Transactions on Communications, 65(8), 3555–3570.

    Google Scholar 

  19. Razavi, R., Hoshyar, R., Imran, M. A., & Wang, Y. (2011). Information theoretic analysis of LDS scheme. IEEE Communications Letters, 15(8), 798–800.

    Google Scholar 

  20. AL-Imari, M., Imran, M. A. C., & Tafazolli, R. (2012). Low Density Spreading for next generation multicarrier cellular systems. In 2012 International conference on future communication networks, Baghdad (pp. 52–57).

  21. Du, Y., Dong, B., Chen, Z., Fang, J. C., & Wang, X. (2016). A fast convergence multiuser detection scheme for uplink SCMA systems. IEEE Wireless Communications Letters, 5(4), 388–391.

    Google Scholar 

  22. Nikopour, H., et al. (2014). SCMA for downlink multiple access of 5G wireless networks. In 2014 IEEE global communications conference, Austin, TX (pp. 3940–3945).

  23. Liu, Y., Ding, Z., Elkashlan, M., & Yuan, J. (2016). Nonorthogonal multiple access in large-scale underlay cognitive radio networks. IEEE Transactions on Vehicular Technology, 65(12), 10152–10157.

    Google Scholar 

  24. Wang, D., & Men, S. (2018). Secure energy efficiency for NOMA based cognitive radio networks with nonlinear energy harvesting. IEEE Access, 6, 62707–62716.

    Google Scholar 

  25. Lv, L., Ni, Q., Ding, Z., & Chen, J. (2017). Application of non-orthogonal multiple access in cooperative spectrum-sharing networks over Nakagami-\(m\) fading channels. IEEE Transactions on Vehicular Technology, 66(6), 5506–5511.

    Google Scholar 

  26. Lv, L., Chen, J., Ni, Q., & Ding, Z. (2017). Design of cooperative non-orthogonal multicast cognitive multiple access for 5G systems: User scheduling and performance analysis. IEEE Transactions on Communications, 65(6), 2641–2656.

    Google Scholar 

  27. Simjee, F. I., & Chou, P. H. (2008). Efficient charging of supercapacitors for extended lifetime of wireless sensor nodes. IEEE Transactions on Power Electronics, 23(3), 1526–1536.

    Google Scholar 

  28. Chen, Z., Law, M., Mak, P., & Martins, R. P. (2017). A single-chip solar energy harvesting IC using integrated photodiodes for biomedical implant applications. IEEE Transactions on Biomedical Circuits and Systems, 11(1), 44–53.

    Google Scholar 

  29. Wang, C., Li, J., Yang, Y., & Ye, F. (2018). Combining solar energy harvesting with wireless charging for hybrid wireless sensor networks. IEEE Transactions on Mobile Computing, 17(3), 560–576.

    Google Scholar 

  30. Stuyts, J., Horn, G., Vandermeulen, W., Driesen, J., & Diehl, M. (2015). Effect of the electrical energy conversion on optimal cycles for pumping airborne wind energy. IEEE Transactions on Sustainable Energy, 6(1), 2–10.

    Google Scholar 

  31. Zhao, L., Tang, L., Liang, J., & Yang, Y. (2017). Synergy of wind energy harvesting and synchronized switch harvesting interface circuit. IEEE/ASME Transactions on Mechatronics, 22(2), 1093–1103.

    Google Scholar 

  32. Zou, Z., Gidmark, A., Charalambous, T., & Johansson, M. (2016). Optimal radio frequency energy harvesting with limited energy arrival knowledge. IEEE Journal on Selected Areas in Communications, 34(12), 3528–3539.

    Google Scholar 

  33. Celik, A., Alsharoa, A., & Kamal, A. E. (2017). Hybrid energy harvesting-based cooperative spectrum sensing and access in heterogeneous cognitive radio networks. IEEE Transactions on Cognitive Communications and Networking, 3(1), 37–48.

    Google Scholar 

  34. Zhu, L., Zhang, J., Xiao, Z., Cao, X., Wu, D. O., & Xia, X. (2018). Joint power control and beamforming for uplink non-orthogonal multiple access in 5G millimeter-wave communications. IEEE Transactions on Wireless Communications, 17(9), 6177–6189.

    Google Scholar 

  35. Ali, M. S., Tabassum, H., & Hossain, E. (2016). Dynamic user clustering and power allocation for uplink and downlink non-orthogonal multiple access (NOMA) systems. IEEE Access, 4, 6325–6343.

    Google Scholar 

  36. Zhai, D., & Du, J. (2018). Spectrum efficient resource management for multi-carrier-based NOMA networks: A graph-based method. IEEE Wireless Communications Letters, 7(3), 388–391.

    Google Scholar 

  37. Thanh, P. D., Hoan, T. N. K., & Koo, I. (2020). Joint resource allocation and transmission mode selection using a POMDP-based hybrid half-duplex/full-duplex scheme for secrecy rate maximization in multi-channel cognitive radio networks. IEEE Sensors Journal, 20(7), 3930–3945.

    Google Scholar 

  38. Shah, H. A., Koo, I., & Kyung, S. K. (2019). Actor–critic-algorithm-based accurate spectrum sensing and transmission framework and energy conservation in energy-constrained wireless sensor network-based cognitive radios. Wireless Communications and Mobile Computing,. https://doi.org/10.1155/2019/6051201.

    Article  Google Scholar 

  39. Li, X., Fang, J., Cheng, W., Duan, H., Chen, Z., & Li, H. (2018). Intelligent power control for spectrum sharing in cognitive radios: A deep reinforcement learning approach. IEEE Access, 6, 25463–25473.

    Google Scholar 

  40. Yang, H., & Xie, X. (2020). An actor–critic deep reinforcement learning approach for transmission scheduling in cognitive internet of things systems. IEEE Systems Journal, 14(1), 51–60.

    Google Scholar 

  41. Zhang, Y., Wang, X. & Xu, Y. (2019). Energy-efficient resource allocation in uplink NOMA systems with deep reinforcement learning. In 2019 11th International conference on wireless communications and signal processing (WCSP), Xi’an, China (pp. 1–6).

  42. Zhang, J., Tao, X., Wu, H., Zhang, N., & Zhang, X. (2020). Deep reinforcement learning for throughput improvement of the uplink grant-free NOMA system. IEEE Internet of Things Journal, 7(7), 6369–6379.

    Google Scholar 

  43. Manimekalai, T., Joan, S. R., & Laxmikandan, T. (2020). Throughput maximization for underlay CR multicarrier NOMA network with cooperative communication. ETRI Journal, 42(6), 846–858.

    Google Scholar 

  44. Zhong, C., Lu, Z., Gursoy, M. C., & Velipasalar, S. (2018). Actor–critic deep reinforcement learning for dynamic multichannel access. In 2018 IEEE global conference on signal and information processing (GlobalSIP), Anaheim, CA, USA (pp. 599–603).

  45. Yang, Z., Feng, L., Chang, Z., Lu, J., Liu, R., Kadoch, M., & Cheriet, M. (2020). Prioritized uplink resource allocation in smart grid backscatter communication networks via deep reinforcement learning. Electronics, 9(4), 1–16.

    Google Scholar 

  46. Yucek, T., & Arslan, H. (2009). A survey of spectrum sensing algorithms for cognitive radio applications. IEEE Communications Surveys & Tutorials, 11(1), 116–130.

    Google Scholar 

  47. Cordeiro, C., & Challapali, K. (2005). Spectrum agile radios: Utilization and sensing architectures. In First IEEE international symposium on new frontiers in dynamic spectrum access networks (DySPAN 2005), Baltimore, MD, USA (pp. 160–169).

  48. Pawelczak, P., Janssen, G. J. M., & Prasad, R. V. (2006). WLC10-4: Performance measures of dynamic spectrum access networks. In IEEE Globecom 2006, San Francisco, CA (pp. 1–6).

  49. Liu, X., & Shankar, S. (2006). Sensing-based opportunistic channel access. Mobile Networks and Applications, 11(4), 577–591.

    Google Scholar 

  50. Cichoń, K., Kliks, A., & Bogucka, H. (2016). Energy-efficient cooperative spectrum sensing: A survey. IEEE Communications Surveys & Tutorials, 18(3), 1861–1886.

    Google Scholar 

  51. Quan, Z., Ma, W.-K., Cui, S., & Sayed, A. (2010). Optimal linear fusion for distributed detection via semidefinite programming. IEEE Transaction Signal Processing, 58(4), 2431–2436.

    MathSciNet  MATH  Google Scholar 

  52. Ribeiro, F., de Campos, M., & Werner, S. (2012). Distributed cooperative spectrum sensing with adaptive combining. In 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), Kyoto, Japan (pp. 3557–3560).

  53. Han, W., Li, J., Li, Z., Si, J., & Zhang, Y. (2013). Efficient soft decision fusion rule in cooperative spectrum sensing. IEEE Transaction Signal Processing, 61(8), 1931–1943.

    MathSciNet  MATH  Google Scholar 

  54. Stevenson, C. R., Chouinard, G., Lei, Z., Hu, W., Shellhammer, S. J., & Caldwell, W. (2009). IEEE 802.22 The first cognitive radio wireless regional area network standard. IEEE Communications Magazine, 47(1), 130–138.

    Google Scholar 

  55. Liang, Y., Zeng, Y., Peh, E. C. Y., & Hoang, A. T. (2008). Sensing-throughput tradeoff for cognitive radio networks. IEEE Transactions on Wireless Communications, 7(4), 1326–1337.

    Google Scholar 

  56. Crites, R. H., & Barto, A. G. (1995). An actor/critic algorithm that is equivalent to Q-learning. In Advances in neural information processing systems, Denver, CO (Vol. 7, pp. 401–408).

  57. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge: MIT Press.

    MATH  Google Scholar 

  58. Singh, S., Jaakkola, T., Littman, M., & Szepesvri, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38(3), 287–308.

    MATH  Google Scholar 

  59. Stone, J. V. (2013). Bayes’ rule: A tutorial introduction to Bayesian analysis. Sheffield: Sebtel Press.

    MATH  Google Scholar 

  60. Konda, V. R., & Tsitsiklis, J. N. (2000). Actor–critic algorithms. In Advances in neural information processing systems, CO (Vol. 12, pp. 1008–1014).

  61. Wang, K. (2018). Optimally myopic scheduling policy for downlink channels with imperfect state observation. IEEE Transactions on Vehicular Technology, 67(7), 5856–5867.

    Google Scholar 

  62. Nguyen, T., Nguyen, V., Lee, J., & Kim, Y. (2019). Sum rate maximization for multi-user wireless powered IoT network with non-linear energy harvester: Time and power allocation. IEEE Access, 7, 149698–149710.

    Google Scholar 

  63. Islam, S. M. R., Avazov, N., Dobre, O. A., & Kwak, K. (2017). Power-domain non-orthogonal multiple access (NOMA) in 5G systems: Potentials and challenges. IEEE Communications Surveys & Tutorials, 19(2), 721–742.

    Google Scholar 

  64. Thanh, P. D., Hoan, T. N. K., Vu-Van, H., & Koo, I. (2019). Efficient attack strategy for legitimate energy-powered eavesdropping in tactical cognitive radio networks. Wireless Networks, 25(6), 3605–3622.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering, University of Ulsan (UOU), Ulsan, Republic of Korea

    Hoang Thi Huong Giang & Insoo Koo

  2. Can Tho University, Can Tho, Vietnam

    Tran Nhut Khai Hoan

Authors
  1. Hoang Thi Huong Giang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tran Nhut Khai Hoan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Insoo Koo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Insoo Koo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Giang, H.T.H., Hoan, T.N.K. & Koo, I. Uplink NOMA-based long-term throughput maximization scheme for cognitive radio networks: an actor–critic reinforcement learning approach. Wireless Netw 27, 1319–1334 (2021). https://doi.org/10.1007/s11276-020-02520-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-020-02520-y

Keywords

Search

Navigation