Skip to main content
SpringerLink
Log in

Reinforcement Learning Based Angle-of-Arrival Detection for Millimeter-Wave Software-Defined Radio Systems

  • Conference paper
  • First Online:
Internet of Things. Advances in Information and Communication Technology (IFIPIoT 2023)

Part of the book series: IFIP Advances in Information and Communication Technology ((IFIPAICT,volume 683))

Included in the following conference series:

Abstract

Millimeter-wave (mmWave) signals experience severe environmental path loss. To mitigate the path loss, beam-forming methods are used to realize directional mmWave beams that can travel longer. Yet, advanced algorithms are needed to track these directional beams by detecting angle-of-arrival (AoA) and aligning the transmit and receive antennas. To realize these advanced beam-forming algorithms in real world scenarios, Software-Defined Radio (SDR) platforms that allow both high-level programming capability and mmWave beam-forming are needed. Using a low-cost mmWave SDR platform, we design and prototype two reinforcement learning (RL) algorithms for AoA detection, i.e., Q- and Double Q-learning. We evaluate these algorithms and study the trade-offs involved in their design.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Seth, S., Yuksel, M., Vosoughi, A.: Forming coalition sets from directional radios. In: MILCOM 2022–2022 IEEE Military Communications Conference (MILCOM), pp. 507–514 (2022)

    Google Scholar 

  2. Jilani, S.F., Munoz, M.O., Abbasi, Q.H., Alomainy, A.: Millimeter-wave liquid crystal polymer based conformal antenna array for 5G applications. IEEE Antennas Wirel. Propag. Lett. 18(1), 84–88 (2019)

    Article  Google Scholar 

  3. Dhruva, T., Rajesh, A., Abirami, K.: Design of conformal multi-band mmWave wearable antenna for 5G applications. In: Proceedings of IEEE International Conference on Smart Electronics and Communication (ICOSEC), pp. 573–577 (2022)

    Google Scholar 

  4. Hu, K., Zhou, Y., Sitaraman, S.K., Tentzeris, M.M.: Fully additively manufactured flexible dual-band slotted patch antenna for 5G/mmWave wearable applications. In: Proceedings of IEEE International Symposium on Antennas and Propagation and USNC-URSI Radio Science Meeting (AP-S/URSI), pp. 878–879 (2022)

    Google Scholar 

  5. Bogale, T., Wang, X., Le, L.: Chapter 9 - mmwave communication enabling techniques for 5g wireless systems: a link level perspective. In: Mumtaz, S., Rodriguez, J., Dai, L. (eds.) mmWave Massive MIMO, pp. 195–225. Academic Press (2017). https://www.sciencedirect.com/science/article/pii/B9780128044186000091

  6. Jean, M., Velazquez, E., Gong, X., Yuksel, M.: A 30 ghz steerable patch array antenna for software-defined radio platforms. SoutheastCon 2023, 856–860 (2023)

    Google Scholar 

  7. Fan, W., Xia, Y., Li, C., Huang, Y.: Channel tracking and aoa acquisition in quantized millimeter wave MIMO systems. IEEE Trans. Vehicular Technol., 1–15 (2023)

    Google Scholar 

  8. Yazdani, H., Seth, S., Vosoughi, A., Yuksel, M.: Throughput-optimal d2d mmwave communication: Joint coalition formation, power, and beam optimization. In: 2022 IEEE Wireless Communications and Networking Conference (WCNC), pp. 1539–1544 (2022)

    Google Scholar 

  9. Dai, Z., He, Y., Vu, T., Trigoni, N., Markham, A.: Deepaoanet: learning angle of arrival from software defined radios with deep neural networks (2021)

    Google Scholar 

  10. Chowdhury, M.M.U., Erden, F., Guvenc, I.: Rss-based q-learning for indoor uav navigation. In: MILCOM 2019–2019 IEEE Military Communications Conference (MILCOM), pp. 121–126 (2019)

    Google Scholar 

  11. Watkins, C., Dayan, P.: Technical note: Q-learning. Mach. Learn. 8, 279–292 (1992)

    Article  MATH  Google Scholar 

  12. Van Hasselt, H.: Double q-learning, pp. 2613–2621 (Jan 2010)

    Google Scholar 

  13. Jaakkola, T., Jordan, M.I., Singh, S.P.: On the convergence of stochastic iterative dynamic programming algorithms. Neural Comput. 6(6), 1185–1201 (1994)

    Article  MATH  Google Scholar 

  14. Tsitsiklis, J.: Asynchronous stochastic approximation and q-learning. In: Proceedings of 32nd IEEE Conference on Decision and Control, vol. 1, pp. 395–400 (1993)

    Google Scholar 

  15. GNU Radio - the free and open source radio ecosystem, GNU Radio. https://www.gnuradio.org

  16. Zafari, F., Gkelias, A., Leung, K.K.: A survey of indoor localization systems and technologies. IEEE Commun. Surv. Tutorials 21(3), 2568–2599 (2019)

    Article  Google Scholar 

  17. Wu, K., Ni, W., Su, T., Liu, R.P., Guo, Y.J.: Recent breakthroughs on angle-of-arrival estimation for millimeter-wave high-speed railway communication. IEEE Commun. Mag. 57(9), 57–63 (2019)

    Article  Google Scholar 

  18. Şahin, A., Sichitiu, M.L., Guvenç, İ.: A Millimeter-Wave Software-Defined Radio for Wireless Experimentation, arXiv:2302.08444, (Feb. 2023)

  19. Gu, X.: A multilayer organic package with 64 dual-polarized antennas for 28ghz 5g communication. In: 2017 IEEE MTT-S International Microwave Symposium (IMS), pp. 1899–1901 (2017)

    Google Scholar 

  20. Aerpaw: Aerial experimentation and research platform for advanced wireless. https://aerpaw.org/

  21. Alkhateeb, A.: Deepmimo: a generic deep learning dataset for millimeter wave and massive MIMO applications. CoRR, vol. abs/ arXiv: 1902.06435, (2019)

  22. Kaveh, M., Barabell, A.: The statistical performance of the music and the minimum-norm algorithms in resolving plane waves in noise. IEEE Trans. Acoust. Speech Signal Process. 34(2), 331–341 (1986)

    Article  Google Scholar 

  23. Pastorino, M., Randazzo, A.: A smart antenna system for direction of arrival estimation based on a support vector regression. IEEE Trans. Antennas Propag. 53, 2161–2168 (2005)

    Article  Google Scholar 

  24. Jean, M., Yuksel, M., Gong, X.: Millimeter-wave software-defined radio testbed with programmable directionality. In: Proceedings IEEE INFOCOM Workshop 2023 (in press)

    Google Scholar 

  25. UBX 10–6000 MHz Rx/TX (40 MHz, N series and X series), Ettus Research, a National Instruments Brand. https://www.ettus.com/all-products/ubx40

  26. ADMV1013, Analog Devices. https://www.analog.com/en/products/admv1013.html

  27. ADMV1014, Analog Devices. https://www.analog.com/en/products/admv1014.html

  28. Kearns, M., Singh, S.: Finite-sample convergence rates for q-learning and indirect algorithms. In: Proceedings of the 1998 Conference on Advances in Neural Information Processing Systems II, pp. 996–1002. MIT Press, Cambridge, MA, USA (1999)

    Google Scholar 

  29. Ernst, D., Geurts, P., Wehenkel, L.: Tree-based batch mode reinforcement learning. J. Mach. Learn. Res. 6, 503–556 (2005)

    MathSciNet  MATH  Google Scholar 

  30. Szepesvári, C.: The asymptotic convergence-rate of q-learning. In: Proceedings of the 10th International Conference on Neural Information Processing Systems, ser. NIPS 1997, pp. 1064–1070. MIT Press, Cambridge, MA, USA (1997)

    Google Scholar 

Download references

Acknowledgment

This was was supported in part by the U.S. National Science Foundation award 1836741.

Author information

Authors and Affiliations

  1. University of Central Florida, Orlando, FL, 32826, USA

    Marc Jean & Murat Yuksel

Authors
  1. Marc Jean
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Murat Yuksel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Marc Jean or Murat Yuksel .

Editor information

Editors and Affiliations

  1. Khalifa University, Abu Dhabi, United Arab Emirates

    Deepak Puthal

  2. University of North Texas, Denton, TX, USA

    Saraju Mohanty

  3. University of Missouri at Kansas City, Kansas City, MO, USA

    Baek-Young Choi

Rights and permissions

Reprints and permissions

Copyright information

© 2024 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jean, M., Yuksel, M. (2024). Reinforcement Learning Based Angle-of-Arrival Detection for Millimeter-Wave Software-Defined Radio Systems. In: Puthal, D., Mohanty, S., Choi, BY. (eds) Internet of Things. Advances in Information and Communication Technology. IFIPIoT 2023. IFIP Advances in Information and Communication Technology, vol 683. Springer, Cham. https://doi.org/10.1007/978-3-031-45878-1_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-45878-1_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-45877-4

  • Online ISBN: 978-3-031-45878-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation