Skip to main content
SpringerLink
Log in

Intelligent routing method based on Dueling DQN reinforcement learning and network traffic state prediction in SDN

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

The traditional routing method makes use of limited information on the network links to make routing decisions, which makes it difficult to adapt to the dynamic and complex network and adjust the router’s forward strategy. To address these issues, this paper proposes an intelligent routing method based on the Software Defined Network (SDN), Dueling DQN (a Deep Reinforcement Learning algorithm) and network traffic state prediction. First, the global network awareness information is obtained with the SDN network measurement mechanism, which is converted into a traffic matrix consisting of multiple network link status information such as bandwidth and delay, etc. Then, the optimal forwarding route under the current network state is generated by predicting the network traffic matrix and the Dueling DQN. The experimental results show that: (1) compared with the traditional Dijkstra and OSPF routing methods, the proposed method significantly improves the network throughput and effectively reduces the network delay and packet loss rate; (2) comparing with the reinforcement learning algorithms DDPG and PPO, the proposed approach achieves a faster convergence state, which improves the efficiency of network routing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Nunes, B. A. A., Mendonca, M., Nguyen, X. N., Obraczka, K., & Turletti, T. (2014). A survey of software-defined networking: Past, present, and future of programmable networks. IEEE Communications Surveys and Tutorials, 16(3), 1617–1634. https://doi.org/10.1109/surv.2014.012214.00180

    Article  Google Scholar 

  2. Sun, P., Yu, M., Freedman, M. J., Rexford, J., & Walker, D. (2015). Hone: Joint host-network traffic management in software-defined networks. Journal of Network and Systems Management, 23(2), 374–399. https://doi.org/10.1007/s10922-014-9321-9

    Article  Google Scholar 

  3. Guerin, R. A., Orda, A., & Williams, D. (1997). QoS routing mechanisms and OSPF extensions. In GLOBECOM 97. IEEE Global Telecommunications Conference, pp. 1903–1908. IEEE. https://doi.org/10.17487/rfc2676

  4. Verma, A., & Bhardwaj, N. (2016). A review on routing information protocol (RIP) and open shortest path first (OSPF) routing protocol. International Journal of Future Generation Communication and Networking, 9(4), 161–170. https://doi.org/10.14257/ijfgcn.2016.9.4.13

    Article  Google Scholar 

  5. Ni, W., Huang, C., Wu, J., & Savoie, M. (2013). Availability of survivable Valiant load balancing (VLB) networks over optical networks. Optical Switching and Networking, 10(3), 274–289. https://doi.org/10.1016/j.osn.2013.02.002

    Article  Google Scholar 

  6. Ibarz, J., Tan, J., Finn, C., Kalakrishnan, M., Pastor, P., & Levine, S. (2021). How to train your robot with deep reinforcement learning: Lessons we have learned. The International Journal of Robotics Research, 40(4–5), 698–721. https://doi.org/10.1177/0278364920987859

    Article  Google Scholar 

  7. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444. https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  8. Botvinick, M., Ritter, S., Wang, J. X., Kurth-Nelson, Z., Blundell, C., & Hassabis, D. (2019). Reinforcement learning, fast and slow. Trends in Cognitive Sciences, 23(5), 408–422. https://doi.org/10.1016/j.tics.2019.02.006

    Article  Google Scholar 

  9. Sherstinsky, A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306. https://doi.org/10.1016/j.physd.2019.132306

    Article  MathSciNet  MATH  Google Scholar 

  10. Ahn, C. W., & Ramakrishna, R. S. (2002). A genetic algorithm for shortest path routing problem and the sizing of populations. IEEE Transactions on Evolutionary Computation, 6(6), 566–579. https://doi.org/10.1109/tevc.2002.804323

    Article  Google Scholar 

  11. Derbel, H., Jarboui, B., Hanafi, S., & Chabchoub, H. (2012). Genetic algorithm with iterated local search for solving a location-routing problem. Expert Systems with Applications, 39(3), 2865–2871. https://doi.org/10.1016/j.eswa.2011.08.146

    Article  MATH  Google Scholar 

  12. Zhang, D. G., Liu, S., Liu, X. H., Zhang, T., & Cui, Y. Y. (2018). Novel dynamic source routing protocol (DSR) based on genetic algorithm-bacterial foraging optimization (GA-BFO). International Journal of Communication Systems, 31(18), 1–20. https://doi.org/10.1002/dac.3824

    Article  Google Scholar 

  13. Parsaei, M. R., Mohammadi, R., & Javidan, R. (2017). A new adaptive traffic engineering method for telesurgery using ACO algorithm over software defined networks. European Research in Telemedicine/La Recherche Europeenne en Telemedecine, 6(3–4), 173–180. https://doi.org/10.1016/j.eurtel.2017.10.003

    Article  Google Scholar 

  14. Jing, S., Muqing, W., Yong, B., & Min, Z. (2017). An improved GAC routing algorithm based on SDN. IEEE International Conference on Computer and Communications (ICCC), pp. 173–176. https://doi.org/10.1109/compcomm.2017.8322535

  15. Lin, C., Wang, K., & Deng, G. (2017). A QoS-aware routing in SDN hybrid networks. Procedia Computer Science, 110, 242–249. https://doi.org/10.1016/j.procs.2017.06.091

    Article  Google Scholar 

  16. Truong Dinh, K., Kukliński, S., Osiński, T., & Wytrębowicz, J. (2020). Heuristic traffic engineering for SDN. Journal of Information and Telecommunication, 4(3), 251–266. https://doi.org/10.1080/24751839.2020.1755528

    Article  Google Scholar 

  17. Ke, C. K., Wu, M. Y., Hsu, W. H., & Chen, C. Y. (2019). Discover the optimal IoT packets routing path of software-defined network via artificial bee colony algorithm. In International Wireless Internet Conference, pp. 147–162. Springer, Cham. https://doi.org/10.1007/978-3-030-52988-8_13

  18. Shokouhifar, M. (2021). FH-ACO: Fuzzy heuristic-based ant colony optimization for joint virtual network function placement and routing. Applied Soft Computing, 107, 107401. https://doi.org/10.1016/j.asoc.2021.107401

    Article  Google Scholar 

  19. Zhang, L., & Lei, Y. (2021). Particle swarm optimization-based information-centric networking intra-domain routing strategy. Internet Technology Letters, 4(1), e196. https://doi.org/10.1002/itl2.196

    Article  Google Scholar 

  20. Valadarsky, A., Schapira, M., Shahaf, D., & Tamar, A. (2017). Learning to route. In Proceedings of the 16th ACM workshop on hot topics in networks, pp. 185–191. https://doi.org/10.1145/3152434.3152441

  21. Sharma, D. K., Dhurandher, S. K., Woungang, I., Srivastava, R. K., Mohananey, A., & Rodrigues, J. J. (2016). A machine learning-based protocol for efficient routing in opportunistic networks. IEEE Systems Journal, 12(3), 2207–2213. https://doi.org/10.1109/jsyst.2016.2630923

    Article  Google Scholar 

  22. Li, W., Li, G., & Yu, X. (2015). A fast traffic classification method based on SDN network. In The 4th International Conference on Electronics, Communications and Networks, pp. 223–229. Beijing, China. https://doi.org/10.1201/b18592-42

  23. Zhou, X., Su, M., Liu, Z., Hu, Y., Sun, B., & Feng, G. (2020). Smart tour route planning algorithm based on naïve Bayes interest data mining machine learning. ISPRS International Journal of Geo-Information, 9(2), 112. https://doi.org/10.3390/ijgi9020112

    Article  Google Scholar 

  24. Yanjun, L., Xiaobo, L., & Osamu, Y. (2014). Traffic engineering framework with machine learning based meta-layer in software-defined networks. In 2014 4th IEEE International Conference on Network Infrastructure and Digital Content, pp. 121–125. IEEE. https://doi.org/10.1109/icnidc.2014.7000278

  25. Tang, F., Mao, B., Fadlullah, Z. M., Kato, N., Akashi, O., Inoue, T., & Mizutani, K. (2017). On removing routing protocol from future wireless networks: A real-time deep learning approach for intelligent traffic control. IEEE Wireless Communications, 25(1), 154–160. https://doi.org/10.1109/mwc.2017.1700244

    Article  Google Scholar 

  26. Mao, B., Tang, F., Fadlullah, Z. M., & Kato, N. (2019). An intelligent route computation approach based on real-time deep learning strategy for software defined communication systems. IEEE Transactions on Emerging Topics in Computing, 9(3), 1554–1565. https://doi.org/10.1109/tetc.2019.2899407

    Article  Google Scholar 

  27. Kato, N., Fadlullah, Z. M., Mao, B., Tang, F., Akashi, O., Inoue, T., & Mizutani, K. (2016). The deep learning vision for heterogeneous network traffic control: Proposal, challenges, and future perspective. IEEE Wireless Communications, 24(3), 146–153. https://doi.org/10.1109/mwc.2016.1600317wc

    Article  Google Scholar 

  28. Hendriks, T., Camelo, M., & Latré, S. (2018). Q 2-routing: A Qos-aware Q-routing algorithm for wireless ad hoc networks. In 2018 14th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob), pp. 108–115. IEEE. https://doi.org/10.1109/wimob.2018.8589161

  29. Chen, T., Gao, X., Liao, T., & Chen, G. (2019). Pache: A packet management scheme of cache in data center networks. IEEE Transactions on Parallel and Distributed Systems, 31(2), 253–265. https://doi.org/10.1109/tpds.2019.2931905

    Article  Google Scholar 

  30. Casas-Velasco, D. M., Rendon, O. M. C., & da Fonseca, N. L. (2020). Intelligent routing based on reinforcement learning for software-defined networking. IEEE Transactions on Network and Service Management, 18(1), 870–881. https://doi.org/10.1109/tnsm.2020.3036911

    Article  Google Scholar 

  31. Jin, Z., Zang, W., Jiang, Y., & Lan, J. (2019). A QLearning based business differentiating routing mechanism in SDN architecture. Journal of Physics: Conference Series, 1168(2), 022025. https://doi.org/10.1088/1742-6596/1168/2/022025

    Article  Google Scholar 

  32. Yin, Y., Huang, C., Wu, D. F., Huang, S., Ashraf, M., & Guo, Q. (2021). Reinforcement learning-based routing algorithm in satellite-terrestrial integrated networks. Wireless Communications and Mobile Computing. https://doi.org/10.1155/2021/3759631

    Article  Google Scholar 

  33. Zhao, L., Wang, J., Liu, J., & Kato, N. (2019). Routing for crowd management in smart cities: A deep reinforcement learning perspective. IEEE Communications Magazine, 57(4), 88–93. https://doi.org/10.1109/mcom.2019.1800603

    Article  Google Scholar 

  34. Chen, Y. R., Rezapour, A., Tzeng, W. G., & Tsai, S. C. (2020). Rl-routing: An sdn routing algorithm based on deep reinforcement learning. IEEE Transactions on Network Science and Engineering, 7(4), 3185–3199. https://doi.org/10.1109/tnse.2020.3017751

    Article  Google Scholar 

  35. Zhang, J., Ye, M., Guo, Z., Yen, C. Y., & Chao, H. J. (2020). CFR-RL: Traffic engineering with reinforcement learning in SDN. IEEE Journal on Selected Areas in Communications, 38(10), 2249–2259. https://doi.org/10.1109/jsac.2020.3000371

    Article  Google Scholar 

  36. Fu, Q., Sun, E., Meng, K., Li, M., & Zhang, Y. (2020). Deep Q-learning for routing schemes in SDN-based data center networks. IEEE Access, 8, 103491–103499. https://doi.org/10.1109/access.2020.2995511

    Article  Google Scholar 

  37. Liu, W. X., Cai, J., Chen, Q. C., & Wang, Y. (2021). DRL-R: Deep reinforcement learning approach for intelligent routing in software-defined data-center networks. Journal of Network and Computer Applications, 177, 102865. https://doi.org/10.1016/j.jnca.2020.102865

    Article  Google Scholar 

  38. Hossain, M. B., & Wei, J. (2019). Reinforcement learning-driven QoS-aware intelligent routing for software-defined networks. In 2019 IEEE global conference on signal and information processing (GlobalSIP) , pp. 1–5. IEEE. https://doi.org/10.1109/globalsip45357.2019.8969320

  39. Yu, C., Lan, J., Guo, Z., & Hu, Y. (2018). DROM: Optimizing the routing in software-defined networks with deep reinforcement learning. IEEE Access, 6, 64533–64539. https://doi.org/10.1109/access.2018.2877686

    Article  Google Scholar 

  40. Zhang, D., & Kabuka, M. R. (2018). Combining weather condition data to predict traffic flow: A GRU-based deep learning approach. IET Intelligent Transport Systems, 12(7), 578–585. https://doi.org/10.1109/mwscas.2017.8053243

    Article  Google Scholar 

  41. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  42. Clark, D. D., Partridge, C., Ramming, J. C., & Wroclawski, J. T. (2003). A knowledge plane for the internet. In Proceedings of the 2003 conference on Applications, technologies, architectures, and protocols for computer communications, pp. 3–10. https://doi.org/10.1145/863955.863957

  43. Mestres, A., Rodriguez-Natal, A., Carner, J., Barlet-Ros, P., Alarcón, E., Solé, M., Muntés-Mulero, V., Meyer, D., Barkai, S., Hibbett, M. J., & Estrada, G. (2017). Knowledge-defined networking. ACM SIGCOMM Computer Communication Review., 47(3), 2–10. https://doi.org/10.1145/3138808.3138810

    Article  Google Scholar 

  44. Xue, X., & Huang, Q. (2022). Generative adversarial learning for optimizing ontology alignment. Expert Systems. https://doi.org/10.1111/exsy.12936

    Article  Google Scholar 

  45. Al Shalabi, L., & Shaaban, Z. (2006). Normalization as a preprocessing engine for data mining and the approach of preference matrix. In 2006 International conference on dependability of computer systems, pp. 207–214. IEEE. https://doi.org/10.1109/depcos-relcomex.2006.38

  46. Casas-Velasco, D. M., Rendon, O. M. C., & da Fonseca, N. L. (2021). DRSIR: A deep reinforcement learning approach for routing in software-defined networking. IEEE Transactions on Network and Service Management. https://doi.org/10.1109/tnsm.2021.3132491

    Article  Google Scholar 

  47. Ban, T. W. (2020). An autonomous transmission scheme using dueling DQN for D2D communication networks. IEEE Transactions on Vehicular Technology, 69(12), 16348–16352. https://doi.org/10.1109/tvt.2020.3041458

    Article  Google Scholar 

  48. White, S. R., Hanson, J. E., Whalley, I., Chess, D. M., & Kephart, J. O. (2004). An architectural approach to autonomic computing. In International Conference on Autonomic Computing, 2004. Proceedings, pp. 2–9. IEEE. https://doi.org/10.1109/icac.2004.1301340

  49. Mininet. Accessed: Jan. 5, 2021. [Online]. Available: http://mininet.org/

  50. Ryu. Accessed: Dec. 31, 2020. [Online]. Available: https://github.com/faucetsdn/ryu

  51. IPerf. Accessed: Jan. 5, 2021. [Online]. Available: https://iperf.fr/

  52. New York Metro IBX data center data sheet. Accessed: Dec. 31, 2020[Online]Available:https://www.equinix.com/resources/data-sheets/nyc-metro-data-sheet/

  53. Li, Y., Cai, Z. P., & Xu, H. (2018). LLMP: Exploiting LLDP for latency measurement in software-defined data center networks. Journal of Computer Science and Technology, 33(2), 277–285. https://doi.org/10.1007/s11390-018-1819-2

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 62161006, No. 61861013 and No. 61662018, in part by the Science and Technology Major Project of Guangxi No. AA18118031, in part by Guangxi Natural Science Foundation of China under Grant No. 2018GXNSFAA050028, in part by Director Fund project of Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education under Grant No. CRKL190102, and in part by Guangxi Key Laboratory of Wireless Wide band Communication and Signal Processing No. GXKL06220110.

Source code

The experimental code can be accessed at https://github.com/GuetYe/experiment-code

Author information

Authors and Affiliations

  1. School of Computer Science and Information Security, Guilin University of Electronic Technology, Guilin, 541004, China

    Linqiang Huang & Yong Wang

  2. Guangxi Key Laboratory of Wireless Wideband Communication and Signal Processing, Guilin University of Electronic Technology, Guilin, 541004, China

    Miao Ye, Hongbing Qiu & Xiaofang Deng

  3. Fujian Provincial Key Laboratory of Big Data Mining and Applications, Fujian University of Technology, Fuzhou, 350118, Fujian, China

    Miao Ye & Xingsi Xue

Authors
  1. Linqiang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miao Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xingsi Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongbing Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaofang Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Miao Ye.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Data availability

The dataset generated during this study by the SDN multi-threaded measurement mechanism designed in this paper through the flow measurement, which includes 1616 flow matrices, can be obtained from the author or accessed at https://github.com/GuetYe/experiment-data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, L., Ye, M., Xue, X. et al. Intelligent routing method based on Dueling DQN reinforcement learning and network traffic state prediction in SDN. Wireless Netw (2022). https://doi.org/10.1007/s11276-022-03066-x

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11276-022-03066-x

Keywords

Search

Navigation