Reinforcement Learning for Autonomous Defence in Software-Defined Networking

  • Yi HanEmail author
  • Benjamin I. P. Rubinstein
  • Tamas Abraham
  • Tansu Alpcan
  • Olivier De Vel
  • Sarah Erfani
  • David Hubczenko
  • Christopher Leckie
  • Paul Montague
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11199)


Despite the successful application of machine learning (ML) in a wide range of domains, adaptability—the very property that makes machine learning desirable—can be exploited by adversaries to contaminate training and evade classification. In this paper, we investigate the feasibility of applying a specific class of machine learning algorithms, namely, reinforcement learning (RL) algorithms, for autonomous cyber defence in software-defined networking (SDN). In particular, we focus on how an RL agent reacts towards different forms of causative attacks that poison its training process, including indiscriminate and targeted, white-box and black-box attacks. In addition, we also study the impact of the attack timing, and explore potential countermeasures such as adversarial training.


Adversarial reinforcement learning Software-defined networking Cyber security Adversarial training 


  1. 1.
    Amazon EC2 Instance Types – Amazon Web Services (AWS).
  2. 2.
  3. 3.
    Mininet: An Instant Virtual Network on your Laptop (2017).
  4. 4.
    OpenDaylight (2017).
  5. 5.
    Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. arXiv:1802.00420 [cs], February 2018
  6. 6.
    Barreno, M., Nelson, B., Joseph, A.D., Tygar, J.D.: The security of machine learning. Mach. Learn. 81(2), 121–148 (2010)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Beaudoin, L.: Autonomic computer network defence using risk states and reinforcement learning. Ph.D. thesis, University of Ottawa (Canada) (2009)Google Scholar
  8. 8.
    Behzadan, V., Munir, A.: Vulnerability of deep reinforcement learning to policy induction attacks. eprint arXiv:1701.04143 (2017)CrossRefGoogle Scholar
  9. 9.
    Bhagoji, A.N., Cullina, D., Mittal, P.: Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. arXiv:1704.02654 (2017)
  10. 10.
    Biggio, B., et al.: Security evaluation of support vector machines in adversarial environments. In: Ma, Y., Guo, G. (eds.) Support Vector Machines Applications, pp. 105–153. Springer, Cham (2014). Scholar
  11. 11.
    Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. In: Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 1467–1474. Omnipress, Edinburgh (2012)Google Scholar
  12. 12.
    Burkard, C., Lagesse, B.: Analysis of causative attacks against SVMs learning from data streams. In: Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics, pp. 31–36. ACM, New York (2017)Google Scholar
  13. 13.
    Carlini, N., Wagner, D.: Defensive distillation is not robust to adversarial examples. arXiv:1607.04311 (2016)
  14. 14.
    Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. eprint arXiv:1608.04644 (2016)
  15. 15.
    Carlini, N., Wagner, D.: Adversarial examples are not easily detected: bypassing ten detection methods. eprint arXiv:1705.07263 (2017)
  16. 16.
    Chung, S.P., Mok, A.K.: Advanced allergy attacks: does a corpus really help? In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 236–255. Springer, Heidelberg (2007). Scholar
  17. 17.
    Das, N., et al.: Keeping the bad guys out: protecting and vaccinating deep learning with JPEG compression. eprint arXiv:1705.02900, May 2017
  18. 18.
    Diakonikolas, I., Kamath, G., Kane, D.M., Li, J., Moitra, A., Stewart, A.: Robust estimators in high dimensions without the computational intractability. In: Proceedings of the 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 655–664, October 2016Google Scholar
  19. 19.
    Everitt, T., Krakovna, V., Orseau, L., Hutter, M., Legg, S.: Reinforcement learning with a corrupted reward channel. eprint arXiv:1705.08417 (2017)
  20. 20.
    Feinman, R., Curtin, R.R., Shintre, S., Gardner, A.B.: Detecting adversarial samples from artifacts. eprint arXiv:1703.00410 (2017)
  21. 21.
    Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. eprint arXiv:1412.6572 (2014)
  22. 22.
    Han, Y., Rubinstein, B.I.P.: Adequacy of the gradient-descent method for classifier evasion attacks. arXiv:1704.01704, April 2017
  23. 23.
    Hasselt, H.V.: Double Q-learning. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) Advances in Neural Information Processing Systems 23, pp. 2613–2621. Curran Associates, Inc. (2010)Google Scholar
  24. 24.
    Hasselt, H.V., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. eprint arXiv:1509.06461, September 2015
  25. 25.
    He, W., Wei, J., Chen, X., Carlini, N., Song, D.: Adversarial example defenses: ensembles of weak defenses are not strong. eprint arXiv:1706.04701 (2017)
  26. 26.
    Hosseini, H., Chen, Y., Kannan, S., Zhang, B., Poovendran, R.: Blocking transferability of adversarial examples in black-box learning systems. eprint arXiv:1703.04318 (2017)
  27. 27.
    Huang, L., Joseph, A.D., Nelson, B., Rubinstein, B.I., Tygar, J.: Adversarial machine learning. In: Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, pp. 43–58. ACM (2011)Google Scholar
  28. 28.
    Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies. eprint arXiv:1702.02284 (2017)
  29. 29.
    Koh, P.W., Liang, P.: understanding black-box predictions via influence functions. arXiv:1703.04730 [cs, stat], March 2017
  30. 30.
    Laishram, R., Phoha, V.V.: Curie: a method for protecting SVM Classifier from poisoning attack. arXiv:1606.01584 [cs], June 2016
  31. 31.
    Li, B., Vorobeychik, Y.: Feature cross-substitution in adversarial classification. In: Proceedings of the 2014 NIPS, NIPS 2014, pp. 2087–2095, MIT Press, Cambridge (2014)Google Scholar
  32. 32.
    Li, B., Wang, Y., Singh, A., Vorobeychik, Y.: Data poisoning attacks on factorization-based collaborative filtering. eprint arXiv:1608.08182 (2016)
  33. 33.
    Li, X., Li, F.: Adversarial examples detection in deep networks with convolutional filter statistics. arXiv:1612.07767 [cs], December 2016
  34. 34.
    Lin, Y.C., Hong, Z.W., Liao, Y.H., Shih, M.L., Liu, M.Y., Sun, M.: Tactics of adversarial attack on deep reinforcement learning agents. eprint arXiv:1703.06748, March 2017
  35. 35.
    Medved, J., Varga, R., Tkacik, A., Gray, K.: OpenDaylight: towards a model-driven SDN controller architecture. In: Proceedings of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, pp. 1–6 (2014)Google Scholar
  36. 36.
    Mei, S., Zhu, X.: Using machine teaching to identify optimal training-set attacks on machine learners. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2871–2877. AAAI Press, Austin (2015)Google Scholar
  37. 37.
    Metzen, J.H., Genewein, T., Fischer, V., Bischoff, B.: On detecting adversarial perturbations. eprint arXiv:1702.04267 (2017)
  38. 38.
    Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, ICML 2016, vol. 48, pp. 1928–1937., New York (2016)Google Scholar
  39. 39.
    Mnih, V., et al.: Playing Atari with Deep Reinforcement Learning. CoRR abs/1312.5602 (2013)Google Scholar
  40. 40.
    Moore, D., Shannon, C., Brown, D.J., Voelker, G.M., Savage, S.: Inferring internet denial-of-service activity. ACM Trans. Comput. Syst. 24(2), 115–139 (2006)CrossRefGoogle Scholar
  41. 41.
    Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. eprint arXiv:1610.08401 (2016)
  42. 42.
    Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: CVPR, pp. 2574–2582 (2016)Google Scholar
  43. 43.
    Nelson, B., et al.: Exploiting machine learning to subvert your spam filter. In: Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET 2008) (2008)Google Scholar
  44. 44.
    Nelson, B., et al.: Query strategies for evading convex-inducing classifiers. J. Mach. Learn. Res. 13(May), 1293–1332 (2012)MathSciNetzbMATHGoogle Scholar
  45. 45.
    Newsome, J., Karp, B., Song, D.: Paragraph: thwarting signature learning by training maliciously. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 81–105. Springer, Heidelberg (2006). Scholar
  46. 46.
    Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: CVPR, pp. 427–436 (2015)Google Scholar
  47. 47.
    Papernot, N., McDaniel, P., Goodfellow, I.: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. eprint arXiv:1605.07277 (2016)
  48. 48.
    Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against deep learning systems using adversarial examples. eprint arXiv:1602.02697 (2016)
  49. 49.
    Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: Proceedings of the European Symposium on Security & Privacy, pp. 372–387 (2016)Google Scholar
  50. 50.
    Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. eprint arXiv:1511.04508 (2015)
  51. 51.
    Pinto, L., Davidson, J., Sukthankar, R., Gupta, A.: Robust adversarial reinforcement learning. eprint arXiv:1703.02702 (2017)
  52. 52.
    Rubinstein, B.I., et al.: ANTIDOTE: understanding and defending against poisoning of anomaly detectors. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, pp. 1–14. ACM (2009)Google Scholar
  53. 53.
    Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized Experience Replay. CoRR abs/1511.05952 (2015)Google Scholar
  54. 54.
    Sengupta, S., Chakraborti, T., Kambhampati, S.: Securing deep neural nets against adversarial attacks with moving target defense. eprint arXiv:1705.07213, May 2017
  55. 55.
    Steinhardt, J., Koh, P.W., Liang, P.: Certified defenses for data poisoning attacks. eprint arXiv:1706.03691, June 2017
  56. 56.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)Google Scholar
  57. 57.
    Szegedy, C., et al.: Intriguing properties of neural networks. eprint arXiv:1312.6199 (2013)
  58. 58.
    Tramèr, F., Kurakin, A., Papernot, N., Boneh, D., McDaniel, P.: Ensemble adversarial training: attacks and defenses. eprint arXiv:1705.07204, May 2017
  59. 59.
    Wang, B., Gao, J., Qi, Y.: A theoretical framework for robustness of (deep) classifiers against adversarial examples. eprint arXiv:1612.00334 (2016)
  60. 60.
    Xiao, H., Xiao, H., Eckert, C.: Adversarial label flips attack on support vector machines. In: Proceedings of the 20th European Conference on Artificial Intelligence. ECAI 2012, pp. 870–875, IOS Press, Amsterdam (2012)Google Scholar
  61. 61.
    Zhang, F., Chan, P.P.K., Biggio, B., Yeung, D.S., Roli, F.: Adversarial feature selection against evasion attacks. IEEE Trans. Cybern. 46(3), 766–777 (2016)CrossRefGoogle Scholar
  62. 62.
    Zheng, S., Song, Y., Leung, T., Goodfellow, I.: Improving the robustness of deep neural networks via stability training. eprint arXiv:1604.04326 (2016)

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Computing and Information SystemsThe University of MelbourneParkvilleAustralia
  2. 2.Defence Science and Technology GroupEdinburghAustralia

Personalised recommendations