Skip to main content
SpringerLink
Log in

Reinforcement Learning for Autonomous Defence in Software-Defined Networking

  • Conference paper
  • First Online:
Decision and Game Theory for Security (GameSec 2018)

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 11199))

Included in the following conference series:

Abstract

Despite the successful application of machine learning (ML) in a wide range of domains, adaptability—the very property that makes machine learning desirable—can be exploited by adversaries to contaminate training and evade classification. In this paper, we investigate the feasibility of applying a specific class of machine learning algorithms, namely, reinforcement learning (RL) algorithms, for autonomous cyber defence in software-defined networking (SDN). In particular, we focus on how an RL agent reacts towards different forms of causative attacks that poison its training process, including indiscriminate and targeted, white-box and black-box attacks. In addition, we also study the impact of the attack timing, and explore potential countermeasures such as adversarial training.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Amazon EC2 Instance Types – Amazon Web Services (AWS). https://aws.amazon.com/ec2/instance-types/

  2. SDN architecture. Technical report, June 2014. https://www.opennetworking.org/wp-content/uploads/2013/02/TR_SDN_ARCH_1.0_06062014.pdf

  3. Mininet: An Instant Virtual Network on your Laptop (2017). http://mininet.org/

  4. OpenDaylight (2017). https://www.opendaylight.org/

  5. Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. arXiv:1802.00420 [cs], February 2018

  6. Barreno, M., Nelson, B., Joseph, A.D., Tygar, J.D.: The security of machine learning. Mach. Learn. 81(2), 121–148 (2010)

    Article  MathSciNet  Google Scholar 

  7. Beaudoin, L.: Autonomic computer network defence using risk states and reinforcement learning. Ph.D. thesis, University of Ottawa (Canada) (2009)

    Google Scholar 

  8. Behzadan, V., Munir, A.: Vulnerability of deep reinforcement learning to policy induction attacks. eprint arXiv:1701.04143 (2017)

    Chapter  Google Scholar 

  9. Bhagoji, A.N., Cullina, D., Mittal, P.: Dimensionality reduction as a defense against evasion attacks on machine learning classifiers. arXiv:1704.02654 (2017)

  10. Biggio, B., et al.: Security evaluation of support vector machines in adversarial environments. In: Ma, Y., Guo, G. (eds.) Support Vector Machines Applications, pp. 105–153. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-02300-7_4

    Chapter  Google Scholar 

  11. Biggio, B., Nelson, B., Laskov, P.: Poisoning attacks against support vector machines. In: Proceedings of the 29th International Conference on International Conference on Machine Learning, pp. 1467–1474. Omnipress, Edinburgh (2012)

    Google Scholar 

  12. Burkard, C., Lagesse, B.: Analysis of causative attacks against SVMs learning from data streams. In: Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics, pp. 31–36. ACM, New York (2017)

    Google Scholar 

  13. Carlini, N., Wagner, D.: Defensive distillation is not robust to adversarial examples. arXiv:1607.04311 (2016)

  14. Carlini, N., Wagner, D.: Towards evaluating the robustness of neural networks. eprint arXiv:1608.04644 (2016)

  15. Carlini, N., Wagner, D.: Adversarial examples are not easily detected: bypassing ten detection methods. eprint arXiv:1705.07263 (2017)

  16. Chung, S.P., Mok, A.K.: Advanced allergy attacks: does a corpus really help? In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 236–255. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74320-0_13

    Chapter  Google Scholar 

  17. Das, N., et al.: Keeping the bad guys out: protecting and vaccinating deep learning with JPEG compression. eprint arXiv:1705.02900, May 2017

  18. Diakonikolas, I., Kamath, G., Kane, D.M., Li, J., Moitra, A., Stewart, A.: Robust estimators in high dimensions without the computational intractability. In: Proceedings of the 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 655–664, October 2016

    Google Scholar 

  19. Everitt, T., Krakovna, V., Orseau, L., Hutter, M., Legg, S.: Reinforcement learning with a corrupted reward channel. eprint arXiv:1705.08417 (2017)

  20. Feinman, R., Curtin, R.R., Shintre, S., Gardner, A.B.: Detecting adversarial samples from artifacts. eprint arXiv:1703.00410 (2017)

  21. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. eprint arXiv:1412.6572 (2014)

  22. Han, Y., Rubinstein, B.I.P.: Adequacy of the gradient-descent method for classifier evasion attacks. arXiv:1704.01704, April 2017

  23. Hasselt, H.V.: Double Q-learning. In: Lafferty, J.D., Williams, C.K.I., Shawe-Taylor, J., Zemel, R.S., Culotta, A. (eds.) Advances in Neural Information Processing Systems 23, pp. 2613–2621. Curran Associates, Inc. (2010)

    Google Scholar 

  24. Hasselt, H.V., Guez, A., Silver, D.: Deep reinforcement learning with double Q-learning. eprint arXiv:1509.06461, September 2015

  25. He, W., Wei, J., Chen, X., Carlini, N., Song, D.: Adversarial example defenses: ensembles of weak defenses are not strong. eprint arXiv:1706.04701 (2017)

  26. Hosseini, H., Chen, Y., Kannan, S., Zhang, B., Poovendran, R.: Blocking transferability of adversarial examples in black-box learning systems. eprint arXiv:1703.04318 (2017)

  27. Huang, L., Joseph, A.D., Nelson, B., Rubinstein, B.I., Tygar, J.: Adversarial machine learning. In: Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence, pp. 43–58. ACM (2011)

    Google Scholar 

  28. Huang, S., Papernot, N., Goodfellow, I., Duan, Y., Abbeel, P.: Adversarial attacks on neural network policies. eprint arXiv:1702.02284 (2017)

  29. Koh, P.W., Liang, P.: understanding black-box predictions via influence functions. arXiv:1703.04730 [cs, stat], March 2017

  30. Laishram, R., Phoha, V.V.: Curie: a method for protecting SVM Classifier from poisoning attack. arXiv:1606.01584 [cs], June 2016

  31. Li, B., Vorobeychik, Y.: Feature cross-substitution in adversarial classification. In: Proceedings of the 2014 NIPS, NIPS 2014, pp. 2087–2095, MIT Press, Cambridge (2014)

    Google Scholar 

  32. Li, B., Wang, Y., Singh, A., Vorobeychik, Y.: Data poisoning attacks on factorization-based collaborative filtering. eprint arXiv:1608.08182 (2016)

  33. Li, X., Li, F.: Adversarial examples detection in deep networks with convolutional filter statistics. arXiv:1612.07767 [cs], December 2016

  34. Lin, Y.C., Hong, Z.W., Liao, Y.H., Shih, M.L., Liu, M.Y., Sun, M.: Tactics of adversarial attack on deep reinforcement learning agents. eprint arXiv:1703.06748, March 2017

  35. Medved, J., Varga, R., Tkacik, A., Gray, K.: OpenDaylight: towards a model-driven SDN controller architecture. In: Proceedings of IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks, pp. 1–6 (2014)

    Google Scholar 

  36. Mei, S., Zhu, X.: Using machine teaching to identify optimal training-set attacks on machine learners. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pp. 2871–2877. AAAI Press, Austin (2015)

    Google Scholar 

  37. Metzen, J.H., Genewein, T., Fischer, V., Bischoff, B.: On detecting adversarial perturbations. eprint arXiv:1702.04267 (2017)

  38. Mnih, V., et al.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning, ICML 2016, vol. 48, pp. 1928–1937. JMLR.org, New York (2016)

    Google Scholar 

  39. Mnih, V., et al.: Playing Atari with Deep Reinforcement Learning. CoRR abs/1312.5602 (2013)

    Google Scholar 

  40. Moore, D., Shannon, C., Brown, D.J., Voelker, G.M., Savage, S.: Inferring internet denial-of-service activity. ACM Trans. Comput. Syst. 24(2), 115–139 (2006)

    Article  Google Scholar 

  41. Moosavi-Dezfooli, S.M., Fawzi, A., Fawzi, O., Frossard, P.: Universal adversarial perturbations. eprint arXiv:1610.08401 (2016)

  42. Moosavi-Dezfooli, S.M., Fawzi, A., Frossard, P.: DeepFool: a simple and accurate method to fool deep neural networks. In: CVPR, pp. 2574–2582 (2016)

    Google Scholar 

  43. Nelson, B., et al.: Exploiting machine learning to subvert your spam filter. In: Proceedings of the First USENIX Workshop on Large-scale Exploits and Emergent Threats (LEET 2008) (2008)

    Google Scholar 

  44. Nelson, B., et al.: Query strategies for evading convex-inducing classifiers. J. Mach. Learn. Res. 13(May), 1293–1332 (2012)

    MathSciNet  MATH  Google Scholar 

  45. Newsome, J., Karp, B., Song, D.: Paragraph: thwarting signature learning by training maliciously. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 81–105. Springer, Heidelberg (2006). https://doi.org/10.1007/11856214_5

    Chapter  Google Scholar 

  46. Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: CVPR, pp. 427–436 (2015)

    Google Scholar 

  47. Papernot, N., McDaniel, P., Goodfellow, I.: Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. eprint arXiv:1605.07277 (2016)

  48. Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A.: Practical black-box attacks against deep learning systems using adversarial examples. eprint arXiv:1602.02697 (2016)

  49. Papernot, N., McDaniel, P., Jha, S., Fredrikson, M., Celik, Z.B., Swami, A.: The limitations of deep learning in adversarial settings. In: Proceedings of the European Symposium on Security & Privacy, pp. 372–387 (2016)

    Google Scholar 

  50. Papernot, N., McDaniel, P., Wu, X., Jha, S., Swami, A.: Distillation as a defense to adversarial perturbations against deep neural networks. eprint arXiv:1511.04508 (2015)

  51. Pinto, L., Davidson, J., Sukthankar, R., Gupta, A.: Robust adversarial reinforcement learning. eprint arXiv:1703.02702 (2017)

  52. Rubinstein, B.I., et al.: ANTIDOTE: understanding and defending against poisoning of anomaly detectors. In: Proceedings of the 9th ACM SIGCOMM Conference on Internet Measurement, pp. 1–14. ACM (2009)

    Google Scholar 

  53. Schaul, T., Quan, J., Antonoglou, I., Silver, D.: Prioritized Experience Replay. CoRR abs/1511.05952 (2015)

    Google Scholar 

  54. Sengupta, S., Chakraborti, T., Kambhampati, S.: Securing deep neural nets against adversarial attacks with moving target defense. eprint arXiv:1705.07213, May 2017

  55. Steinhardt, J., Koh, P.W., Liang, P.: Certified defenses for data poisoning attacks. eprint arXiv:1706.03691, June 2017

  56. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, 1st edn. MIT Press, Cambridge (1998)

    Google Scholar 

  57. Szegedy, C., et al.: Intriguing properties of neural networks. eprint arXiv:1312.6199 (2013)

  58. Tramèr, F., Kurakin, A., Papernot, N., Boneh, D., McDaniel, P.: Ensemble adversarial training: attacks and defenses. eprint arXiv:1705.07204, May 2017

  59. Wang, B., Gao, J., Qi, Y.: A theoretical framework for robustness of (deep) classifiers against adversarial examples. eprint arXiv:1612.00334 (2016)

  60. Xiao, H., Xiao, H., Eckert, C.: Adversarial label flips attack on support vector machines. In: Proceedings of the 20th European Conference on Artificial Intelligence. ECAI 2012, pp. 870–875, IOS Press, Amsterdam (2012)

    Google Scholar 

  61. Zhang, F., Chan, P.P.K., Biggio, B., Yeung, D.S., Roli, F.: Adversarial feature selection against evasion attacks. IEEE Trans. Cybern. 46(3), 766–777 (2016)

    Article  Google Scholar 

  62. Zheng, S., Song, Y., Leung, T., Goodfellow, I.: Improving the robustness of deep neural networks via stability training. eprint arXiv:1604.04326 (2016)

Download references

Author information

Authors and Affiliations

  1. School of Computing and Information Systems, The University of Melbourne, Parkville, Australia

    Yi Han, Benjamin I. P. Rubinstein, Tansu Alpcan, Sarah Erfani & Christopher Leckie

  2. Defence Science and Technology Group, Edinburgh, Australia

    Tamas Abraham, Olivier De Vel, David Hubczenko & Paul Montague

Authors
  1. Yi Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benjamin I. P. Rubinstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tamas Abraham
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tansu Alpcan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Olivier De Vel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sarah Erfani
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David Hubczenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Christopher Leckie
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Paul Montague
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yi Han .

Editor information

Editors and Affiliations

  1. University of Washington, Seattle, WA, USA

    Linda Bushnell

  2. University of Washington, Seattle, WA, USA

    Radha Poovendran

  3. University of Illinois at Urbana–Champaign, Urbana, IL, USA

    Tamer Başar

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Han, Y. et al. (2018). Reinforcement Learning for Autonomous Defence in Software-Defined Networking. In: Bushnell, L., Poovendran, R., Başar, T. (eds) Decision and Game Theory for Security. GameSec 2018. Lecture Notes in Computer Science(), vol 11199. Springer, Cham. https://doi.org/10.1007/978-3-030-01554-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-01554-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01553-4

  • Online ISBN: 978-3-030-01554-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation