Skip to main content
SpringerLink
Log in

Strategic Learning for Active, Adaptive, and Autonomous Cyber Defense

  • Chapter
  • First Online:
Adaptive Autonomous Secure Cyber Systems

Abstract

The increasing instances of advanced attacks call for a new defense paradigm that is active, autonomous, and adaptive, named as the ‘3A’ defense paradigm. This chapter introduces three defense schemes that actively interact with attackers to increase the attack cost and gather threat information, i.e., defensive deception for detection and counter-deception, feedback-driven Moving Target Defense (MTD), and adaptive honeypot engagement. Due to the cyber deception, external noise, and the absent knowledge of the other players’ behaviors and goals, these schemes possess three progressive levels of information restrictions, i.e., from the parameter uncertainty, the payoff uncertainty, to the environmental uncertainty. To estimate the unknown and reduce the uncertainty, we adopt three different strategic learning schemes that fit the associated information restrictions. All three learning schemes share the same feedback structure of sensation, estimation, and actions so that the most rewarding policies get reinforced and converge to the optimal ones in autonomous and adaptive fashions. This work aims to shed lights on proactive defense strategies, lay a solid foundation for strategic learning under incomplete information, and quantify the tradeoff between the security and costs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Verizon, "Data breach investigation report," (2019), Retrieved from https://enterprise.verizon.com/resources/reports/dbir/

  2. D. Shackleford, “Combatting cyber risks in the supply chain,” SANS. org, 2015.

    Google Scholar 

  3. L. Huang and Q. Zhu, “Adaptive strategic cyber defense for advanced persistent threats in critical infrastructure networks,” ACM SIGMETRICS Performance Evaluation Review, vol. 46, no. 2, pp. 52–56, 2019.

    Article  Google Scholar 

  4. ——, “Analysis and computation of adaptive defense strategies against advanced persistent threats for cyber-physical systems,” in International Conference on Decision and Game Theory for Security. Springer, 2018, pp. 205–226.

    Google Scholar 

  5. L. Huang, Q. Zhu, “A dynamic games approach to proactive defense strategies against Advanced Persistent Threats in cyber-physical systems,” Computers & Security, vol. 89, 101660, 2020. https://doi.org/10.1016/j.cose.2019.101660

    Article  Google Scholar 

  6. Q. Zhu and T. Başar, “Game-theoretic approach to feedback-driven multi-stage moving target defense,” in International Conference on Decision and Game Theory for Security. Springer, 2013, pp. 246–263.

    Google Scholar 

  7. L. Huang and Q. Zhu, “Adaptive Honeypot Engagement through Reinforcement Learning of Semi-Markov Decision Processes,” arXiv e-prints, p. arXiv:1906.12182, Jun 2019.

    Google Scholar 

  8. J. Pawlick, E. Colbert, and Q. Zhu, “A game-theoretic taxonomy and survey of defensive deception for cybersecurity and privacy,” arXiv preprint arXiv:1712.05441, 2017.

    Google Scholar 

  9. F. J. Stech, K. E. Heckman, and B. E. Strom, “Integrating cyber-d&d into adversary modeling for active cyber defense,” in Cyber deception. Springer, 2016, pp. 1–22.

    Google Scholar 

  10. K. E. Heckman, M. J. Walsh, F. J. Stech, T. A. O’boyle, S. R. DiCato, and A. F. Herber, “Active cyber defense with denial and deception: A cyber-wargame experiment,” computers & security, vol. 37, pp. 72–77, 2013.

    Article  Google Scholar 

  11. J. Gómez-Hernández, L. Álvarez-González, and P. García-Teodoro, “R-locker: Thwarting ransomware action through a honeyfile-based approach,” Computers & Security, vol. 73, pp. 389–398, 2018.

    Article  Google Scholar 

  12. N. Virvilis, B. Vanautgaerden, and O. S. Serrano, “Changing the game: The art of deceiving sophisticated attackers,” in 2014 6th International Conference On Cyber Conflict (CyCon 2014). IEEE, 2014, pp. 87–97.

    Google Scholar 

  13. J. Pawlick, E. Colbert, and Q. Zhu, “Modeling and analysis of leaky deception using signaling games with evidence,” IEEE Transactions on Information Forensics and Security, 2018.

    Google Scholar 

  14. S. Jajodia, A. K. Ghosh, V. Swarup, C. Wang, and X. S. Wang, Moving target defense: creating asymmetric uncertainty for cyber threats. Springer Science & Business Media, 2011, vol. 54.

    Google Scholar 

  15. G. S. Kc, A. D. Keromytis, and V. Prevelakis, “Countering code-injection attacks with instruction-set randomization,” in Proceedings of the 10th ACM conference on Computer and communications security. ACM, 2003, pp. 272–280.

    Google Scholar 

  16. A. Clark, Q. Zhu, R. Poovendran, and T. Başar, “Deceptive routing in relay networks,” in International Conference on Decision and Game Theory for Security. Springer, 2012, pp. 171–185.

    Google Scholar 

  17. H. Maleki, S. Valizadeh, W. Koch, A. Bestavros, and M. van Dijk, “Markov modeling of moving target defense games,” in Proceedings of the 2016 ACM Workshop on Moving Target Defense. ACM, 2016, pp. 81–92.

    Google Scholar 

  18. C. R. Hecker, “A methodology for intelligent honeypot deployment and active engagement of attackers,” Ph.D. dissertation, 2012.

    Google Scholar 

  19. Q. D. La, T. Q. Quek, J. Lee, S. Jin, and H. Zhu, “Deceptive attack and defense game in honeypot-enabled networks for the internet of things,” IEEE Internet of Things Journal, vol. 3, no. 6, pp. 1025–1035, 2016.

    Article  Google Scholar 

  20. J. Pawlick, T. T. H. Nguyen, and Q. Zhu, “Optimal timing in dynamic and robust attacker engagement during advanced persistent threats,” CoRR, vol. abs/1707.08031, 2017. [Online]. Available: http://arxiv.org/abs/1707.08031

  21. J. Pawlick and Q. Zhu, “A Stackelberg game perspective on the conflict between machine learning and data obfuscation,” in Information Forensics and Security (WIFS), 2016 IEEE International Workshop on. IEEE, 2016, pp. 1–6. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/7823893/

  22. Q. Zhu, A. Clark, R. Poovendran, and T. Basar, “Deployment and exploitation of deceptive honeybots in social networks,” in Decision and Control (CDC), 2013 IEEE 52nd Annual Conference on. IEEE, 2013, pp. 212–219.

    Google Scholar 

  23. Q. Zhu, H. Tembine, and T. Basar, “Hybrid learning in stochastic games and its applications in network security,” Reinforcement Learning and Approximate Dynamic Programming for Feedback Control, pp. 305–329, 2013.

    Google Scholar 

  24. Q. Zhu, Z. Yuan, J. B. Song, Z. Han, and T. Başar, “Interference aware routing game for cognitive radio multi-hop networks,” Selected Areas in Communications, IEEE Journal on, vol. 30, no. 10, pp. 2006–2015, 2012.

    Article  Google Scholar 

  25. Q. Zhu, L. Bushnell, and T. Basar, “Game-theoretic analysis of node capture and cloning attack with multiple attackers in wireless sensor networks,” in Decision and Control (CDC), 2012 IEEE 51st Annual Conference on. IEEE, 2012, pp. 3404–3411.

    Google Scholar 

  26. Q. Zhu, A. Clark, R. Poovendran, and T. Başar, “Deceptive routing games,” in Decision and Control (CDC), 2012 IEEE 51st Annual Conference on. IEEE, 2012, pp. 2704–2711.

    Google Scholar 

  27. Q. Zhu, H. Li, Z. Han, and T. Basar, “A stochastic game model for jamming in multi-channel cognitive radio systems.” in ICC, 2010, pp. 1–6.

    Google Scholar 

  28. Z. Xu and Q. Zhu, “Secure and practical output feedback control for cloud-enabled cyber-physical systems,” in Communications and Network Security (CNS), 2017 IEEE Conference on. IEEE, 2017, pp. 416–420.

    Google Scholar 

  29. ——, “A Game-Theoretic Approach to Secure Control of Communication-Based Train Control Systems Under Jamming Attacks,” in Proceedings of the 1st International Workshop on Safe Control of Connected and Autonomous Vehicles. ACM, 2017, pp. 27–34. [Online]. Available: http://dl.acm.org/citation.cfm?id=3055381

  30. ——, “Cross-layer secure cyber-physical control system design for networked 3d printers,” in American Control Conference (ACC), 2016. IEEE, 2016, pp. 1191–1196. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/7525079/

  31. M. J. Farooq and Q. Zhu, “Modeling, analysis, and mitigation of dynamic botnet formation in wireless iot networks,” IEEE Transactions on Information Forensics and Security, 2019.

    Google Scholar 

  32. Z. Xu and Q. Zhu, “A cyber-physical game framework for secure and resilient multi-agent autonomous systems,” in Decision and Control (CDC), 2015 IEEE 54th Annual Conference on. IEEE, 2015, pp. 5156–5161.

    Google Scholar 

  33. L. Huang, J. Chen, and Q. Zhu, “A large-scale markov game approach to dynamic protection of interdependent infrastructure networks,” in International Conference on Decision and Game Theory for Security. Springer, 2017, pp. 357–376.

    Google Scholar 

  34. J. Chen, C. Touati, and Q. Zhu, “A dynamic game analysis and design of infrastructure network protection and recovery,” ACM SIGMETRICS Performance Evaluation Review, vol. 45, no. 2, p. 128, 2017.

    Article  Google Scholar 

  35. F. Miao, Q. Zhu, M. Pajic, and G. J. Pappas, “A hybrid stochastic game for secure control of cyber-physical systems,” Automatica, vol. 93, pp. 55–63, 2018.

    Article  MathSciNet  Google Scholar 

  36. Y. Yuan, Q. Zhu, F. Sun, Q. Wang, and T. Basar, “Resilient control of cyber-physical systems against denial-of-service attacks,” in Resilient Control Systems (ISRCS), 2013 6th International Symposium on. IEEE, 2013, pp. 54–59.

    Google Scholar 

  37. S. Rass and Q. Zhu, “GADAPT: A Sequential Game-Theoretic Framework for Designing Defense-in-Depth Strategies Against Advanced Persistent Threats,” in Decision and Game Theory for Security, ser. Lecture Notes in Computer Science, Q. Zhu, T. Alpcan, E. Panaousis, M. Tambe, and W. Casey, Eds. Cham: Springer International Publishing, 2016, vol. 9996, pp. 314–326.

    MATH  Google Scholar 

  38. Q. Zhu, Z. Yuan, J. B. Song, Z. Han, and T. Basar, “Dynamic interference minimization routing game for on-demand cognitive pilot channel,” in Global Telecommunications Conference (GLOBECOM 2010), 2010 IEEE. IEEE, 2010, pp. 1–6.

    Google Scholar 

  39. T. Zhang and Q. Zhu, “Strategic defense against deceptive civilian gps spoofing of unmanned aerial vehicles,” in International Conference on Decision and Game Theory for Security. Springer, 2017, pp. 213–233.

    Google Scholar 

  40. L. Huang and Q. Zhu, “Analysis and computation of adaptive defense strategies against advanced persistent threats for cyber-physical systems,” in International Conference on Decision and Game Theory for Security, 2018.

    Google Scholar 

  41. ——, “Adaptive strategic cyber defense for advanced persistent threats in critical infrastructure networks,” in ACM SIGMETRICS Performance Evaluation Review, 2018.

    Google Scholar 

  42. J. Pawlick, S. Farhang, and Q. Zhu, “Flip the cloud: Cyber-physical signaling games in the presence of advanced persistent threats,” in Decision and Game Theory for Security. Springer, 2015, pp. 289–308.

    Google Scholar 

  43. S. Farhang, M. H. Manshaei, M. N. Esfahani, and Q. Zhu, “A dynamic bayesian security game framework for strategic defense mechanism design,” in Decision and Game Theory for Security. Springer, 2014, pp. 319–328.

    Google Scholar 

  44. Q. Zhu and T. Başar, “Dynamic policy-based ids configuration,” in Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on. IEEE, 2009, pp. 8600–8605.

    Google Scholar 

  45. Q. Zhu, H. Tembine, and T. Basar, “Network security configurations: A nonzero-sum stochastic game approach,” in American Control Conference (ACC), 2010. IEEE, 2010, pp. 1059–1064.

    Google Scholar 

  46. Q. Zhu, H. Tembine, and T. Başar, “Heterogeneous learning in zero-sum stochastic games with incomplete information,” in 49th IEEE conference on decision and control (CDC). IEEE, 2010, pp. 219–224.

    Google Scholar 

  47. J. Chen and Q. Zhu, “Security as a Service for Cloud-Enabled Internet of Controlled Things under Advanced Persistent Threats: A Contract Design Approach,” IEEE Transactions on Information Forensics and Security, 2017. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/7954676/

  48. R. Zhang, Q. Zhu, and Y. Hayel, “A Bi-Level Game Approach to Attack-Aware Cyber Insurance of Computer Networks,” IEEE Journal on Selected Areas in Communications, vol. 35, no. 3, pp. 779–794, 2017. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/7859343/

  49. R. Zhang and Q. Zhu, “Attack-aware cyber insurance of interdependent computer networks,” 2016.

    Book  Google Scholar 

  50. W. A. Casey, Q. Zhu, J. A. Morales, and B. Mishra, “Compliance control: Managed vulnerability surface in social-technological systems via signaling games,” in Proceedings of the 7th ACM CCS International Workshop on Managing Insider Security Threats. ACM, 2015, pp. 53–62.

    Google Scholar 

  51. Y. Hayel and Q. Zhu, “Attack-aware cyber insurance for risk sharing in computer networks,” in Decision and Game Theory for Security. Springer, 2015, pp. 22–34.

    Google Scholar 

  52. ——, “Epidemic protection over heterogeneous networks using evolutionary poisson games,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 8, pp. 1786–1800, 2017.

    Article  Google Scholar 

  53. Q. Zhu, C. Fung, R. Boutaba, and T. Başar, “Guidex: A game-theoretic incentive-based mechanism for intrusion detection networks,” Selected Areas in Communications, IEEE Journal on, vol. 30, no. 11, pp. 2220–2230, 2012.

    Article  Google Scholar 

  54. Q. Zhu, C. A. Gunter, and T. Basar, “Tragedy of anticommons in digital right management of medical records.” in HealthSec, 2012.

    Google Scholar 

  55. Q. Zhu, C. Fung, R. Boutaba, and T. Başar, “A game-theoretical approach to incentive design in collaborative intrusion detection networks,” in Game Theory for Networks, 2009. GameNets’ 09. International Conference on. IEEE, 2009, pp. 384–392.

    Google Scholar 

  56. T. E. Carroll and D. Grosu, “A game theoretic investigation of deception in network security,” Security and Commun. Nets., vol. 4, no. 10, pp. 1162–1172, 2011.

    Article  Google Scholar 

  57. J. Pawlick and Q. Zhu, “A Stackelberg game perspective on the conflict between machine learning and data obfuscation,” IEEE Intl. Workshop on Inform. Forensics and Security, 2016.

    Google Scholar 

  58. T. Zhang and Q. Zhu, “Dynamic differential privacy for ADMM-based distributed classification learning,” IEEE Transactions on Information Forensics and Security, vol. 12, no. 1, pp. 172–187, 2017. [Online]. Available: http://ieeexplore.ieee.org/abstract/document/7563366/

  59. S. Farhang, Y. Hayel, and Q. Zhu, “Phy-layer location privacy-preserving access point selection mechanism in next-generation wireless networks,” in Communications and Network Security (CNS), 2015 IEEE Conference on. IEEE, 2015, pp. 263–271.

    Google Scholar 

  60. T. Zhang and Q. Zhu, “Distributed privacy-preserving collaborative intrusion detection systems for vanets,” IEEE Transactions on Signal and Information Processing over Networks, vol. 4, no. 1, pp. 148–161, 2018.

    Article  MathSciNet  Google Scholar 

  61. N. Zhang, W. Yu, X. Fu, and S. K. Das, “gPath: A game-theoretic path selection algorithm to protect tor’s anonymity,” in Decision and Game Theory for Security. Springer, 2010, pp. 58–71.

    Google Scholar 

  62. A. Garnaev, M. Baykal-Gursoy, and H. V. Poor, “Security games with unknown adversarial strategies,” IEEE transactions on cybernetics, vol. 46, no. 10, pp. 2291–2299, 2015.

    Article  Google Scholar 

  63. Q. Zhu, H. Tembine, and T. Başar, “Distributed strategic learning with application to network security,” in Proceedings of the 2011 American Control Conference. IEEE, 2011, pp. 4057–4062.

    Google Scholar 

  64. A. Servin and D. Kudenko, “Multi-agent reinforcement learning for intrusion detection: A case study and evaluation,” in German Conference on Multiagent System Technologies. Springer, 2008, pp. 159–170.

    Google Scholar 

  65. P. M. Djurić and Y. Wang, “Distributed bayesian learning in multiagent systems: Improving our understanding of its capabilities and limitations,” IEEE Signal Processing Magazine, vol. 29, no. 2, pp. 65–76, 2012.

    Article  Google Scholar 

  66. G. Chalkiadakis and C. Boutilier, “Coordination in multiagent reinforcement learning: a bayesian approach,” in Proceedings of the second international joint conference on Autonomous agents and multiagent systems. ACM, 2003, pp. 709–716.

    Google Scholar 

  67. Z. Chen and D. Marculescu, “Distributed reinforcement learning for power limited many-core system performance optimization,” in Proceedings of the 2015 Design, Automation & Test in Europe Conference & Exhibition. EDA Consortium, 2015, pp. 1521–1526.

    Google Scholar 

  68. J. C. Harsanyi, “Games with incomplete information played by “bayesian” players, i–iii part i. the basic model,” Management science, vol. 14, no. 3, pp. 159–182, 1967.

    Article  MathSciNet  Google Scholar 

  69. M. E. Taylor and P. Stone, “Transfer learning for reinforcement learning domains: A survey,” Journal of Machine Learning Research, vol. 10, no. Jul, pp. 1633–1685, 2009.

    Google Scholar 

Download references

Acknowledgements

This research is partially supported by awards ECCS-1847056, CNS-1720230, CNS-1544782, and SES-1541164 from National Science of Foundation (NSF), award 2015-ST-061-CIRC01 from U. S. Department of Homeland Security, and grant W911NF-19-1-0041 from Army Research Office (ARO).

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, New York University, Brooklyn, NY, USA

    Linan Huang & Quanyan Zhu

Authors
  1. Linan Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quanyan Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quanyan Zhu .

Editor information

Editors and Affiliations

  1. Center for Secure Information Systems, George Mason University, Fairfax, VA, USA

    Sushil Jajodia

  2. Thayer School of Engineering, Dartmouth College, Hanover, NH, USA

    George Cybenko

  3. Department of Computer Science, Dartmouth College, Hanover, NH, USA

    V.S. Subrahmanian

  4. MS T310, MITRE Corporation, McLean, VA, USA

    Vipin Swarup

  5. Computing and Information Science Division, Army Research Office, Durham, NC, USA

    Cliff Wang

  6. Computer Science & Engineering, University of Michigan, Ann Arbor, MI, USA

    Michael Wellman

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Huang, L., Zhu, Q. (2020). Strategic Learning for Active, Adaptive, and Autonomous Cyber Defense. In: Jajodia, S., Cybenko, G., Subrahmanian, V., Swarup, V., Wang, C., Wellman, M. (eds) Adaptive Autonomous Secure Cyber Systems. Springer, Cham. https://doi.org/10.1007/978-3-030-33432-1_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33432-1_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33431-4

  • Online ISBN: 978-3-030-33432-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation