Skip to main content
SpringerLink
Log in

Adaptive Cyber Defenses for Botnet Detection and Mitigation

  • Chapter
  • First Online:
Adversarial and Uncertain Reasoning for Adaptive Cyber Defense

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 11830))

Abstract

Organizations increasingly rely on complex networked systems to maintain operational efficiency. While the widespread adoption of network-based IT solutions brings significant benefits to both commercial and government organizations, it also exposes them to an array of novel threats. Specifically, malicious actors can use networks of compromised and remotely controlled hosts, known as botnets, to execute a number of different cyber-attacks and engage in criminal or otherwise unauthorized activities. Most notably, botnets can be used to exfiltrate highly sensitive data from target networks, including military intelligence from government agencies and proprietary data from enterprise networks. What makes the problem even more complex is the recent trend towards stealthier and more resilient botnet architectures, which depart from traditional centralized architectures and enable botnets to evade detection and persist in a system for extended periods of time. A promising approach to botnet detection and mitigation relies on Adaptive Cyber Defense (ACD), a novel and game-changing approach to cyber defense. We show that detecting and mitigating stealthy botnets is a multi-faceted problem that requires addressing multiple related research challenges, and show how an ACD approach can help us address these challenges effectively.

The work presented in this chapter was support by the Army Research Office under grant W911NF-13-1-0421.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For the sake of presentation, we assume that those shown are the only possible temporal k-placements in ).

  2. 2.

    A free Java graph library available at http://jgrapht.sourceforge.net.

  3. 3.

    Attempted access to a honeypot can be assumed an indicator of malicious activity.

  4. 4.

    Both \(\varDelta t_{mon}\) and \(\varDelta t_{clean}\) are defined as a fraction of an epoch.

References

  1. APT1: Exposing one of China’s cyber espionage units. Technical report, Mandiant, February 2013

    Google Scholar 

  2. Lateral movement: how do threat actors move deeper into your network? Technical report, Trend Micro (2013)

    Google Scholar 

  3. Alpcan, T., Başar, T.: An intrusion detection game with limited observations. In: Proceedings of the 12th International Symposium on Dynamic Games and Applications (ISDG 2006), Sophia-Antipolis, France, July 2006

    Google Scholar 

  4. Ankerst, M., Breunig, M.M., Kriegel, H.P., Sander, J.: OPTICS: ordering points to identify the clustering structure. In: Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data (SIGMOD 1999), pp. 49–60. ACM, Philadelphia, May 1999

    Google Scholar 

  5. Beigi, E.B., Jazi, H.H., Stakhanova, N., Ghorbani, A.A.: Towards effective feature selection in machine learning-based botnet detection approaches. In: Proceedings of the IEEE Conference on Communications and Network Security (IEEE CNS 2014), pp. 247–255. IEEE, San Francisco, October 2014

    Google Scholar 

  6. Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)

    MATH  Google Scholar 

  7. Chadha, R., et al.: CyberVAN: a cyber security virtual assured network testbed. In: Proceedings of the 2016 IEEE Military Communications Conference (MILCOM 2016), pp. 1125–1130. IEEE, Baltimore, November 2016

    Google Scholar 

  8. Collins, M.P., Shimeall, T.J., Faber, S., Janies, J., Weaver, R., Shon, M.D., Kadane, J.B.: Using uncleanliness to predict future botnet addresses. In: Proceedings of the 7th ACM SIGCOMM Internet Measurement Conference (IMC 2007), pp. 93–104. ACM, San Diego, October 2007

    Google Scholar 

  9. Faloutsos, M., Faloutsos, P., Faloutsos, C.: On power-law relationships of the Internet topology. ACM SIGCOMM Comput. Commun. Rev. 29(4), 251–262 (1999)

    Article  MATH  Google Scholar 

  10. Fredman, M.L., Tarjan, R.E.: Fibonacci heaps and their uses in improved network optimization algorithms. J. ACM (JACM) 34(3), 596–615 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  11. Frescura, F.A.M., Engelbrecht, C.A., Frank, B.S.: Significance tests for periodogram peaks, June 2007. https://arxiv.org/abs/0706.2225

  12. Ganesan, R., Jajodia, S., Shah, A., Cam, H.: Dynamic scheduling of cybersecurity analysts for minimizing risk using reinforcement learning. ACM Trans. Intell. Syst. Technol. 8(1) (2016)

    Article  Google Scholar 

  13. García, S., Grill, M., Stiborek, J., Zunino, A.: An empirical comparison of botnet detection methods. Comput. Secur. 45, 100–123 (2014)

    Article  Google Scholar 

  14. Gosavi, A.: Simulation-Based Optimization: Parametric Optimization Techniques and Reinforcement Learning, Operations Research/Computer Science Interfaces, vol. 55, 2nd edn. Springer, New York (2003)

    Book  MATH  Google Scholar 

  15. Gu, G., Perdisci, R., Zhang, J., Lee, W.: BotMiner: clustering analysis of network traffic for protocol- and structure-independent botnet detection. In: Proceedings of the 17th USENIX Security Symposium (USENIX Security 2008), pp. 139–154. USENIX Association, San Jose, July 2008

    Google Scholar 

  16. Gu, G., Porras, P., Yegneswaran, V., Fong, M., Lee, W.: BotHunter: detecting malware infection through IDS-driven dialog correlation. In: Proceedings of the 16th USENIX Security Symposium (USENIX Security 2007), pp. 167–182. USENIX Association, August 2007

    Google Scholar 

  17. Jain, M., Korzhyk, D., Vaněk, O., Conitzer, V., Pěchouček, M., Tambe, M.: A double oracle algorithm for zero-sum security games on graphs. In: Proceedings of the 10th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS 2011), pp. 327–334. IFAAMAS, Taipei, May 2011

    Google Scholar 

  18. Jajodia, S., Ghosh, A.K., Swarup, V., Wang, C., Wang, X.S. (eds.): Moving Target Defense: Creating Asymmetric Uncertainty for Cyber Threats, Advances in Information Security, vol. 54. Springer, New York (2011). https://doi.org/10.1007/978-1-4614-0977-9

    Book  Google Scholar 

  19. Kaspersky Labs: Kaspersky lab and ITU research reveals new advanced cyber threat, May 2012. http://usa.kaspersky.com/about-us/press-center/press-releases/kaspersky-lab-and-itu-research-reveals-new-advanced-cyber-threat

  20. Khalil, K., Qian, Z., Yu, P., Krishnamurthy, S., Swam, A.: Optimal monitor placement for detection of persistent threats. In: Proceedings of the IEEE Global Communications Conference (IEEE GLOBECOM 2016). IEEE, Washington, DC, December 2016

    Google Scholar 

  21. Kiekintveld, C., Jain, M., Tsai, J., Pita, J., Ordóñez, F., Tambe, M.: Computing optimal randomized resource allocations for massive security games. In: Proceedings of the 8th International Conference on Autonomous Agents and Multi-Agent Systems, pp. 689–696. IFAAMAS, Budapest, May 2009

    Google Scholar 

  22. Korzhyk, D., Yin, Z., Kiekintveld, C., Conitzer, V., Tambe, M.: Stackelberg vs. Nash in security games: an extended investigation of interchangeability, equivalence, and uniqueness. J. Artif. Intell. Res. 41(2), 297–327 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  23. Manadhata, P.K., Wing, J.M.: An attack surface metric. IEEE Trans. Softw. Eng. 37(3), 371–386 (2011)

    Article  Google Scholar 

  24. Marschalek, M., Kimayong, P., Gong, F.: POS malware revisited - look what we found inside your cashdesk. Cyphort labs special report, Cyphort, Inc. (2014)

    Google Scholar 

  25. McMahan, H.B., Gordon, G.J., Blum, A.: Planning in the presence of cost functions controlled by an adversary. In: Proceedings of the 20th International Conference on Machine Learning (ICML 2003), pp. 536–543. AAAI Press, Washington DC, August 2003

    Google Scholar 

  26. Medina, A., Lakhina, A., Matta, I., Byers, J.: BRITE: an approach to universal topology generation. In: Proceedings of the 9th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, pp. 346–353. IEEE, Cincinnati, August 2001

    Google Scholar 

  27. Merritt, E.: New POS malware emerges - Punkey, April 2015. https://www.trustwave.com/Resources/SpiderLabs-Blog/New-POS-Malware-Emerges--Punkey/

  28. Moreira Moura, G.C.: Internet Bad Neighborhoods. Ph.D. thesis, University of Twente, The Netherlands, March 2013

    Google Scholar 

  29. Nascimento, J.M., Powell, W.B.: An optimal approximate dynamic programming algorithm for the lagged asset acquisition problem. Math. Oper. Res. 34(1), 210–237 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  30. Nguyen, T.H., Wellman, M.P., Singh, S.: A stackelberg game model for botnet data exfiltration. In: Rass, S., An, B., Kiekintveld, C., Fang, F., Schauer, S. (eds.) GameSec 2017. LNCS, vol. 10575, pp. 151–170. Springer, Vienna (2017). https://doi.org/10.1007/978-3-319-68711-7_9

    Chapter  MATH  Google Scholar 

  31. Powell, W.B.: Approximate Dynamic Programming: Solving the Curses of Dimensionality, 2nd edn. Wiley, Hoboken (2011)

    Book  MATH  Google Scholar 

  32. Rossow, C., Andriesse, D., Werner, T., Stone-Gross, B., Plohmann, D., Dietrich, C.J., Bos, H.: SoK: P2PWNED - modeling and evaluating the resilience of peer-to-peer botnets. In: Proceedings of the 2013 IEEE Symposium on Security and Privacy (S&P 2013), pp. 97–111. IEEE, Berkeley (2013)

    Google Scholar 

  33. Scargle, J.D.: Studies in astronomical time series analysis. ii-statistical aspects of spectral analysis of unevenly spaced data. Astrophys. J. 263, 835–853 (1982)

    Article  Google Scholar 

  34. Schmidt, S., Alpcan, T., Albayrak, Ş., Başar, T., Mueller, A.: A malware detector placement game for intrusion detection. In: Lopez, J., Hämmerli, B.M. (eds.) CRITIS 2007. LNCS, vol. 5141, pp. 311–326. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-89173-4_26

    Chapter  Google Scholar 

  35. Shinoda, Y., Ikai, K., Itoh, M.: Vulnerabilities of passive internet threat monitors. In: Proceedings of the 14th USENIX Security Symposium (USENIX Security 2005), pp. 209–224. USENIX Association, Baltimore, August 2005

    Google Scholar 

  36. Shmatikov, V., Wang, M.H.: Security against probe-response attacks in collaborative intrusion detection. In: Proceedings of the 2007 Workshop on Large Scale Attack Defense, pp. 129–136. ACM, Kyoto, August 2007

    Google Scholar 

  37. Spring, N., Mahajan, R., Wetherall, D., Anderson, T.: Measuring ISP topologies with Rocketfuel. IEEE/ACM Trans. Netw. 12(1), 2–16 (2004)

    Article  Google Scholar 

  38. Stinson, E., Mitchell, J.C.: Towards systematic evaluation of the evadability of bot/botnet detection methods. In: Proceedings of the 2nd USENIX Workshop on Offensive Technologies. USENIX Association, San Jose, July 2008

    Google Scholar 

  39. Stoica, P., Moses, R.L.: Introduction to Spectral Analysis, 1st edn. Prentice Hall, Upper Saddle River (1997)

    MATH  Google Scholar 

  40. Sweeney, P.J.: Designing effective and stealthy botnets for cyber espionage and interdiction: finding the cyber high ground. Ph.D. thesis, Thayer School of Engineering, Darthmouth College, August 2014

    Google Scholar 

  41. Venkatesan, S., Albanese, M., Cybenko, G., Jajodia, S.: A moving target defense approach to disrupting stealthy botnets. In: Proceedings of the 3rd ACM Workshop on Moving Target Defense (MTD 2016), pp. 37–46. ACM, Vienna, October 2016

    Google Scholar 

  42. Venkatesan, S., Albanese, M., Jajodia, S.: Disrupting stealthy botnets through strategic placement of detectors. In: Proceedings of the 3rd IEEE Conference on Communications and Network Security (IEEE CNS 2015), pp. 55–63. IEEE, Florence, September 2015. Best Paper Runner-up Award

    Google Scholar 

  43. Venkatesan, S., Albanese, M., Shah, A., Ganesan, R., Jajodia, S.: Detecting stealthy botnets in a resource-constrained environment using reinforcement learning. In: Proceedings of the 4th ACM Workshop on Moving Target Defense (MTD 2017), pp. 75–85. ACM, Dallas, October 2017

    Google Scholar 

  44. Vlachos, M., Yu, P., Castelli, V.: On periodicity detection and structural periodic similarity. In: Proceedings of the 5th SIAM International Conference on Data Mining (SDM 2005), pp. 449–460. SIAM, Newport Beach, April 2005

    Google Scholar 

  45. Wang, Y., Wen, S., Xiang, Y., Zhou, W.: Modeling the propagation of worms in networks: a survey. IEEE Commun. Surv. Tutorials 16(2), 942–960 (2014)

    Article  Google Scholar 

  46. Wellman, M.P., Prakash, A.: Empirical game-theoretic analysis of an adaptive cyber-defense scenario (preliminary report). In: Poovendran, R., Saad, W. (eds.) GameSec 2014. LNCS, vol. 8840, pp. 43–58. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-12601-2_3

    Chapter  MATH  Google Scholar 

  47. West, M.: Preventing system intrusions. In: Network and System Security, pp. 29–56, , 2nd edn. Syngress (2014)

    Google Scholar 

  48. Zeng, Y., Hu, X., Shin, K.G.: Detection of botnets using combined host- and network-level information. In: Proceedings of the IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2010), pp. 291–300. IEEE, Chicago, June 2010

    Google Scholar 

  49. Zhang, J., Perdisci, R., Lee, W., Luo, X., Sarfraz, U.: Building a scalable system for stealthy P2P-botnet detection. IEEE Trans. Inf. Forensics Secur. 9(1), 27–38 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Massimiliano Albanese & Sushil Jajodia

  2. Perspecta Labs, Basking Ridge, NJ, USA

    Sridhar Venkatesan

  3. Dartmouth College, Hanover, NH, USA

    George Cybenko

  4. University of Oregon, Eugene, OR, USA

    Thanh Nguyen

Authors
  1. Massimiliano Albanese
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sushil Jajodia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sridhar Venkatesan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. George Cybenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thanh Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Massimiliano Albanese .

Editor information

Editors and Affiliations

  1. George Mason University, Fairfax, VA, USA

    Sushil Jajodia

  2. Dartmouth College, Hanover, NH, USA

    George Cybenko

  3. Pennsylvania State University, University Park, PA, USA

    Peng Liu

  4. Army Research Laboratory, Triangle Park, NC, USA

    Cliff Wang

  5. University of Michigan, Ann Arbor, MI, USA

    Michael Wellman

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Albanese, M., Jajodia, S., Venkatesan, S., Cybenko, G., Nguyen, T. (2019). Adaptive Cyber Defenses for Botnet Detection and Mitigation. In: Jajodia, S., Cybenko, G., Liu, P., Wang, C., Wellman, M. (eds) Adversarial and Uncertain Reasoning for Adaptive Cyber Defense. Lecture Notes in Computer Science(), vol 11830. Springer, Cham. https://doi.org/10.1007/978-3-030-30719-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30719-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30718-9

  • Online ISBN: 978-3-030-30719-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation