Skip to main content

A Game Theoretical Framework for Inter-process Adversarial Intervention Detection

Part of the Lecture Notes in Computer Science book series (LNSC,volume 11199)


In this paper, we propose and analyze a two-level game theoretical framework to detect advanced and persistent threats across processes. The two-level framework adapted facilitates abstraction of the complexity of process level interactions between defense mechanisms and adversaries from easier to interpret and more flexible system-level interaction. At the process-level, program anomaly detection algorithms have already been proposed to detect anomalous program behavior by comparing monitored activities with the predetermined expected behavior. This had led to significant detection performance initially until advanced adversaries modified the attacks to remain undetected. Therefore, we propose defense mechanisms that anticipate the reaction of advanced evaders and seek to maximize the complexity of undetectable attacks at the expense of additional false alarm rate. Furthermore, in the system-level, we propose defense mechanisms to detect adversarial intervention across processes through the assessment of all process activities together in a cohesive way so that the advanced adversaries need to craft their attacks further to remain undetected also at the system-level. This further increases the cost of complexity for the attacker, and correspondingly degrades the motivation to attack. We provide a game theoretical incentive analysis for both defenders and adversaries, and characterize pure and mixed strategy equilibria. We also analyze the coupling between the two levels of the game.


  • Host-based intrusion detection
  • Mimicry attacks
  • Games-in-games
  • Anomaly detection
  • Process monitoring
  • Stackelberg games
  • Advanced persistent threats

M. O. Sayin—This research was supported by the U.S. Office of Naval Research (ONR) MURI grant N00014-16-1-2710.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


  1. 1.

    We have used infimum here since, the set of indefinite length strings, \(\varSigma ^*\), not being compact, there is no immediate guarantee of existence of a minimum.

  2. 2.

    With abuse of notation, we denote \(r^*_i\) as a function of \(\pmb {a}^*_i\) in order to show the dependence explicitly.

  3. 3.

    For notational simplicity, we do not represent \(\alpha _{\omega }\)’s dependence on subset k explicitly.

  4. 4.

    For notational simplicity, we have dropped the subset index k.


  1. Başar, T., Olsder, G.: Dynamic Noncooperative Game Theory. Society for Industrial Mathematics (SIAM) Series in Classics, Applied Mathematics (1999)

    Google Scholar 

  2. Barabosch, T., Gerhards-Padilla, E.: Host-based code injection attacks: a popular technique used by malware. In: The 9th International Conference on Malicious and Unwanted Software: The Americas (MALWARE) (2014)

    Google Scholar 

  3. Brangetto, P., Aubyn, M.K.-S.: Economic aspects of national cyber security strategies. Technical report, NATO Cooperative Cyber Defense Centre of Excellence Tallinn, Estonia (2015)

    Google Scholar 

  4. Dritsoula, L., Loiseau, P., Musacchio, J.: A game-theoretic analysis of adversarial classification. IEEE Trans. Inf. Forensics Secur. 12(12), 3094–3109 (2017)

    CrossRef  Google Scholar 

  5. Du, M., Li, F., Zheng, G., Srikumar, V.: DeepLog: anomaly detection and diagnosis from system logs through deep learning. In: Proceedings of the 24th ACM Conference on Computer and Communications Security (2017)

    Google Scholar 

  6. Elisan, C.C.: Malware, Rootkits and Botnets: A Beginner’s Guide. McGraw-Hill, New York (2013)

    Google Scholar 

  7. Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of the IEEE Security and Privacy (2003)

    Google Scholar 

  8. Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for Unix processes. In: Proceedings of IEEE Symposium on Security and Privacy (1996)

    Google Scholar 

  9. Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: Proceedings of the 13th Conference on USENIX Security Symposium (2004)

    Google Scholar 

  10. Ghafouri, A., Abbas, W., Laszka, A., Vorobeychik, Y., Koutsoukos, X.: Optimal thresholds for anomaly-based intrusion detection in dynamical environments. In: Zhu, Q., Alpcan, T., Panaousis, E., Tambe, M., Casey, W. (eds.) GameSec 2016. LNCS, vol. 9996, pp. 415–434. Springer, Cham (2016).

    CrossRef  MATH  Google Scholar 

  11. Ji, Y., et al.: RAIN: refinable attack investigation with on-demand inter-process information flow tracking. In: Proceedings of the 24th ACM Conference on Computer and Communications Security (2017)

    Google Scholar 

  12. Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of the 14th USENIX Security Symposium (2005)

    Google Scholar 

  13. Manzoor, E., Milajerdi, S.M., Akoglu, L.: Fast memory-efficient anomaly detection in streaming heterogeneous graphs. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016)

    Google Scholar 

  14. Parampalli, C., Sekar, R., Johanson, R.: A practical mimicry attack against powerful system-call monitors. In: Proceedings of the 2008 ACM Symposium on Information, Computer and Communications Security (2008)

    Google Scholar 

  15. Shu, X., Yao, D.D., Ryder, B.G.: A formal framework for program anomaly detection. In: Bos, H., Monrose, F., Blanc, G. (eds.) RAID 2015. LNCS, vol. 9404, pp. 270–292. Springer, Cham (2015).

    CrossRef  Google Scholar 

  16. Silberschatz, A., Galvin, P.B., Gagne, G.: Operating System Concepts. Wiley, Hoboken (2013)

    MATH  Google Scholar 

  17. Somayaji, A., Forest, S.: Automated response using system-call delays. In: Proceedings of the 9th USENIX Security Symposium (2000)

    Google Scholar 

  18. Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the Symposium on Security and Privacy (2001)

    Google Scholar 

  19. Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security (2002)

    Google Scholar 

  20. Yao, D., Shu, X., Cheng, L., Stolfo, S.J.: Anomaly Detection as a Service: Challenges, Advances, and Opportunities. Synthesis Lectures on Information Security, Privacy, and Thrust #22, Morgan & Claypool Publishers (2017)

    Google Scholar 

  21. Zhu, Q., Başar, T.: Game-theoretic methods for robustness, security, and resiliency of cyberphysical control systems: games-in-games principle for cross-layer resilient control systems. IEEE Control Syst. Mag. 35(1), 46–65 (2015)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Muhammed O. Sayin .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sayin, M.O., Hosseini, H., Poovendran, R., Başar, T. (2018). A Game Theoretical Framework for Inter-process Adversarial Intervention Detection. In: Bushnell, L., Poovendran, R., Başar, T. (eds) Decision and Game Theory for Security. GameSec 2018. Lecture Notes in Computer Science(), vol 11199. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-01553-4

  • Online ISBN: 978-3-030-01554-1

  • eBook Packages: Computer ScienceComputer Science (R0)