Skip to main content
SpringerLink
Log in
Book cover

International Symposium on Recent Advances in Intrusion Detection

RAID 2015: Research in Attacks, Intrusions, and Defenses pp 270–292Cite as

A Formal Framework for Program Anomaly Detection

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Computer Science ((LNSC,volume 9404))

Abstract

Program anomaly detection analyzes normal program behaviors and discovers aberrant executions caused by attacks, misconfigurations, program bugs, and unusual usage patterns. The merit of program anomaly detection is its independence from attack signatures, which enables proactive defense against new and unknown attacks. In this paper, we formalize the general program anomaly detection problem and point out two of its key properties. We present a unified framework to present any program anomaly detection method in terms of its detection capability. We prove the theoretical accuracy limit for program anomaly detection with an abstract detection machine. We show how existing solutions are positioned in our framework and illustrate the gap between state-of-the-art methods and the theoretical accuracy limit. We also point out some potential modeling features for future program anomaly detection evolution.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Instruction addresses are unique identifiers of specific instructions.

  2. 2.

    Context-sensitive languages correspond to pushdown automata.

  3. 3.

    The hierarchy is reasoned via Chomsky hierarchy [12], which presents the hierarchical relation among formal grammars/languages.

  4. 4.

    For example, one detection approach \(\varLambda _a\) in L-2 without argument analysis could be less capable of detecting attacks than an approach \(\varLambda _b\) in L-3 with argument analysis.

  5. 5.

    n can be either a fixed value or a variable [45, 63].

  6. 6.

    Lookahead pair methods are subsequent variants of n-gram methods [35].

  7. 7.

    Probabilistic PDA has not been explored by the anomaly detection community.

  8. 8.

    Calling context sensitivity (or context sensitivity in short) in program analysis should be distinguished from the term context-sensitive in formal languages. The latter characterizes cross-serial dependencies in a trace, while the former identifies each event (e.g., a system call) in a trace more precisely.

  9. 9.

    Dynamically assigned transitions cannot be precisely pinpointed from static analysis.

References

  1. Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity. In: Proceedings of ACM CCS, pp. 340–353 (2005)

    Google Scholar 

  2. Anderson, J.P.: Computer security technology planning study. Technicl report, DTIC (October (1972)

    Google Scholar 

  3. Bach, M., Charney, M., Cohn, R., Demikhovsky, E., Devor, T., Hazelwood, K., Jaleel, A., Luk, C.K., Lyons, G., Patil, H., Tal, A.: Analyzing parallel programs with pin. Computer 43(3), 34–41 (2010)

    Article  Google Scholar 

  4. Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: Proceedings of IEEE S & P, May 2006

    Google Scholar 

  5. Bletsch, T., Jiang, X., Freeh, V.W., Liang, Z.: Jump-oriented programming: a new class of code-reuse attack. In: Proceedings of ASIACCS, pp. 30–40 (2011)

    Google Scholar 

  6. Bresnan, J., Bresnan, R.M., Peters, S., Zaenen, A.: Cross-serial dependencies in Dutch. In: Savitch, W.J., Bach, E., Marsh, W., Safran-Naveh, G. (eds.) The Formal Complexity of Natural Language, vol. 33, pp. 286–319. Springer, Heidelberg (1987)

    Chapter  Google Scholar 

  7. Canali, D., Lanzi, A., Balzarotti, D., Kruegel, C., Christodorescu, M., Kirda, E.: A quantitative study of accuracy in system call-based malware detection. In: Proceedings of ISSTA, pp. 122–132 (2012)

    Google Scholar 

  8. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection for discrete sequences: a survey. IEEE TKDE 24(5), 823–839 (2012)

    Google Scholar 

  9. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 1–58 (2009)

    Article  Google Scholar 

  10. Checkoway, S., Davi, L., Dmitrienko, A., Sadeghi, A.R., Shacham, H., Winandy, M.: Return-oriented programming without returns. In: Proceedings of ACM CCS, pp. 559–572 (2010)

    Google Scholar 

  11. Chen, S., Xu, J., Sezer, E.C., Gauriar, P., Iyer, R.K.: Non-control-data attacks are realistic threats. In: Proceedings of USENIX Security, vol. 14, pp. 12–12 (2005)

    Google Scholar 

  12. Chomsky, N.: Three models for the description of language. IRE Trans. Inf. Theory 2(3), 113–124 (1956)

    Article  MATH  Google Scholar 

  13. Cowan, C., Pu, C., Maier, D., Hintony, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: StackGuard: automatic adaptive detection and prevention of buffer-overflow attacks. In: Proceedings of USENIX Security, vol. 7, p. 5 (1998)

    Google Scholar 

  14. Cox, B., Evans, D., Filipi, A., Rowanhill, J., Hu, W., Davidson, J., Knight, J., Nguyen-Tuong, A., Hiser, J.: N-variant systems: a secretless framework for security through diversity. In: Proceedings of USENIX Security, vol. 15 (2006)

    Google Scholar 

  15. Denning, D.E.: An intrusion-detection model. IEEE TSE 13(2), 222–232 (1987)

    Google Scholar 

  16. Endler, D.: Intrusion detection: applying machine learning to solaris audit data. In: Proceedings of ACSAC, pp. 268–279, December 1998

    Google Scholar 

  17. Eskin, E., Lee, W., Stolfo, S.: Modeling system calls for intrusion detection with dynamic window sizes. In: Proceedings of DARPA Information Survivability Conference and Exposition II, vol.1, pp. 165–175 (2001)

    Google Scholar 

  18. Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of IEEE Security and Privacy (2003)

    Google Scholar 

  19. Feng, H., Giffin, J., Huang, Y., Jha, S., Lee, W., Miller, B.: Formalizing sensitivity in static analysis for intrusion detection. In: Proceedings of IEEE Security and Privacy, pp. 194–208, May 2004

    Google Scholar 

  20. Fogla, P., Sharif, M., Perdisci, R., Kolesnikov, O., Lee, W.: Polymorphic blending attacks. In: Proceedings of USENIX Security, pp. 241–256 (2006)

    Google Scholar 

  21. Forrest, S., Hofmeyr, S., Somayaji, A.: The evolution of system-call monitoring. In: Proceedings of ACSAC, pp. 418–430, December 2008

    Google Scholar 

  22. Forrest, S., Perelson, A., Allen, L., Cherukuri, R.: Self-nonself discrimination in a computer. In: Proceedings of IEEE Security and Privacy, pp. 202–212, May 1994

    Google Scholar 

  23. Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for unix processes. In: Proceedings of IEEE Security and Privacy, pp. 120–128 (1996)

    Google Scholar 

  24. Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: Proceedings of USENIX Security, vol. 13, p. 8 (2004)

    Google Scholar 

  25. Gao, D., Reiter, M.K., Song, D.: Behavioral distance for intrusion detection. In: Proceedings of RAID, pp. 63–81 (2006)

    Google Scholar 

  26. Gao, D., Reiter, M.K., Song, D.: Behavioral distance measurement using hidden Markov models. In: Zamboni, D., Kruegel, C. (eds.) RAID 2006. LNCS, vol. 4219, pp. 19–40. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  27. Ghosh, A.K., Schwartzbard, A.: A study in using neural networks for anomaly and misuse detection. In: Proceedings of USENIX Security, vol. 8, p. 12 (1999)

    Google Scholar 

  28. Giffin, J.T., Dagon, D., Jha, S., Lee, W., Miller, B.P.: Environment-sensitive intrusion detection. In: Proceedings of RAID, pp. 185–206 (2006)

    Google Scholar 

  29. Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: Proceedings of USENIX Security, pp. 61–79 (2002)

    Google Scholar 

  30. Giffin, J.T., Jha, S., Miller, B.P.: Efficient context-sensitive intrusion detection. In: Proceedings of NDSS (2004)

    Google Scholar 

  31. Gopalakrishna, R., Spafford, E.H., Vitek, J.: Efficient intrusion detection using automaton inlining. In: Proceedings of IEEE Security and Privacy, pp. 18–31, May 2005

    Google Scholar 

  32. Gu, Z., Pei, K., Wang, Q., Si, L., Zhang, X., Xu, D.: Leaps: detecting camouflaged attacks with statistical learning guided by program analysis. In: Processing of DSN, June 2015

    Google Scholar 

  33. Hofmeyr, S.: Primary response technical white paper. http://www.ttivanguard.com/austinreconn/primaryresponse.pdf. Accessed August 2015

  34. Hopcroft, J.E.: Introduction to Automata Theory, Languages, and Computation. Pearson Education India, New Delhi (1979)

    MATH  Google Scholar 

  35. Inoue, H., Somayaji, A.: Lookahead pairs and full sequences: a tale of two anomaly detection methods. In: Proceedings of ASIA, pp. 9–19 (2007)

    Google Scholar 

  36. Kosoresow, A., Hofmeyer, S.: Intrusion detection via system call traces. IEEE Softw. 14(5), 35–42 (1997)

    Article  Google Scholar 

  37. Kruegel, C., Mutz, D., Robertson, W., Valeur, F.: Bayesian event classification for intrusion detection. In: Proceedings of ACSAC, pp. 14–23, December 2003

    Google Scholar 

  38. Kruegel, C., Kirda, E., Mutz, D., Robertson, W., Vigna, G.: Automating mimicry attacks using static binary analysis. In: Proceedings of USENIX Security, vol. 14, p. 11 (2005)

    Google Scholar 

  39. Kuznetsov, V., Szekeres, L., Payer, M., Candea, G., Sekar, R., Song, D.: Code-pointer integrity. In: Proceedings of USENIX OSDI, pp. 147–163 (2014)

    Google Scholar 

  40. Lee, W., Stolfo, S.J.: Data mining approaches for intrusion detection. In: Proceedings of USENIX Security, vol. 7, p. 6 (1998)

    Google Scholar 

  41. Liao, Y., Vemuri, V.: Use of k-nearest neighbor classifier for intrusion detection. Comput. Secur. 21(5), 439–448 (2002)

    Article  Google Scholar 

  42. Liebchen, C., Negro, M., Larsen, P., Davi, L., Sadeghi, A.R., Crane, S., Qunaibit, M., Franz, M., Conti, M.: Losing control: on the effectiveness of control-flow integrity under stack attacks. In: Proceedings of ACM CCS (2015)

    Google Scholar 

  43. Liu, Z., Bridges, S.M., Vaughn, R.B.: Combining static analysis and dynamic learning to build accurate intrusion detection models. In: Proceedings of IWIA, pp. 164–177, March 2005

    Google Scholar 

  44. Maggi, F., Matteucci, M., Zanero, S.: Detecting intrusions through system call sequence and argument analysis. IEEE TDSC 7(4), 381–395 (2010)

    Google Scholar 

  45. Marceau, C.: Characterizing the behavior of a program using multiple-length n-grams. In: Proceedings of NSPW, pp. 101–110 (2000)

    Google Scholar 

  46. Mutz, D., Valeur, F., Vigna, G., Kruegel, C.: Anomalous system call detection. ACM TISSEC 9(1), 61–93 (2006)

    Article  Google Scholar 

  47. Niu, B., Tan, G.: Modular control-flow integrity. SIGPLAN Not. 49(6), 577–587 (2014)

    Article  Google Scholar 

  48. Papadimitriou, C.H.: Computational Complexity. John Wiley and Sons Ltd., New York (2003)

    Google Scholar 

  49. Pullum, G.K.: Context-freeness and the computer processing of human languages. In: Proceedings of ACL, Stroudsburg, PA, USA, pp. 1–6 (1983)

    Google Scholar 

  50. Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of IEEE Security and Privacy, pp. 144–155 (2001)

    Google Scholar 

  51. Shacham, H.: The geometry of innocent flesh on the bone: Return-into-libc without function calls (on the x86). In: Proceedings of ACM CCS, pp. 552–561 (2007)

    Google Scholar 

  52. Sharif, M., Singh, K., Giffin, J.T., Lee, W.: Understanding precision in host based intrusion detection. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 21–41. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  53. Shieber, S.M.: Evidence against the context-freeness of natural language. In: Kulas, J., Fetzer, J.H., Rankin, T.L. (eds.) The Formal Complexity of Natural Language, vol. 33, pp. 320–334. Springer, Heidelberg (1987)

    Chapter  Google Scholar 

  54. Shu, X., Yao, D., Ramakrishnan, N.: Unearthing stealthy program attacks buried in extremely long execution paths. In: Proceedings of ACM CCS (2015)

    Google Scholar 

  55. Sufatrio, Yap, R.: Improving host-based IDS with argument abstraction to prevent mimicry attacks. In: Proceedings of RAID, pp. 146–164 (2006)

    Google Scholar 

  56. Szekeres, L., Payer, M., Wei, T., Song, D.: SoK: eternal war in memory. In: Proceedings of IEEE Security and Privacy, pp. 48–62 (2013)

    Google Scholar 

  57. Tandon, G., Chan, P.K.: On the learning of system call attributes for host-based anomaly detection. IJAIT 15(6), 875–892 (2006)

    Google Scholar 

  58. Vendicator: StackShield. http://www.angelfire.com/sk/stackshield/. Accessed August 2015

  59. Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of IEEE Security and Privacy, pp. 156–168 (2001)

    Google Scholar 

  60. Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of ACM CCS, pp. 255–264 (2002)

    Google Scholar 

  61. Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: alternative data models. In: Proceedings of IEEE S&P, pp. 133–145 (1999)

    Google Scholar 

  62. Wee, K., Moon, B.: Automatic generation of finite state automata for detecting intrusions using system call sequences. In: Proceedings of MMM-ACNS (2003)

    Google Scholar 

  63. Wespi, A., Dacier, M., Debar, H.: Intrusion detection using variable-length audit trail patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, pp. 110–129. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  64. Xu, K., Yao, D., Ryder, B.G., Tian, K.: Probabilistic program modeling for high-precision anomaly classification. In: Proceedings of IEEE CSF (2015)

    Google Scholar 

Download references

Acknowledgments

This work has been supported by ONR grant N00014-13-1-0016. The authors would like to thank Trent Jaeger, Gang Tan, R. Sekar, David Evans and Dongyan Xu for their feedback on this work. The authors would like to thank anonymous reviewers for their comments on stochastic languages.

Author information

Authors and Affiliations

  1. Department of Computer Science, Virginia Tech, Blacksburg, VA, 24060, USA

    Xiaokui Shu, Danfeng (Daphne) Yao & Barbara G. Ryder

Authors
  1. Xiaokui Shu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danfeng (Daphne) Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Barbara G. Ryder
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaokui Shu .

Editor information

Editors and Affiliations

  1. Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands

    Herbert Bos

  2. University of North Carolina at Chapel H, Chapel-Hill, USA

    Fabian Monrose

  3. Université Paris-Saclay, Evry, France

    Gregory Blanc

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Shu, X., Yao, D.(., Ryder, B.G. (2015). A Formal Framework for Program Anomaly Detection. In: Bos, H., Monrose, F., Blanc, G. (eds) Research in Attacks, Intrusions, and Defenses. RAID 2015. Lecture Notes in Computer Science(), vol 9404. Springer, Cham. https://doi.org/10.1007/978-3-319-26362-5_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-26362-5_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-26361-8

  • Online ISBN: 978-3-319-26362-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation