On the Detection of Anomalous System Call Arguments

  • Christopher Kruegel
  • Darren Mutz
  • Fredrik Valeur
  • Giovanni Vigna
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2808)


Learning-based anomaly detection systems build models of the expected behavior of applications by analyzing events that are generated during their normal operation. Once these models have been established, subsequent events are analyzed to identify deviations, given the assumption that anomalies usually represent evidence of an attack.

Host-based anomaly detection systems often rely on system call traces to build models and perform intrusion detection. Recently, these systems have been criticized, and it has been shown how detection can be evaded by executing an attack using a carefully crafted exploit. This weakness is caused by the fact that existing models do not take into account all available features of system calls. In particular, some attacks will go undetected because the models do not make use of system call arguments. To solve this problem, we have developed an anomaly detection technique that utilizes the information contained in these parameters. Based on our approach, we developed a host-based intrusion detection system that identifies attacks using a composition of various anomaly metrics.

This paper presents our detection techniques and the tool based on them. The experimental evaluation shows that it is possible to increase both the effectiveness and the precision of the detection process compared to previous approaches. Nevertheless, the system imposes only minimal overhead.


Intrusion detection anomaly models system calls 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Advisory: Input validation problems in wuftpd (2000),
  2. 2.
    Advisory: Buffer overflow in linuxconf (2002),
  3. 3.
    Bernaschi, M., Gabrielli, E., Mancini, L.V.: REMUS: a Security-Enhanced Operating System. ACM Transactions on Information and System Security 5(36) (February 2002)Google Scholar
  4. 4.
    Billingsley, P.: Probability and Measure, 3rd edn. Wiley-Interscience, Hoboken (April 1995)zbMATHGoogle Scholar
  5. 5.
    Chari, S.N., Cheng, P.-C.: Bluebox: A policy-driven, host-based intrusion detection system. In: Proceedings of the 2002 ISOC Symposium on Network and Distributed System Security (NDSS 2002), San Diego, CA (2002)Google Scholar
  6. 6.
    Denning, D.E.: An Intrusion Detection Model. IEEE Transactions on Software Engineering 13(2), 222–232 (1987)CrossRefGoogle Scholar
  7. 7.
    Eckmann, S.T., Vigna, G., Kemmerer, R.A.: STATL: An Attack Language for State-based Intrusion Detection. Journal of Computer Security 10(1/2), 71–104 (2002)Google Scholar
  8. 8.
    Forrest, S.: A Sense of Self for UNIX Processes. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, pp. 120–128 (May 1996)Google Scholar
  9. 9.
    Ghosh, A.K., Wanken, J., Charron, F.: Detecting Anomalous and Unknown Intrusions Against Programs. In: Proceedings of the Annual Computer Security Application Conference (ACSAC 1998), Scottsdale, AZ, pp. 259–267 (December 1998)Google Scholar
  10. 10.
    Goldberg, I., Wagner, D., Thomas, R., Brewer, E.A.: A secure environment for untrusted helper applications. In: Proceedings of the 6th Usenix Security Symposium, San Jose, CA, USA (1996)Google Scholar
  11. 11.
    Javitz, H.S., Valdes, A.: The SRI IDES Statistical Anomaly Detector. In: Proceedings of the IEEE Symposium on Security and Privacy (May 1991)Google Scholar
  12. 12.
    Ko, C., Ruschitzka, M., Levitt, K.: Execution Monitoring of Security-Critical Programs in Distributed Systems: A Specification-based Approach. In: Proceedings of the 1997 IEEE Symposium on Security and Privacy, pp. 175–187 (May 1997)Google Scholar
  13. 13.
    MIT Lincoln Laboratory. DARPA Intrusion Detection Evaluation (1999),
  14. 14.
    Lee, S.Y., Low, W.L., Wong, P.Y.: Learning Fingerprints for a Database Intrusion Detection System. In: Gollmann, D., Karjoth, G., Waidner, M. (eds.) ESORICS 2002. LNCS, vol. 2502, p. 264. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  15. 15.
    Lee, W., Stolfo, S., Chan, P.: Learning Patterns from Unix Process Execution Traces for Intrusion Detection. In: Proceedings of the AAAI Workshop: AI Approaches to Fraud Detection and Risk Management (July 1997)Google Scholar
  16. 16.
    Lee, W., Stolfo, S., Mok, K.: Mining in a Data-flow Environment: Experience in Network Intrusion Detection. In: Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 1999), San Diego, CA (August 1999)Google Scholar
  17. 17.
    Lindqvist, U., Porras, P.A.: Detecting Computer and Network Misuse with the Production-Based Expert System Toolset (P-BEST). In: IEEE Symposium on Security and Privacy, Oakland, California, pp. 146–161 (May 1999)Google Scholar
  18. 18.
    Paxson, V.: Bro: A System for Detecting Network Intruders in Real-Time. In: Proceedings of the 7th USENIX Security Symposium, San Antonio, TX (January 1998)Google Scholar
  19. 19.
    Provos, N.: Improving host security with system call policies. In: Proceedings of the 12th Usenix Security Symposium, Washington, DC (2003)Google Scholar
  20. 20.
    SNARE - System iNtrusion Analysis and Reporting Environment,
  21. 21.
    Stolcke, A., Omohundro, S.: HiddenMarkov Model Induction by Bayesian Model Merging. In: Advances in Neural Information Processing Systems (1993)Google Scholar
  22. 22.
    Stolcke, A., Omohundro, S.: Inducing probabilistic grammars by bayesian model merging. In: International Conference on Grammatical Inference (1994)Google Scholar
  23. 23.
    Tan, K., Maxion, R.: “Why 6?” Defining the Operational Limits of Stide, an Anomaly-Based Intrusion Detector. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, pp. 188–202 (May 2002)Google Scholar
  24. 24.
    Tan, K.M.C., Killourhy, K.S., Maxion, R.A.: Undermining an Anomaly-Based Intrusion Detection System Using Common Exploits. In: Proceedings of the 5th International Symposium on Recent Advances in Intrusion Detection, Zurich, Switzerland, pp. 54–73 (October 2002)Google Scholar
  25. 25.
    Wagner, D., Dean, D.: Intrusion Detection via Static Analysis. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA. IEEE Press, Los Alamitos (May 2001)Google Scholar
  26. 26.
    Wagner, D., Soto, P.: Mimicry Attacks on Host-Based Intrusion Detection Systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security, Washington DC, USA, pp. 255–264 (November 2002)Google Scholar
  27. 27.
    Warrender, C., Forrest, S., Pearlmutter, B.A.: Detecting intrusions using system calls: Alternative data models. In: IEEE Symposium on Security and Privacy, pp. 133–145 (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Christopher Kruegel
    • 1
  • Darren Mutz
    • 1
  • Fredrik Valeur
    • 1
  • Giovanni Vigna
    • 1
  1. 1.Reliable Software Group, Department of Computer ScienceUniversity of CaliforniaSanta Barbara

Personalised recommendations