Selecting and Improving System Call Models for Anomaly Detection

  • Alessandro Frossi
  • Federico Maggi
  • Gian Luigi Rizzo
  • Stefano Zanero
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5587)

Abstract

We propose a syscall-based anomaly detection system that incorporates both deterministic and stochastic models. We analyze in detail two alternative approaches for anomaly detection over system call sequences and arguments, and propose a number of modifications that significantly improve their performance. We begin by comparing them and analyzing their respective performance in terms of detection accuracy. Then, we outline their major shortcomings, and propose various changes in the models that can address them: we show how targeted modifications of their anomaly models, as opposed to the redesign of the global system, can noticeably improve the overall detection accuracy. Finally, the impact of these modifications are discussed by comparing the performance of the two original implementations with two modified versions complemented with our models.

Keywords

Anomaly Detection System Call Models Deterministic Models Stochastic Models Self Organizing Map 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security 6(3), 151–180 (1998)CrossRefGoogle Scholar
  2. 2.
    Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: IEEE Symposium on Security and Privacy, May 2006, pp. 15–62 (May 2006)Google Scholar
  3. 3.
    Maggi, F., Matteucci, M., Zanero, S.: Detecting intrusions through system call sequence and argument analysis. IEEE Transactions on Dependable and Secure Computing (accepted for publication)Google Scholar
  4. 4.
    Sharif, M.I., Singh, K., Giffin, J.T., Lee, W.: Understanding precision in host based intrusion detection. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 21–41. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Zanero, S.: Unsupervised Learning Algorithms for Intrusion Detection. PhD thesis, Politecnico di Milano T.U., Milano, Italy (May 2006)Google Scholar
  6. 6.
    Han, J., Kamber, M.: Data Mining: concepts and techniques. Morgan-Kauffman, San Francisco (2000)MATHGoogle Scholar
  7. 7.
    Cabrera, J.B.D., Lewis, L., Mehara, R.: Detection and classification of intrusion and faults using sequences of system calls. ACM SIGMOD Record 30(4) (2001)Google Scholar
  8. 8.
    Casas-Garriga, G., Díaz, P., Balcázar, J.: ISSA: An integrated system for sequence analysis. Technical Report DELIS-TR-0103, Universitat Paderborn (2005)Google Scholar
  9. 9.
    Ourston, D., Matzner, S., Stump, W., Hopkins, B.: Applications of hidden markov models to detecting multi-stage network attacks. In: HICSS, p. 334 (2003)Google Scholar
  10. 10.
    Jha, S., Tan, K., Maxion, R.A.: Markov chains, classifiers, and intrusion detection. In: Proceedings of the 14th IEEE Workshop on Computer Security Foundations (CSFW 2001), Washington, DC, USA, June 2001, pp. 206–219. IEEE Computer Society Press, Los Alamitos (2001)CrossRefGoogle Scholar
  11. 11.
    Joanes, D., Gill, C.: Comparing Measures of Sample Skewness and Kurtosis. The Statistician 47(1), 183–189 (1998)Google Scholar
  12. 12.
    Elmagarmid, A., Ipeirotis, P., Verykios, V.: Duplicate Record Detection: A Survey. IEEE Transactions on Knowledge and Data Engineering 19(1), 1–16 (2007)CrossRefGoogle Scholar
  13. 13.
    Somervuo, P.J.: Online algorithm for the self-organizing map of symbol strings. Neural Netw. 17(8-9), 1231–1239 (2004)CrossRefGoogle Scholar
  14. 14.
    Kohonen, T., Somervuo, P.: Self-organizing maps of symbol strings. Neurocomputing 21(1-3), 19–30 (1998)CrossRefMATHGoogle Scholar
  15. 15.
    Zanero, S.: Flaws and frauds in the evaluation of IDS/IPS technologies. In: Proc. of FIRST 2007 - Forum of Incident Response and Security Teams, Sevilla, Spain (June 2007)Google Scholar
  16. 16.
    Maggi, F., Zanero, S., Iozzo, V.: Seeing the invisible - forensic uses of anomaly detection and machine learning. ACM Operating Systems Review (April 2008)Google Scholar
  17. 17.
    Bace, R.G.: Intrusion detection. Macmillan Publishing Co., Inc., Indianapolis (2000)Google Scholar
  18. 18.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for Unix processes. In: Proceedings of the 1996 IEEE Symposium on Security and Privacy, Washington, DC, USA. IEEE Computer Society, Los Alamitos (1996)Google Scholar
  19. 19.
    Forrest, S., Perelson, A.S., Allen, L., Cherukuri, R.: Self-nonself discrimination in a computer. In: SP 1994: Proceedings of the 1994 IEEE Symposium on Security and Privacy, Washington, DC, USA, p. 202. IEEE Computer Society, Los Alamitos (1994)CrossRefGoogle Scholar
  20. 20.
    Somayaji, A., Forrest, S.: Automated response using system–call delays. In: Proceedings of the 9th USENIX Security Symposium, Denver, CO (August 2000)Google Scholar
  21. 21.
    Michael, C.C., Ghosh, A.: Simple, state-based approaches to program-based anomaly detection. ACM Trans. Inf. Syst. Secur. 5(3), 203–237 (2002)CrossRefGoogle Scholar
  22. 22.
    Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the 2001 IEEE Symposium on Security and Privacy, Washington, DC, USA. IEEE Computer Society Press, Los Alamitos (2001)Google Scholar
  23. 23.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: SP 2001: Proceedings of the 2001 IEEE Symposium on Security and Privacy, Washington, DC, USA, pp. 156–168. IEEE Computer Society Press, Los Alamitos (2001)CrossRefGoogle Scholar
  24. 24.
    Giffin, J.T., Dagon, D., Jha, S., Lee, W., Miller, B.P.: Environment-sensitive intrusion detection. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 185–206. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  25. 25.
    Feng, H., Kolesnikov, O., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings. 2003 Symposium on Security and Privacy, 2003, May 11-14, pp. 62–75 (2003)Google Scholar
  26. 26.
    Warrender, C., Forrest, S., Pearlmutter, B.A.: Detecting intrusions using system calls: Alternative data models. In: IEEE Symposium on Security and Privacy, pp. 133–145 (1999)Google Scholar
  27. 27.
    Jha, S., Tan, K., Maxion, R.A.: Markov chains, classifiers, and intrusion detection. In: CSFW 2001: Proceedings of the 14th IEEE Workshop on Computer Security Foundations, pp. 206–219. IEEE Computer Society, Washington (2001)CrossRefGoogle Scholar
  28. 28.
    Yeung, D.Y., Ding, Y.: Host-based intrusion detection using dynamic and static behavioral models. Pattern Recognition 36, 229–243 (2003)CrossRefMATHGoogle Scholar
  29. 29.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: CCS 2002: Proceedings of the 9th ACM conference on Computer and communications security, pp. 255–264. ACM, New York (2002)Google Scholar
  30. 30.
    Krügel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 326–343. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  31. 31.
    Tandon, G., Chan, P.: Learning rules from system call arguments and sequences for anomaly detection. In: ICDM Workshop on Data Mining for Computer Security (DMSEC), pp. 20–29 (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Alessandro Frossi
    • 1
  • Federico Maggi
    • 1
  • Gian Luigi Rizzo
    • 1
  • Stefano Zanero
    • 1
  1. 1.Dipartimento di Elettronica e InformazionePolitecnico di MilanoItaly

Personalised recommendations