Detecting Illegal System Calls Using a Data-Oriented Detection Model

  • Jonathan-Christofer Demay
  • Frédéric Majorczyk
  • Eric Totel
  • Frédéric Tronel
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 354)


The most common anomaly detection mechanisms at application level consist in detecting a deviation of the control-flow of a program. A popular method to detect such anomaly is the use of application sequences of system calls. However, such methods do not detect mimicry attacks or attacks against the integrity of the system call parameters. To enhance such detection mechanisms, we propose an approach to detect in the application the corruption of data items that have an influence on the system calls. This approach consists in building automatically a data-oriented behaviour model of an application by static analysis of its source code. The proposed approach is illustrated on various examples, and an injection method is experimented to obtain an approximation of the detection coverage of the generated mechanisms.


  1. 1.
    Hofmeyr, S.A., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security (1998)Google Scholar
  2. 2.
    Kruegel, C., Kirda, E., Mutz, D., Robertson, W.: Automating mimicry attacks using static binary analysis. In: 14th Conference on USENIX Security Symposium (2005)Google Scholar
  3. 3.
    Kruegel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 326–343. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  4. 4.
    Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: 2006 IEEE Symposium on Security and Privacy (S&P 2006) (2006)Google Scholar
  5. 5.
    Mutz, D., Robertson, W., Vigna, G., Kemmerer, R.: Exploiting execution context for the detection of anomalous system calls. In: Kruegel, C., Lippmann, R., Clark, A. (eds.) RAID 2007. LNCS, vol. 4637, pp. 1–20. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Feng, H., Kolesnikov, O., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: IEEE Symposium on Security and Privacy, p. 65 (2003)Google Scholar
  7. 7.
    CEA: Frama-c, framework for modular analysis of cGoogle Scholar
  8. 8.
    Abadi, M., Budiu, M., Erlingsson, U., Ligatti, J.: Control-flow integrity. In: CCS 2005: Proceedings of the 12th ACM Conference on Computer and Communications Security (2005)Google Scholar
  9. 9.
    Kiriansky, V., Bruening, D., Amarasinghe, S.: Secure execution via program shepherding. In: Proceedings of the Usenix Security Symposium (2002)Google Scholar
  10. 10.
    Chen, S., Xu, J., Sezer, E., Gauriar, P., Iyer, R.: Non-control-data attacks are realistic threats. In: Usenix Security Symposium (2005)Google Scholar
  11. 11.
    Akritidis, P., Cadar, C., Raiciu, C., Costa, M., Castro, M.: Preventing memory error exploits with wit. In: 2008 IEEE Symposium on Security and Privacy (2008)Google Scholar
  12. 12.
    Castro, M., Costa, M., Harris, T.: Securing software by enforcing data-flow integrity. In: 7th USENIX Symposium on Operating Systems Design and Implementation (2006)Google Scholar
  13. 13.
    Demay, J.C., Totel, E., Tronel, F.: Sidan: a tool dedicated to software instrumentation for detecting attacks on non-control-data. In: 4th International Conference on Risks and Security of Internet and Systems (CRISIS 2009), Toulouse (October 2009)Google Scholar
  14. 14.
    Weiser, M.: Program slicing. IEEE Transactions on Software Engineering (1982)Google Scholar
  15. 15.
    Kuck, D.J., Kuhn, R.H., Padua, D.A., Leasure, B., Wolfe, M.: Dependence graphs and its use in optimization. In: 8th ACM Symposium on Principles of Programming Languages (1981)Google Scholar
  16. 16.
    Cousot, P., Cousot, R.: Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In: Proceedings of the 4th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (1977)Google Scholar
  17. 17.
    Granger, P.: Static analysis of arithmetical congruences. International Journal of Computer Mathematics 30, 165–190 (1989)zbMATHCrossRefGoogle Scholar
  18. 18.
    Cousot, P., Halbwachs, N.: Automatic discovery of linear restraints among variables of a program. In: Proceedings of the 5th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (1978)Google Scholar
  19. 19.
    Karr, M.: Affine relationships among variables of a program. Acta Informatica, 133–151 (1976)Google Scholar
  20. 20.
    Granger, P.: Static analysis of linear congruence equalities among variables of a program. In: TAPSOFT 1991, pp. 169–192 (1991)Google Scholar
  21. 21.
    Goloubeva, O., Rebaudengo, M., Reorda, M.S., Violante, M.: Soft-error detection using control flow assertions. In: Proceedings of the 18th IEEE International Symposium on Defect and Fault Tolerance in VLSI Systems (DFT 2003) (2003)Google Scholar
  22. 22.
    Vemu, R., Abraham, J.A.: Ceda: Control-flow error detection through assertions. In: Proceedings of the 12th IEEE International On-Line Testing Symposium (2006)Google Scholar
  23. 23.
    Neves, N., Antunes, J., Correia, M., Verissimo, P., Neves, R.: Using attack injection to discover new vulnerabilities. In: Conference on Dependable Systems and Networks (2006)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Jonathan-Christofer Demay
    • 1
  • Frédéric Majorczyk
    • 2
  • Eric Totel
    • 1
  • Frédéric Tronel
    • 1
  1. 1.SupelecRennesFrance
  2. 2.IRISA / Université de Rennes 1RennesFrance

Personalised recommendations