Automatic Extraction of Accurate Application-Specific Sandboxing Policy

  • Lap Chung Lam
  • Tzi-cker Chiueh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3224)


One of the most dangerous cybersecurity threats is control hijacking attacks, which hijack the control of a victim application, and execute arbitrary system calls assuming the identity of the victim program’s effective user. System call monitoring has been touted as an effective defense against control hijacking attacks because it could prevent remote attackers from inflicting damage upon a victim system even if they can successfully compromise certain applications running on the system. However, the Achilles’ heel of the system call monitoring approach is the construction of accurate system call behavior model that minimizes false positives and negatives. This paper describes the design, implementation, and evaluation of a Program semantics-Aware Intrusion Detection system called Paid, which automatically derives an application-specific system call behavior model from the application’s source code, and checks the application’s run-time system call pattern against this model to thwart any control hijacking attacks. The per-application behavior model is in the form of the sites and ordering of system calls made in the application, as well as its partial control flow. Experiments on a fully working Paid prototype show that Paid can indeed stop attacks that exploit non-standard security holes, such as format string attacks that modify function pointers, and that the run-time latency and throughput penalty of Paid are under 11.66% and 10.44%, respectively, for a set of production-mode network server applications including Apache, Sendmail, Ftp daemon, etc.


intrusion detection system call graph sandboxing mimicry attack non-deterministic finite state automaton 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Acharya, A., Mandar, R.: Mapbox: Using parameterized behavior classes to confine untrusted applications. In: Proceedings of the Tenth USENIX Security Symposium (2000)Google Scholar
  2. 2.
    Alexandrov, A., Kmiec, P., Schauser, K.: Consh: A confined execution environment for internet computations. In: USENIX Ann. Technical Conf. (1999)Google Scholar
  3. 3.
    Balfanz, D., Simon, D.R.: Windowbox: a simple security model for the connected desktop. In: Proceedings of the 4th USENIX Windows Systems Symposium, pp. 37–48 (2000)Google Scholar
  4. 4.
    CERT Corrdingation Center. Cert summary cs-2003-01, (2003)
  5. 5.
    cker Chiueh, T., Hsu, F.-H.: Rad: A compiler time solution to buffer overflow attacks. In: Proceedings of International Conference on Distributed Computing Systems (ICDCS), Phoenix, Arizona (April 2001)Google Scholar
  6. 6.
    Cowan, C., Pu, C., Maier, D., Hinton, H., Walpole, J., Bakke, P., Beattie, S., Grier, A., Wagle, P., Zhang, Q.: Stackguard: Automatic adaptive detection and prevention of buffer-overflow attacks. In: Proceedings of the Seventh USENIX Security Symposium, San Antonio, Texas, January 1998, pp. 63–78 (1998)Google Scholar
  7. 7.
    Etho, H.: Gcc extension for protecting applications from stack-smashing attacks,
  8. 8.
    Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of the IEEE Symposium on Security and Privacy, Berkeley, CA, May 2003, pp. 62–76. IEEE Press, Los Alamitos (2003)Google Scholar
  9. 9.
    Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: USENIX Security Symposium (August 2002)Google Scholar
  10. 10.
    Giffin, J.T., Jha, S., Miller, B.P.: Efficient context-sensitive intrusion detection. In: 11th Annual Network and Distributed System Security Symposium (February 2004)Google Scholar
  11. 11.
    Goldberg, I., Wagner, D., Thomas, R., Brewer, E.A.: A secure environment for untrusted helper applications. In: Proceedings of the 6th Usenix Security Symposium, San Jose, CA, USA (1996)Google Scholar
  12. 12.
    Hastings, R., Joyce, B.: Purify: Fast detection of memory leaks and access errors. In: Proceedings of the Winter USENIX Conference, pp. 125–136 (1992)Google Scholar
  13. 13.
    Hofmeyr, S., Forrest, S., Somayaji, A.: Intrusion detection using sequences of system calls. Journal of Computer Security 6(3) (1998)Google Scholar
  14. 14.
    Kiriansky, V., Bruening, D., Amarasinghe, S.: Secure execution via program shepherding. In: 11th USENIX Security Symposium (August 2002)Google Scholar
  15. 15.
    Nguyen, N., Reiher, P., Kuenning, G.H.: Detecting insider threats by monitoring system call activity. In: IEEE Information Assurance Workshop, United States Military Academy West Point, New York (June 2003)Google Scholar
  16. 16.
    Prasad, M., cker Chiueh, T.: A binary rewriting approach to stack-based buffer overflow attacks. In: Proceedings of 2003 USENIX Conference (June 2003)Google Scholar
  17. 17.
    Prevelakis, V., Spinellis, D.: Sandboxing applications. In: Proceedings of the FREENIX Track: 2001 USENIX Annual Technical Conference, Berkeley, CA, June 2001, pp. 119–126. USENIX Association (2001)Google Scholar
  18. 18.
    Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based method for detecting anomalous program behaviors. IEEE Symposium on Security and Privacy, 144–155 (2001)Google Scholar
  19. 19.
    Solar Designer. Non-executable user stack,
  20. 20.
    TESO Security. x86/linux wu ftpd remote root exploit,
  21. 21.
    Vendicator. Stackshield: A “stack smashing” technique protection tool for linux,
  22. 22.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the IEEE Symposium on Security and Privacy, Oakland, CA, May 2001, pp. 156–169. IEEE Press, Los Alamitos (2001)Google Scholar
  23. 23.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security (November 2002)Google Scholar
  24. 24.
    Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: Alternative data models (May 1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Lap Chung Lam
    • 1
  • Tzi-cker Chiueh
    • 1
  1. 1.Rether Networks, Inc.CentereachUSA

Personalised recommendations