Automated Discovery of Mimicry Attacks

  • Jonathon T. Giffin
  • Somesh Jha
  • Barton P. Miller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4219)


Model-based anomaly detection systems restrict program execution by a predefined model of allowed system call sequences. These systems are useful only if they detect actual attacks. Previous research developed manually-constructed mimicry and evasion attacks that avoided detection by hiding a malicious series of system calls within a valid sequence allowed by the model. Our work helps to automate the discovery of such attacks. We start with two models: a program model of the application’s system call behavior and a model of security-critical operating system state. Given unsafe OS state configurations that describe the goals of an attack, we then find system call sequences allowed as valid execution by the program model that produce the unsafe configurations. Our experiments show that we can automatically find attack sequences in models of programs such as wu-ftpd and passwd that previously have only been discovered manually. When undetected attacks are present, we frequently find the sequences with less than 2 seconds of computation.


IDS evaluation model checking attacks model-based anomaly detection 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ball, T., Rajamani, S.K.: Bebop: A symbolic model checker for boolean programs. In: 7th International SPIN Workshop on Model Checking of Software. Stanford, California (2000)Google Scholar
  2. 2.
    Besson, F., Jensen, T., Métayer, D.L., Thorn, T.: Model checking security properties of control-flow graphs. Journal of Computer Security 9, 217–250 (2001)Google Scholar
  3. 3.
    Chen, H., Wagner, D.: MOPS: An infrastructure for examining security properties of software. In: 9th ACM Conference on Computer and Communications Security (CCS), Washington, DC (November 2002)Google Scholar
  4. 4.
    Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. MIT Press, Cambridge (2000)Google Scholar
  5. 5.
    Esparza, J., Hansel, D., Rossmanith, P., Schwoon, S.: Efficient algorithms for model checking pushdown systems. In: Computer Aided Verification (CAV), Chicago, Illinois (July 2000)Google Scholar
  6. 6.
    Feng, H.H., Giffin, J.T., Huang, Y., Jha, S., Lee, W., Miller, B.P.: Formalizing sensitivity in static analysis for intrusion detection. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2004)Google Scholar
  7. 7.
    Forrest, S.: Data sets—synthetic FTP (1998),
  8. 8.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for UNIX processes. In: IEEE Symposium on Security and Privacy, Oakland, California (May 1996)Google Scholar
  9. 9.
    Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: USENIX Security Symposium, San Diego, California (August 2004)Google Scholar
  10. 10.
    Giffin, J.T., Dagon, D., Jha, S., Lee, W., Miller, B.P.: Environment-sensitive intrusion detection. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 185–206. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: 11th USENIX Security Symposium, San Francisco, California (August 2002)Google Scholar
  12. 12.
    Gopalakrishna, R., Spafford, E.H., Vitek, J.: Efficient intrusion detection using automaton inlining. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2005)Google Scholar
  13. 13.
    Guttman, J.D., Herzog, A.L., Ramsdell, J.D., Skorupka, C.W.: Verifying information flow goals in Security-Enhanced Linux. Journal of Computer Security 13, 115–134 (2005)Google Scholar
  14. 14.
    Lam, L.C., Chiueh, T.-c.: Automatic extraction of accurate application-specific sandboxing policy. In: Jonsson, E., Valdes, A., Almgren, M. (eds.) RAID 2004. LNCS, vol. 3224, pp. 1–20. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  15. 15.
    Ramakrishnan, C.R., Sekar, R.: Model-based vulnerability analysis of computer systems. In: 2nd International Workshop on Verification, Model Checking and Abstract Interpretation, Pisa, Italy (September 1998)Google Scholar
  16. 16.
    Schneider, F.B.: Enforceable security policies. ACM Transactions on Information and System Security 3(1), 30–50 (2000)CrossRefGoogle Scholar
  17. 17.
    Schwoon, S.: Model-Checking Pushdown Systems. Ph.D. dissertation, Technische Universität München (June 2002)Google Scholar
  18. 18.
    Schwoon, S.: Moped—a model-checker for pushdown systems (2006),
  19. 19.
    Sekar, R., Bendre, M., Bollineni, P., Dhurjati, D.: A fast automaton-based method for detecting anomalous program behaviors. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2001)Google Scholar
  20. 20.
    Tan, K.M.C., Killourhy, K.S., Maxion, R.A.: Undermining an anomaly-based intrusion detection system using common exploits. In: Wespi, A., Vigna, G., Deri, L. (eds.) RAID 2002. LNCS, vol. 2516, p. 54. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  21. 21.
    Tan, K., Maxion, R.A.: “Why 6?” Defining the operational limits of stide, an anomaly based intrusion detector. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2002)Google Scholar
  22. 22.
    Tan, K., McHugh, J., Killourhy, K.: Hiding intrusions: From the abnormal to the normal and beyond. In: 5th International Workshop on Information Hiding, Noordwijkerhout, Netherlands (October 2002)Google Scholar
  23. 23.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: IEEE Symposium on Security and Privacy, Oakland, California (May 2001)Google Scholar
  24. 24.
    Wagner, D., Soto, P.: Mimicry attacks on host based intrusion detection systems. In: 9th ACM Conference on Computer and Communications Security, Washington, DC (November 2002)Google Scholar
  25. 25.
    Walker, B.J., Kemmerer, R.A., Popek, G.J.: Specification and verification of the UCLA Unix security kernel. Communications of the ACM 23(2) (February 1980)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jonathon T. Giffin
    • 1
  • Somesh Jha
    • 1
  • Barton P. Miller
    • 1
  1. 1.Computer Sciences DepartmentUniversity of Wisconsin 

Personalised recommendations