Behavioral Distance Measurement Using Hidden Markov Models

  • Debin Gao
  • Michael K. Reiter
  • Dawn Song
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4219)


The behavioral distance between two processes is a measure of the deviation of their behaviors. Behavioral distance has been proposed for detecting the compromise of a process, by computing its behavioral distance from another process executed on the same input. Provided that the two processes are diverse and so unlikely to fall prey to the same attacks, an increase in behavioral distance might indicate the compromise of one of them. In this paper we propose a new approach to behavioral distance calculation using a new type of Hidden Markov Model. We also empirically evaluate the intrusion detection capability of our proposal when used to measure the distance between the system-call behaviors of diverse web servers. Our experiments show that it detects intrusions with substantially greater accuracy and with performance overhead comparable to that of prior proposals.


intrusion detection anomaly detection system call behavioral distance 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abd-El-Malek, M., Ganger, G.R., Goodson, G.R., Reiter, M.K., Wylie, J.J.: Fault-scalable Byzantine fault-tolerant services. In: Proceedings of the 20th ACM Symposium on Operating Systems Principles, pp. 59–74 (October 2005)Google Scholar
  2. 2.
    Alvisi, L., Malkhi, D., Pierce, E., Reiter, M.K.: Fault detection for Byzantine quorum systems. IEEE Transactions on Parallel Distributed Systems 12(9) (September 2001)Google Scholar
  3. 3.
    Baum, L.E., Petrie, T.: Statistical inference for probabilistic functions of finite state Markov chains. Ann. Math. Statist. 37, 1554–1563 (1966)zbMATHCrossRefMathSciNetGoogle Scholar
  4. 4.
    Bhatkar, S., Chaturvedi, A., Sekar, R.: Dataflow anomaly detection. In: Proceedings of the 2006 IEEE Symposium on Security and Privacy (2006)Google Scholar
  5. 5.
    Buskens, R.W., Bianchini Jr., R.P.: Distributed on-line diagnosis in the presence of arbitrary faults. In: Proceedings of the 23rd International Symposium on Fault-Tolerant Computing, pp. 470–479 (June 1993)Google Scholar
  6. 6.
    Cachin, C., Poritz, J.A.: Secure intrusion-tolerant replication on the Internet. In: Proceedings of the 2002 International Conference on Dependable Systems and Networks (2002)Google Scholar
  7. 7.
    Castro, M., Liskov, B.: Practical Byzantine fault tolerance and proactive recovery. ACM Transactions on Computer Systems 20(4) (November 2002)Google Scholar
  8. 8.
    Castro, M., Rodrigues, R., Liskov, B.: BASE: Using abstraction to improve fault tolerance. ACM Transactions on Computer Systems 21(3) (August 2003)Google Scholar
  9. 9.
    Chen, L., Avizienis, A.: N-version programming: A fault-tolerance approach to reliability of software operation. In: Proceedings of the 8th International Symposium on Fault-Tolerant Computing, pp. 3–9 (1978)Google Scholar
  10. 10.
    Cho, S.-B., Han, S.-J.: Two sophisticated techniques to improve HMM-based intrusion detection systems. In: Vigna, G., Krügel, C., Jonsson, E. (eds.) RAID 2003. LNCS, vol. 2820, pp. 207–219. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  11. 11.
    Cox, B., Evans, D., Filipi, A., Rowanhill, J., Hu, W., Davidson, J., Knight, J., Nguyen-Tuong, A., Hiser, J.: N-variant systems – A secretless framework for security through diversity. In: Proceedings of the 15th USENIX Security Symposium (August 2006)Google Scholar
  12. 12.
    Davis, R.I.A., Lovell, B.C., Caelli, T.: Improved estimation of Hidden Markov Model parameters from multiple observation sequences. In: Proceedings of the 16th International Conference on Pattern Recognition (ICPR 2002) (2002)Google Scholar
  13. 13.
    Feng, H.H., Giffin, J.T., Huang, Y., Jha, S., Lee, W., Miller, B.P.: Formalizing sensitivity in static analysis for intrusion detection. In: Proceedings of the 2004 IEEE Symposium on Security and Privacy (2004)Google Scholar
  14. 14.
    Feng, H.H., Kolesnikov, O.M., Fogla, P., Lee, W., Gong, W.: Anomaly detection using call stack information. In: Proceedings of the 2003 IEEE Symposium on Security and Privacy (2003)Google Scholar
  15. 15.
    Forrest, S., Hofmeyr, S.A., Somayaji, A., Longstaff, T.A.: A sense of self for Unix processes. In: Proceedings of the 1996 IEEE Symposium on Security and Privacy (1996)Google Scholar
  16. 16.
    Gao, D., Reiter, M.K., Song, D.: Gray-box extraction of execution graph for anomaly detection. In: Proceedings of the 11th ACM Conference on Computer & Communication Security (2004)Google Scholar
  17. 17.
    Gao, D., Reiter, M.K., Song, D.: On gray-box program tracking for anomaly detection. In: Proceedings of the 13th USENIX Security Symposium (2004)Google Scholar
  18. 18.
    Gao, D., Reiter, M.K., Song, D.: Behavioral distance for intrusion detection. In: Valdes, A., Zamboni, D. (eds.) RAID 2005. LNCS, vol. 3858, pp. 63–81. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  19. 19.
    Giffin, J.T., Jha, S., Miller, B.P.: Detecting manipulated remote call streams. In: Proceedings of the 11th USENIX Security Symposium (2002)Google Scholar
  20. 20.
    Giffin, J.T., Jha, S., Miller, B.P.: Efficient context-sensitive intrusion detection. In: Proceedings of Symposium on Network and Distributed System Security (2004)Google Scholar
  21. 21.
    Krügel, C., Mutz, D., Valeur, F., Vigna, G.: On the detection of anomalous system call arguments. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 326–343. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  22. 22.
    Lamport, L.: The implementation of reliable distributed multiprocess systems. Computer Networks 2, 95–114 (1978)MathSciNetGoogle Scholar
  23. 23.
    Meyer, I.M., Durbin, R.: Comparative ab initio prediction of gene structures using pair HMMs. Oxford University Press, Oxford (2002)Google Scholar
  24. 24.
    Pachter, L., Alexandersson, M., Cawley, S.: Applications of generalized pair Hidden Markov Models to alignment and gene finding problems. Computational Biology 9(2) (2002)Google Scholar
  25. 25.
    Rabiner, L.R.: A tutorial on Hidden Markov Models and selected applications in speech recognition. Proceedings of IEEE (February 1989)Google Scholar
  26. 26.
    Reiter, M.K.: Secure agreement protocols: Reliable and atomic group multicast in Rampart. In: Proceedings of the 2nd ACM Conference on Computer and Communication Security, pp. 68–80 (November 1994)Google Scholar
  27. 27.
    Schneider, F.B.: Implementing fault-tolerant services using the state machine approach: A tutorial. ACM Computing Surveys 22(4), 299–319 (1990)CrossRefGoogle Scholar
  28. 28.
    Sekar, R., Bendre, M., Dhurjati, D., Bollineni, P.: A fast automaton-based method for detecting anomalous program behaviors. In: Proceedings of the 2001 IEEE Symposium on Security and Privacy (2001)Google Scholar
  29. 29.
    Sellers, P.H.: On the theory and computation of evolutionary distances. SIAM J. Appl. Math. 26, 787–793 (1974)zbMATHCrossRefMathSciNetGoogle Scholar
  30. 30.
    Shin, K., Ramanathan, P.: Diagnosis of processors with Byzantine faults in a distributed computing system. In: Proceedings of the 17th International Symposium on Fault-Tolerant Computing, pp. 55–60 (1987)Google Scholar
  31. 31.
    Tan, K., McHugh, J., Killourhy, K.: Hiding intrusions: From the abnormal to the normal and beyond. In: Petitcolas, F.A.P. (ed.) IH 2002. LNCS, vol. 2578, pp. 1–17. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  32. 32.
    Wagner, D., Dean, D.: Intrusion detection via static analysis. In: Proceedings of the 2001 IEEE Symposium on Security and Privacy (2001)Google Scholar
  33. 33.
    Wagner, D., Soto, P.: Mimicry attacks on host-based intrusion detection systems. In: Proceedings of the 9th ACM Conference on Computer and Communications Security (2002)Google Scholar
  34. 34.
    Warrender, C., Forrest, S., Pearlmutter, B.: Detecting intrusions using system calls: alternative data models. In: Proceedings of the 1999 IEEE Symposium on Security and Privacy (1999)Google Scholar
  35. 35.
    Wespi, A., Dacier, M., Debar, H.: Intrusion detection using variable-length audit trail patterns. In: Debar, H., Mé, L., Wu, S.F. (eds.) RAID 2000. LNCS, vol. 1907, p. 110. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  36. 36.
    Yin, J., Martin, J., Venkataramani, A., Alvisi, L., Dahlin, M.: Separating agreement from execution for Byzantine fault tolerant services. In: Proceedings of the 19th ACM Symposium on Operating System Principles (October 2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Debin Gao
    • 1
  • Michael K. Reiter
    • 1
  • Dawn Song
    • 1
  1. 1.Carnegie Mellon University 

Personalised recommendations