Detecting Malicious Sessions Through Traffic Fingerprinting Using Hidden Markov Models

  • Sami ZhiouaEmail author
  • Adnene Ben Jabeur
  • Mahjoub Langar
  • Wael Ilahi
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 152)


Almost any malware attack involves data communication between the infected host and the attacker host/server allowing the latter to remotely control the infected host. The remote control is achieved through opening different types of sessions such as remote desktop, webcam video streaming, file transfer, etc. In this paper, we present a traffic analysis based malware detection technique using Hidden Markov Model (HMM). The main contribution is that the proposed system does not only detect malware infections but also identifies with precision the type of malicious session opened by the attacker. The empirical analysis shows that the proposed detection system has a stable identification precision of 90 % and that it allows to identify between 40 % and 75 % of all malicious sessions in typical network traffic.


Malware detection Hidden Markov Model (HMM) Malicious sessions Traffic analysis 


  1. 1.
    Siroski, M., Honig, A.: Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, San Francisco (2012)Google Scholar
  2. 2.
    Falliere, N., Murchu, L., Chien, E.: W32.stuxnet dossier. Technical report, Symantec Security Response, February 2011Google Scholar
  3. 3.
    Gostev, A.: The flame: Questions and answers. Technical report, Kaspersky, May 2012Google Scholar
  4. 4.
    Bencsáth, B., Pék, G., Buttyán, L., Félegyházi, M.: Duqu: analysis, detection, and lessons learned. In: ACM European Workshop on System Security (EuroSec). ACM (2012)Google Scholar
  5. 5.
    Leyden, J.: Hack on Saudi Aramco hit 30,000 workstations, oil firm admits (2012).
  6. 6.
    Sun, Q., Simon, D.R., Wang, Y.M., Russell, W., Padmanabhan, V.N., Qiu, L.: Statistical identification of encrypted web browsing traffic. In: Proceedings of the 2002 IEEE Symposium on Security and Privacy, SP 2002, p. 19. IEEE Computer Society, Washington, DC (2002)Google Scholar
  7. 7.
    Liberatore, M., Levine, B.N.: Inferring the source of encrypted http connections. In: Proceedings of the 13th ACM conference on Computer and Communications Security, CCS 2006, pp. 255–263. ACM, New York (2006)Google Scholar
  8. 8.
    Herrmann, D., Wendolsky, R., Federrath, H.: Website fingerprinting: attacking popular privacy enhancing technologies with the multinomial naive-bayes classifier. In: Proceedings of the 2009 ACM Workshop on Cloud Computing Security, CCSW 2009, pp. 31–42. ACM, New York (2009)Google Scholar
  9. 9.
    Panchenko, A., Niessen, L., Zinnen, A., Engel, T.: Website fingerprinting in onion routing based anonymization networks. In: Proceedings of the 10th Annual ACM Workshop on Privacy in the Electronic Society, WPES 2011, pp. 103–114. ACM, New York (2011)Google Scholar
  10. 10.
    Cai, X., Zhang, X.C., Joshi, B., Johnson, R.: Touching from a distance: website fingerprinting attacks and defenses. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security, CCS 2012, pp. 605–616. ACM, New York (2012)Google Scholar
  11. 11.
    Wang, T., Goldberg, I.: Improved website fingerprinting on tor. In: 12th ACM Workshop on Privacy in the Electronic Society, WPES 2013. ACM (2013)Google Scholar
  12. 12.
    Dingledine, R., Mathewson, N., Syverson, P.: Tor : the second-generation onion router. In: Proceedings of the 13th Usenix Security Symposium, August 2004Google Scholar
  13. 13.
  14. 14.
    Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  15. 15.
    Durbin, R., Eddy, S.: Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press, Cambridge (1998)CrossRefzbMATHGoogle Scholar
  16. 16.
    Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1137–1143 (1995)Google Scholar

Copyright information

© Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2015

Authors and Affiliations

  • Sami Zhioua
    • 1
    Email author
  • Adnene Ben Jabeur
    • 2
  • Mahjoub Langar
    • 3
  • Wael Ilahi
    • 3
  1. 1.King Fahd University of Petroleum and MineralsDhahranSaudi Arabia
  2. 2.École PolytechniqueLa MarsaTunisia
  3. 3.École Nationale des Ingénieurs de TunisTunisTunisia

Personalised recommendations