Skip to main content
SpringerLink
Log in

Survey: Intrusion Detection Systems in Encrypted Traffic

  • Conference paper
  • First Online:
Internet of Things, Smart Spaces, and Next Generation Networks and Systems (ruSMART 2016, NEW2AN 2016)

Part of the book series: Lecture Notes in Computer Science ((LNCCN,volume 9870))

Abstract

Intrusion detection system, IDS, traditionally inspects the payload information of packets. This approach is not valid in encrypted traffic as the payload information is not available. There are two approaches, with different detection capabilities, to overcome the challenges of encryption: traffic decryption or traffic analysis. This paper presents a comprehensive survey of the research related to the IDSs in encrypted traffic. The focus is on traffic analysis, which does not need traffic decryption. One of the major limitations of the surveyed researches is that most of them are concentrating in detecting the same limited type of attacks, such as brute force or scanning attacks. Both the security enhancements to be derived from using the IDS and the security challenges introduced by the encrypted traffic are discussed. By categorizing the existing work, a set of conclusions and proposals for future research directions are presented.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Koch, R.: Towards next-generation intrusion detection. In: 2011 3rd International Conference on Cyber Conflict (ICCC), pp. 1–18 (2011)

    Google Scholar 

  2. Barati, M., Abdullah, A., Mahmod, R., Mustapha, N., Udzir, N.I.: Feature selection for IDS in encrypted traffic using genetic algorithm. In: Proceedings of the 4th International Conference on Computing and Informatics, (ICCI 2013), pp. 279–285 (2013)

    Google Scholar 

  3. Dyer, K.P., Coull, S.E., Ristenpart, T., Shrimpton, T.: Peek-a-Boo, i still see you: why efficient traffic analysis countermeasures fail. In: 2012 IEEE Symposium on Security and Privacy (SP), pp. 332–346 (2012)

    Google Scholar 

  4. Sperotto, A., Schaffrath, G., Sadre, R., Morariu, C., Pras, A., Stiller, B.: An overview of IP flow-based intrusion detection. IEEE Commun. Surv. Tutor. 12(3), 343–356 (2010)

    Article  Google Scholar 

  5. Engen, V.: Machine learning for network based intrusion detection. Bournemouth University (2010)

    Google Scholar 

  6. Paradis, J.G., Zimmerman, M.L.: The MIT Guide to Science and Engineering Communication. MIT Press, Cambridge (2002)

    Google Scholar 

  7. Liberatore, M., Levine, B.N.: Inferring the source of encrypted HTTP connections. In: Proceedings of the 13th ACM Conference on Computer and Communications Security, pp. 255–263 (2006)

    Google Scholar 

  8. Hintz, A.: Fingerprinting websites using traffic analysis. In: Dingledine, R., Syverson, P.F. (eds.) PET 2002. LNCS, vol. 2482, pp. 171–178. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  9. Bissias, G.D., Liberatore, M., Jensen, D., Levine, B.N.: Privacy vulnerabilities in encrypted HTTP streams. In: Danezis, G., Martin, D. (eds.) PET 2005. LNCS, vol. 3856, pp. 1–11. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  10. Augustin, M., Balaz, A.: Intrusion detection with early recognition of encrypted application. In: 2011 15th IEEE International Conference on Intelligent Engineering Systems (INES), pp. 245–247 (2011)

    Google Scholar 

  11. Raymond, J.-F.: Traffic analysis: protocols, attacks, design issues, and open problems. In: Federrath, H. (ed.) Designing Privacy Enhancing Technologies. LNCS, vol. 2009, pp. 10–29. Springer, Heidelberg (2001)

    Chapter  Google Scholar 

  12. Alshammari, R., Lichodzijewski, P.I., Heywood, M., Zincir-Heywood, A.N.: Classifying SSH encrypted traffic with minimum packet header features using genetic programming. In: Proceedings of the 11th Annual Conference Companion on Genetic and Evolutionary Computation Conference: Late Breaking Papers, New York, NY, USA, pp. 2539–2546 (2009)

    Google Scholar 

  13. Alshammari, R., Zincir-Heywood, A.N.: A flow based approach for SSH traffic detection. In: IEEE International Conference on Systems, Man and Cybernetics, ISIC, pp. 296–301 (2007)

    Google Scholar 

  14. Alshammari, R., Zincir-Heywood, A.N.: Investigating two different approaches for encrypted traffic classification. In: Sixth Annual Conference on Privacy, Security and Trust, PST 2008, pp. 156–166 (2008)

    Google Scholar 

  15. Alshammari, R., Zincir-Heywood, A.N.: Machine learning based encrypted traffic classification: identifying SSH and skype. In: IEEE Symposium on Computational Intelligence for Security and Defense Applications, CISDA 2009, pp. 1–8 (2009)

    Google Scholar 

  16. Alshammari, R., Zincir-Heywood, A.N.: Can encrypted traffic be identified without port numbers, IP addresses and payload inspection? Comput. Netw. 55(6), 1326–1350 (2011)

    Article  Google Scholar 

  17. Arndt, D.J., Zincir-Heywood, A.N.: A comparison of three machine learning techniques for encrypted network traffic analysis. In: 2011 IEEE Symposium on Computational Intelligence for Security and Defense Applications (CISDA), pp. 107–114 (2011)

    Google Scholar 

  18. Bacquet, C., Gumus, K., Tizer, D., Zincir-Heywood, A.N., Heywood, M.I.: A comparison of unsupervised learning techniques for encrypted traffic identification. J. Inf. Assur. Secur. 5, 464–472 (2010)

    Google Scholar 

  19. Cao, Z., Cao, S., Xiong, G., Guo, L.: Progress in study of encrypted traffic classification. In: Yuan, Y., Wu, X., Lu, Y. (eds.) Trustworthy Computing and Services, pp. 78–86. Springer, Berlin Heidelberg (2012)

    Google Scholar 

  20. Erman, J., Mahanti, A., Arlitt, M., Cohen, I., Williamson, C.: Offline/realtime traffic classification using semi-supervised learning. Perform. Eval. 64(9–12), 1194–1213 (2007)

    Article  Google Scholar 

  21. Maiolini, G., Baiocchi, A., Rizzi, A., Di Iollo, C.: Statistical classification of services tunneled into SSH connections by a K-means based learning algorithm. In: Proceedings of the 6th International Wireless Communications and Mobile Computing Conference, New York, NY, USA, pp. 742–746 (2010)

    Google Scholar 

  22. Abimbola, A.A., Munoz, J.M., Buchanan, W.J.: NetHost-Sensor: investigating the capture of end-to-end encrypted intrusive data. Comput. Secur. 25(6), 445–451 (2006)

    Article  Google Scholar 

  23. Kilic, F., et al.: iDeFEND: intrusion detection framework for encrypted network data. In: Reiter, M. (ed.) CANS 2015. LNCS, vol. 9476, pp. 111–118. Springer, Heidelberg (2015). doi:10.1007/978-3-319-26823-1_8

    Chapter  Google Scholar 

  24. Goh, V.T., Zimmermann, J., Looi, M.: Towards intrusion detection for encrypted networks. In: International Conference on Availability, Reliability and Security, ARES 2009, pp. 540–545 (2009)

    Google Scholar 

  25. Goh, V.T., Zimmermann, J., Looi, M.: Experimenting with an intrusion detection system for encrypted networks. Int. J. Bus. Intell. Data Min. 5(2), 172–191 (2010)

    Article  Google Scholar 

  26. Goh, V.T., Zimmermann, J., Looi, M.: Intrusion detection system for encrypted networks using secret-sharing schemes. In: International Journal of Cryptology Research, Hotel Equatorial, Melaka, Malaysia (2010)

    Google Scholar 

  27. Hellemons, L., Hendriks, L., Hofstede, R., Sperotto, A., Sadre, R., Pras, A.: SSHCure: a flow-based SSH intrusion detection system. In: Sadre, R., Novotný, J., Čeleda, P., Waldburger, M., Stiller, B. (eds.) AIMS 2012. LNCS, vol. 7279, pp. 86–97. Springer, Heidelberg (2012)

    Chapter  Google Scholar 

  28. Amoli, P.V., Hämäläinen, T.: A real time unsupervised NIDS for detecting unknown and encrypted network attacks in high speed network. In: 2013 IEEE International Workshop on Measurements and Networking Proceedings (M N), pp. 149–154 (2013)

    Google Scholar 

  29. Amoli, P.V., Hämäläinen, T., David, G., Zolotukhin, M., Mirzamohammad, M.: Unsupervised network intrusion detection systems for zero-day fast-spreading attacks and botnets. Int. J. Digit. Content Technol. Its Appl. 10(2), 1–13 (2016)

    Google Scholar 

  30. Joglekar, S.P., Tate, S.R.: ProtoMon: embedded monitors for cryptographic protocol intrusion detection and prevention. In: Proceedings of the International Conference on Information Technology: Coding and Computing, ITCC 2004, vol. 1, pp. 81–88 (2004)

    Google Scholar 

  31. Yamada, A., Miyake, Y., Takemori, K., Studer, A., Perrig, A.: Intrusion detection for encrypted web accesses. In: 21st International Conference on Advanced Information Networking and Applications Workshops, AINAW 2007, vol. 1, pp. 569–576 (2007)

    Google Scholar 

  32. Foroushani, V.A., Adibnia, F., Hojati, E.: Intrusion detection in encrypted accesses with SSH protocol to network public servers. In: International Conference on Computer and Communication Engineering, ICCCE 2008, pp. 314–318 (2008)

    Google Scholar 

  33. Koch, R., Rodosek, G.D.: Command evaluation in encrypted remote sessions. In: 2010 4th International Conference on Network and System Security (NSS), pp. 299–305 (2010)

    Google Scholar 

  34. Koch, R., Rodosek, G.D.: Security system for encrypted environments (S2E2). In: Jha, S., Sommer, R., Kreibich, C. (eds.) RAID 2010. LNCS, vol. 6307, pp. 505–507. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  35. Barati, M., Abdullah, A., Udzir, N., Behzadi, M., Mahmod, R., Mustapha, N.: Intrusion detection system in secure shell traffic in cloud environment. J. Comput. Sci. 10(10), 2029 (2014)

    Article  Google Scholar 

  36. Koch, R., Golling, M., Rodosek, G.D.: Behavior-based intrusion detection in encrypted environments. Commun. Mag. IEEE 52(7), 124–131 (2014)

    Article  Google Scholar 

  37. Zolotukhin, M., Hämäläinen, T., Kokkonen, T., Niemelä, A., Siltanen, J.: Data mining approach for detection of DDoS attacks utilizing SSL/TLS protocol. In: Balandin, S., Andreev, S., Koucheryavy, Y. (eds.) NEW2AN/ruSMART 2015. LNCS, vol. 9247, pp. 274–285. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  38. McHugh, J.: Testing intrusion detection systems: a critique of the 1998 and 1999 DARPA intrusion detection system evaluations as performed by lincoln laboratory. ACM Trans. Inf. Syst. Secur. 3(4), 262–294 (2000)

    Article  Google Scholar 

  39. Mahoney, M.V., Chan, P.K.: An analysis of the 1999 DARPA/Lincoln laboratory evaluation data for network anomaly detection. In: Vigna, G., Kruegel, C., Jonsson, E. (eds.) RAID 2003. LNCS, vol. 2820, pp. 220–237. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Jyväskylä, Jyväskylä, Finland

    Tiina Kovanen, Gil David & Timo Hämäläinen

Authors
  1. Tiina Kovanen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gil David
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Timo Hämäläinen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tiina Kovanen .

Editor information

Editors and Affiliations

  1. Tampere University of Technology , Tampere, Finland

    Olga Galinina

  2. FRUCT Oy , Helsinki, Finland

    Sergey Balandin

  3. Tampere University of Technology , Tampere, Finland

    Yevgeni Koucheryavy

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing AG

About this paper

Cite this paper

Kovanen, T., David, G., Hämäläinen, T. (2016). Survey: Intrusion Detection Systems in Encrypted Traffic. In: Galinina, O., Balandin, S., Koucheryavy, Y. (eds) Internet of Things, Smart Spaces, and Next Generation Networks and Systems. ruSMART NEW2AN 2016 2016. Lecture Notes in Computer Science(), vol 9870. Springer, Cham. https://doi.org/10.1007/978-3-319-46301-8_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-46301-8_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-46300-1

  • Online ISBN: 978-3-319-46301-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation