Anomaly Detection from Network Logs Using Diffusion Maps

  • Tuomo Sipola
  • Antti Juvonen
  • Joel Lehtonen
Conference paper
Part of the IFIP Advances in Information and Communication Technology book series (IFIPAICT, volume 363)


The goal of this study is to detect anomalous queries from network logs using a dimensionality reduction framework. The fequencies of 2-grams in queries are extracted to a feature matrix. Dimensionality reduction is done by applying diffusion maps. The method is adaptive and thus does not need training before analysis. We tested the method with data that includes normal and intrusive traffic to a web server. This approach finds all intrusions in the dataset.


intrusion detection anomaly detection n-grams diffusion map data mining machine learning 


  1. 1.
    Bengio, Y., Delalleau, O., Roux, N.L., Paiement, J.F., Vincent, P., Ouimet, M.: Spectral Dimensionality Reduction. In: Feature Extraction. Studies in Fuzziness and Soft Computing, pp. 519–550. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: A survey. ACM Comput. Surv. 41(3), 1–58 (2009)CrossRefGoogle Scholar
  3. 3.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001),
  4. 4.
    Chung, F.R.K.: Spectral Graph Theory, p. 2. AMS Press, Providence (1997)zbMATHGoogle Scholar
  5. 5.
    Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. Proceedings of the National Academy of Sciences of the United States of America 102, 7426 (2005)CrossRefGoogle Scholar
  6. 6.
    Coifman, R.R., Lafon, S.: Diffusion maps. Applied and Computational Harmonic Analysis 21(1), 5–30 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  7. 7.
    Damashek, M.: Gauging similarity with n-grams: Language-independent categorization of text. Science 267(5199), 843 (1995)CrossRefGoogle Scholar
  8. 8.
    David, G.: Anomaly Detection and Classification via Diffusion Processes in Hyper-Networks. Ph.D. thesis, Tel-Aviv University (2009)Google Scholar
  9. 9.
    Han, J., Kamber, M.: Data mining: concepts and techniques. Morgan Kaufmann, San Francisco (2006)Google Scholar
  10. 10.
    İzmirli, Ö.: Tonal-atonal classification of music audio using diffusion maps. In: 10th International Society for Music Information Retrieval Conference (ISMIR 2009) (2009)Google Scholar
  11. 11.
    Kannan, R., Vempala, S., Vetta, A.: On clusterings: Good, bad and spectral. J. ACM 51, 497–515 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  12. 12.
    Keller, Y., Coifman, R., Lafon, S., Zucker, S.: Audio-visual group recognition using diffusion maps. IEEE Transactions on Signal Processing 58(1), 403–413 (2010)MathSciNetCrossRefGoogle Scholar
  13. 13.
    von Luxburg, U.: A tutorial on spectral clustering. Statistics and Computing 17, 395–416 (2007)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Meila, M., Shi, J.: Learning segmentation by random walks. In: NIPS, pp. 873–879 (2000)Google Scholar
  15. 15.
    Mukkamala, S., Sung, A.: A comparative study of techniques for intrusion detection (2003)Google Scholar
  16. 16.
    Nadler, B., Lafon, S., Coifman, R., Kevrekidis, I.G.: Diffusion maps – a probabilistic interpretation for spectral embedding and clustering algorithms. In: Barth, T.J., Griebel, M., Keyes, D.E., Nieminen, R.M., Roose, D., Schlick, T., Gorban, A.N., Kégl, B., Wunsch, D.C., Zinovyev, A.Y. (eds.) Principal Manifolds for Data Visualization and Dimension Reduction. Lecture Notes in Computational Science and Engineering, vol. 58, pp. 238–260. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: Analysis and an algorithm. In: Advances in Neural Information Processing Systems, vol. 14, pp. 849–856. MIT Press, Cambridge (2001)Google Scholar
  18. 18.
    Nguyen-Tuong, A., Guarnieri, S., Greene, D., Shirley, J., Evans, D.: Automatically hardening web applications using precise tainting. In: Sasaki, R., Qing, S., Okamoto, E., Yoshiura, H. (eds.) Security and Privacy in the Age of Ubiquitous Computing. IFIP AICT, vol. 181, pp. 295–307. Springer, Boston (2005)CrossRefGoogle Scholar
  19. 19.
    Patcha, A., Park, J.: An overview of anomaly detection techniques: Existing solutions and latest technological trends. Computer Networks 51(12), 3448–3470 (2007)CrossRefGoogle Scholar
  20. 20.
    Ramadas, M., Ostermann, S., Tjaden, B.: Detecting anomalous network traffic with self-organizing maps. In: Vigna, G., Krügel, C., Jonsson, E. (eds.) RAID 2003. LNCS, vol. 2820, pp. 36–54. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  21. 21.
    Schclar, A., Averbuch, A., Rabin, N., Zheludev, V., Hochman, K.: A diffusion framework for detection of moving vehicles. Digital Signal Processing 20(1), 111–122 (2010)CrossRefGoogle Scholar
  22. 22.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(8), 888–905 (2000)CrossRefGoogle Scholar
  23. 23.
    Tran, Q., Duan, H., Li, X.: One-class support vector machine for anomaly network traffic detection. China Education and Research Network (CERNET) (2004)Google Scholar
  24. 24.
    Tran, Q.A., Zhang, Q., Li, X.: Evolving training model method for one-class svm. In: IEEE International Conference on Systems, Man and Cybernetics, vol. 3, pp. 2388–2393 (2003)Google Scholar
  25. 25.
    Turkka, J., Ristaniemi, T., David, G., Averbuch, A.: Anomaly detection framework for tracing problems in radio networks. In: Proc. to ICN 2011 (2011)Google Scholar

Copyright information

© International Federation for Information Processing 2011

Authors and Affiliations

  • Tuomo Sipola
    • 1
  • Antti Juvonen
    • 1
  • Joel Lehtonen
    • 1
  1. 1.Department of Mathematical Information TechnologyUniversity of JyväskyläFinland

Personalised recommendations