East European Conference on Advances in Databases and Information Systems

ADBIS 2015: New Trends in Databases and Information Systems pp 197-206 | Cite as

Unsupervised Network Anomaly Detection in Real-Time on Big Data

  • Juliette Dromard
  • Gilles Roudière
  • Philippe Owezarski
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 539)

Abstract

Network anomaly detection relies on intrusion detection systems based on knowledge databases. However, building this knowledge may take time as it requires manual inspection of experts. Actual detection systems are unable to deal with 0-day attack or new user’s behavior and in consequence they may fail in correctly detecting intrusions. Unsupervised network anomaly detectors overcome this issue as no previous knowledge is required. In counterpart, these systems may be very slow as they need to learn traffic’s pattern in order to acquire the necessary knowledge to detect anomalous flows. To improve speed, these systems are often only exposed to sampled traffic, harmful traffic may then avoid the detector examination. In this paper, we propose to take advantage of new distributed computing framework in order to speed up an Unsupervised Network Anomaly Detector Algorithm, UNADA. The evaluation shows that the execution time can be improved by a factor of 13 allowing UNADA to process large traces of traffic in real time.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Apache spark - lightning-fast cluster computing. https://spark.apache.org/ (accessed April 29, 2015)
  2. 2.
    Grid5000. https://www.grid5000.fr (accessed April 29, 2015)
  3. 3.
    Andrade, G., Ramos, G., Madeira, D., Sachetto, R., Ferreira, R., Rocha, L.: G-dbscan: A GPU accelerated algorithm for density-based clustering. Procedia Computer Science, 369–378 (2013)Google Scholar
  4. 4.
    Brauckhoff, D., Tellenbach, B., Wagner, A., May, M., Lakhina, A.: Impact of packet sampling on anomaly detection metrics. In: Proc. of the 6th ACM SIGCOMM Conference on Internet Measurement, pp. 159–164 (2006)Google Scholar
  5. 5.
    Casas, P., Mazel, J., Owezarski, P.: Unsupervised network intrusion detection systems: Detecting the unknown without knowledge. Computer Communications, 772–783 (2012)Google Scholar
  6. 6.
    Celenk, M., Conley, T., Willis, J., Graham, J.: Anomaly detection and visualization using fisher discriminant clustering of network entropy. In: Third International Conference on Digital Information Management, pp. 216–220, November 2008Google Scholar
  7. 7.
    Dewaele, G., Fukuda, K., Borgnat, P., Abry, P., Cho, K.: Extracting hidden anomalies using sketch and non gaussian multiresolution statistical detection procedures. In: Proc. of the 2007 Workshop on Large Scale Attack Defense, pp. 145–152. ACM (2007)Google Scholar
  8. 8.
    Ester, M., peter Kriegel, H., S, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise, pp. 226–231. AAAI Press (1996)Google Scholar
  9. 9.
    Ester, M., Kriegel, H.P., Sander, J., Wimmer, M., Xu, X.: Incremental clustering for mining in a data warehousing environment. In: Proc. of the 24rd International Conference on Very Large Data Bases, pp. 323–333 (1998)Google Scholar
  10. 10.
    Fahad, A., Alshatri, N., Tari, Z., Alamri, A., Khalil, I., Zomaya, A., Foufou, S., Bouras, A.: A survey of clustering algorithms for big data: Taxonomy and empirical analysis. IEEE Transactions on Emerging Topics in Computing, 267–279, September 2014Google Scholar
  11. 11.
    Fontugne, R., Mazel, J., Fukuda, K.: Hashdoop: a mapreduce framework for network anomaly detection. In: INFOCOM WKSHPS, pp. 494–499, April 2014Google Scholar
  12. 12.
    Fontugne, R., Fukuda, K.: A hough-transform-based anomaly detector with an adaptive time interval. SIGAPP Appl. Comput. Rev., 41–51 (2011)Google Scholar
  13. 13.
    Gu, Y., McCallum, A., Towsley, D.: Detecting anomalies in network traffic using maximum entropy estimation. In: Proc. of the 5th ACM SIGCOMM Conference on Internet Measurement, pp. 32–32 (2005)Google Scholar
  14. 14.
    Kanda, Y., Fukuda, K., Sugawara, T.: Evaluation of anomaly detection based on sketch and pca. In: GLOBECOM 2010, pp. 1–5. IEEE (2010)Google Scholar
  15. 15.
    Kriegel, H.P., Kroger, P., Zimek, A.: Clustering high-dimensional data: A survey on subspace clustering, pattern-based clustering, and correlation clustering. ACM Trans. Knowl. Discov. Data (2009)Google Scholar
  16. 16.
    Lakhina, A., Crovella, M., Diot, C.: Diagnosing network-wide traffic anomalies. In: Proc. of ACM SIGCOMM 2004, pp. 219–230, Auguest 2004Google Scholar
  17. 17.
    Patcha, A., Park, J.M.: An overview of anomaly detection techniques: Existing solutions and latest technological trends. Comput. Netw., 3448–3470 (2007)Google Scholar
  18. 18.
    Portnoy, L., Eskin, E., Stolfo, S.: Intrusion detection with unlabeled data using clustering. In: Proc. of ACM CSS Workshop on Data Mining Applied to Security, pp. 5–8 (2001)Google Scholar
  19. 19.
    Wei, X., Huang, H., Tian, S.: A grid-based clustering algorithm for network anomaly detection. In: The First International Symposium on Data, Privacy, and E-Commerce, ISDPE 2007, pp. 104–106, November 2007Google Scholar
  20. 20.
    Xin, R.S., Rosen, J., Zaharia, M., Franklin, M.J., Shenker, S., Stoica, I.: Shark: SQL and rich analytics at scale. In: Proc. of the 2013 ACM SIGMOD International Conference on Management of Data, pp. 13–24 (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Juliette Dromard
    • 1
  • Gilles Roudière
    • 1
  • Philippe Owezarski
    • 1
  1. 1.CNRS, LAASToulouse Cedex 4France

Personalised recommendations