Skip to main content
SpringerLink
Log in

Anomaly Detection in Distributed Streams

  • Conference paper
  • First Online:
Information and Communication Technology for Intelligent Systems ( ICTIS 2020)

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 196))

  • 1148 Accesses

Abstract

Anomalies refer to any non-conforming patterns to the expected behavior in the system. The detection of anomaly in real time from logs arriving at very high velocity and are in huge volume requires a distributed framework with high throughput and low latency. In this research, statistical method has been implemented for finding the suspicious associations in Spark Streaming, a highly scalable distributed and streaming framework. The models were deployed in both local mode as well as in cluster mode to perform anomaly detection on server logs.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Kline, K., Nam, S., Barford, P., Plonka, A., Ron, A.: Traffic anomaly detection at fine time scales with Bayes. In: International Conference on Internet Monitoring and Protection (2008)

    Google Scholar 

  2. Mukkamala, S., Sung, A., Ribeiro, B.: Model selection for kernel based intrusion detection systems. In: Proceedings of International Conference on Adaptive and Natural Computing Algorithm (2005)

    Google Scholar 

  3. Manocha, S., Girolami, M.: An empirical analysis of the probabilistic k-nearest neighbor classifier. Pattern Recogn. Lett. 28, 1818–1824 (2007)

    Article  Google Scholar 

  4. Angiulli, F., Pizzuti, C.: Fast outlier detection in high dimen-sional spaces. Lecture Notes Comput. Sci. 2431, 43–78 (2002)

    MATH  Google Scholar 

  5. Ramaswamy, S., Rastogi, R., Shim, K.: Efficient algorithms for mining outliers from large data sets. In: Proceedings of the 2000 ACM SIGMOD, pp. 427–438 (2000)

    Google Scholar 

  6. Alnafessah, A., Casale, G.: Artificial neural networks based techniques for anomaly detection in apache spark (2019)

    Google Scholar 

  7. Duda, R.O., Hart, P., Stork, D.: Pattern Classification, 2nd edn (2001)

    Google Scholar 

  8. Rawat, S.: Efficient data mining algorithms for intrusion detection. In: Proceedings of the 4th Conference on Engineering of Intelligent Systems (EIS’04) (2005)

    Google Scholar 

  9. Patcha, A., Park, J.-M.: An overview of anomaly detection techniques: Existing solutions and latest technological trends. Comput. Networks 51, 3448–3470 (2007)

    Article  Google Scholar 

  10. Chandola, V., Banarjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Survey J. 41 (2009)

    Google Scholar 

  11. Gu, Y., McCallum, A., Towsley, D.: Detecting anomalies in network traffic using maximum entropy estimation. In: Proceedings of IMC (2005)

    Google Scholar 

  12. Lee, W., Xiang, D.: Information-theoretic measures for anomaly detection. In: Proceedings of IEEE Symposium on Security and Privacy (2001)

    Google Scholar 

  13. Rettig, S., Khayati, M.: Online anomaly detection over big data streams (2015)

    Google Scholar 

  14. Ahmad, S., Lavin, A., Purdy, S., Agha, Z.: Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262, 134–147 (2017)

    Article  Google Scholar 

  15. Otey, M., Ghoting, A., Parthasarathy, S.: Fast distributed outlier detection in mixed-attribute data sets. Data Mining Knowl. Discovery 12, 203–228 (2006)

    Article  MathSciNet  Google Scholar 

  16. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surveys 41, 1–58 (2009)

    Google Scholar 

  17. Solaimani, M., Iftekhar, M., Khan, L., Thuraisingham, B.: Statistical technique for online anomaly detection using spark over heterogeneous data from multi-source vmware performance data. In: IEEE International Conference on Big Data (2014)

    Google Scholar 

  18. Dunning, T.: Better anomaly detection with the t-digest (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LogPoint, Kathmandu, Nepal

    Rupesh Karn, Suvrat Ram Joshi & Umanga Bista

  2. Institute of Engineering, Tribhuvan University, Kathmandu, Nepal

    Basanta Joshi, Daya Sagar Baral & Aman Shakya

Authors
  1. Rupesh Karn
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Suvrat Ram Joshi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umanga Bista
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Basanta Joshi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Daya Sagar Baral
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Aman Shakya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Basanta Joshi .

Editor information

Editors and Affiliations

  1. Department of Electrical and Electronics Engineering, University of the Ryukyus, Nishihara, Japan

    Tomonobu Senjyu

  2. Sinhgad Technical Education Society, SKNCOE, Pune, India

    Parikshit N. Mahalle

  3. Universiti Putra Malaysia, Serdang, Malaysia

    Thinagaran Perumal

  4. Global Knowledge Research Foundation, Udaipur, Rajasthan, India

    Amit Joshi

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Karn, R., Joshi, S.R., Bista, U., Joshi, B., Baral, D.S., Shakya, A. (2021). Anomaly Detection in Distributed Streams. In: Senjyu, T., Mahalle, P.N., Perumal, T., Joshi, A. (eds) Information and Communication Technology for Intelligent Systems. ICTIS 2020. Smart Innovation, Systems and Technologies, vol 196. Springer, Singapore. https://doi.org/10.1007/978-981-15-7062-9_14

Download citation

Publish with us

Policies and ethics

Search

Navigation