Skip to main content
SpringerLink
Log in

Real-Time Anomaly Detection from Environmental Data Streams

  • Chapter
  • First Online:
Book cover AGILE 2015

Part of the book series: Lecture Notes in Geoinformation and Cartography ((LNGC))

Abstract

Modern sensor networks monitor a wide range of phenomena. They are applied in environmental monitoring, health care, optimization of industrial processes, social media, smart city solutions, and many other domains. All in all, they provide a continuously pulse of the almost infinite activities that are happening in the physical space—and in cyber space. The handling of the massive amounts of generated measurements poses a series of (Big Data) challenges. Our work addresses one of these challenges: the detection of anomalies in real-time. In this paper, we propose a generic solution to this problem, and introduce a system that is capable of detecting anomalies, generating notifications, and displaying the recent situation to the user. We apply CUSUM a statistical control algorithm and adopt it so that it can be used inside the Storm framework—a robust and scalable real-time processing framework. We present a proof of concept implementation from the area of environmental monitoring.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Storm, Apache Incubator, http://storm.apache.org.

  2. 2.

    Apache S4, Apache Incubator, http://incubator.apache.org/s4.

  3. 3.

    Samza, Apache Incubator, http://samza.incubator.apache.org.

  4. 4.

    Spark Streaming, Apache Incubator, http://spark.apache.org/streaming.

  5. 5.

    Storm Trident, Apache Incubator, http://storm.apache.org/documentation/Trident-API-Overview.html.

  6. 6.

    ActiveMQ framework, Apache Software Foundation, http://activemq.apache.org.

  7. 7.

    OpenWire protocol, Apache Incubator, http://activemq.apache.org/openwire.html.

  8. 8.

    STOMP protocol, Apache Incubator, http://activemq.apache.org/stomp.html.

  9. 9.

    http://www.citma.gva.es/web/calidad-ambiental/datos-on-line.

  10. 10.

    Leaflet: An Open-Source JavaScript Library for Mobile-Friendly Interactive Maps, http://leafletjs.com.

  11. 11.

    ESRI, http://www.esri.com.

  12. 12.

    Bootstrap, Twitter http://getbootstrap.com.

  13. 13.

    jQuery, jQuery Foundation http://jquery.com.

  14. 14.

    Highcharts JS, Highcharts AS http://www.highcharts.com.

References

  • Barratt, B., & Fuller, G. (2014). Intervention assessments in the control of PM10 emissions from an urban waste transfer station. Environmental Science: Processes and Impacts, 16(6), 1328–1337.

    Google Scholar 

  • Barratt, B., Atkinson, R., Anderson, H., Beevers, S., Kelly, F., Mudway, I., & Wilkinson, P. (2007). Investigation into the use of the cusum technique in identifying changes in mean air pollution levels following introduction of a traffic management scheme. Atmospheric Environment, 41(8), 1784–1791.

    Article  Google Scholar 

  • Buschmann, F., Meunier, R., Rohnert, H., & Sommerlad, P. (1996). Pattern-oriented software architecture: A system of patterns (Vol. 1). New York: Wiley.

    Google Scholar 

  • Carslaw, D., Ropkins, K., & Bell, M. (2006). Change-point detection of gaseous and particulate traffic-related pollutants at a roadside location. Environmental Science and Technology, 40(22), 6912–6918.

    Article  Google Scholar 

  • Charles, J., & Jeh-Nan, P. (2002). Evaluating environmental performance using statistical process control techniques. European Journal of Operational Research, 139(1), 68–83.

    Article  Google Scholar 

  • Chelani, A. (2011). Change detection using cusum and modified cusum method in air pollutant concentrations at traffic site in Delhi. Stochastic Environmental Research and Risk Assessment, 25(6), 827–834.

    Article  Google Scholar 

  • Chuen-Sheng, C. (1995). A multi-layer neural network model for detecting changes in the process mean. Computers and Industrial Engineering, 28(1), 51–61.

    Article  Google Scholar 

  • Cox, S. (2007). Observations and measurements part 1—observation schema. Technical report, Open Geospatial Consortium (OGC).

    Google Scholar 

  • De Francisci Morales, G. (2013). Samoa: A platform for mining big data streams. In Proceedings of the 22nd International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, WWW ’13 Companion (pp. 777–778).

    Google Scholar 

  • Fielding, R. (2000). Representational state transfer (rest) (Chap. 5). Fielding Dissertation.

    Google Scholar 

  • Grigg, O., Farewell, V., & Spiegelhalter, D. (2003). Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts. Statistical Methods in Medical Research, 12(2), 147–170.

    Google Scholar 

  • Guh, R., & Hsieh, Y. (1999). A neural network based model for abnormal pattern recognition of control charts. Computers and Industrial Engineering, 36(1), 97–108.

    Article  Google Scholar 

  • Hapner, M., Burridge, R., Sharma, R., Fialli, J., & Stout, K. (2002). Java message service. Santa Clara, CA: Sun Microsystems Inc.

    Google Scholar 

  • Hickson, I. (2011). The WebSocket API. W3C Working Draft WD-websockets-20110929, September.

    Google Scholar 

  • Jeske, D., Montes De Oca, V., Bischoff, W., & Marvasti, M. (2009). Cusum techniques for timeslot sequences with applications to network surveillance. Computational Statistics Data Analysis, 53(12), 4332–4344.

    Article  Google Scholar 

  • Kortuem, G., Kawsar, F., Fitton, D., & Sundramoorthy, V. (2010). Smart objects as building blocks for the internet of things. IEEE Internet Computing, 14(1), 44–51.

    Article  Google Scholar 

  • Lucas, J. (1982). Combined Shewhart-CUSUM quality control schemes. Journal of Quality Technology, 14(2).

    Google Scholar 

  • Manovich, L. (2012). Trending: The promises and the challenges of big social data. In M. K. Gold (Ed.), Debates in the digital humanities (pp. 460–475). Minneapolis: U of Minnesota P.

    Google Scholar 

  • Mesnil, B., & Petitgas, P. (2009). Detection of changes in time-series of indicators using cusum control charts. Aquatic Living Resources, 22(02), 187–192.

    Google Scholar 

  • Morris, S., & Paradiso, J. (2002). Shoe-integrated sensor system for wireless gait analysis and real-time feedback. In Engineering in Medicine and Biology, 2002. 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES Conference, 2002. Proceedings of the Second Joint, IEEE (Vol. 3, pp. 2468–2469).

    Google Scholar 

  • Nativi, S., Craglia, M., & Pearlman, J. (2012). The brokering approach for multidisciplinary interoperability: A position paper. International Journal of Spatial Data Infrastructures Research, 7, 1–15.

    Google Scholar 

  • Osanaiye, P., & Talabi, C. (1989). On some non-manufacturing applications of counted data cumulative sum (CUSUM) control chart schemes. The Statistician, 38(4), 251–257.

    Google Scholar 

  • Page, E. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100–115.

    Article  Google Scholar 

  • Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop distributed file system. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), IEEE (pp. 1–10).

    Google Scholar 

  • Simoncelli, D., Dusi, M., Gringoli, R., & Niccolini, S. (2013). Stream-monitoring with blockmon: Convergence of network measurements and data analytics platforms. SIGCOMM Computer Communication Review, 43(2), 29–36.

    Article  Google Scholar 

  • Sitaram, D., Srinivasaraghavan, H., Agarwal, K., Agrawal, A., Joshi, N., & Ray, D. (2013). Pipelining acoustic model training for speech recognition using storm. In 2013 Fifth International Conference on Computational Intelligence, Modelling and Simulation (CIMSim) (pp. 219–224).

    Google Scholar 

  • Sunderrajan, A., Aydt, H., & Knoll, A. (2014). Real time load prediction and outliers detection using storm. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, ACM, New York, NY, USA, DEBS ’14 (pp. 294–297).

    Google Scholar 

  • Trilles, S., Belmonte, O., Diaz, L., & Huerta, J. (2014). Mobile access to sensor networks by using GIS standards and restful services. IEEE Sensors Journal, 14(12), 4143–4153.

    Article  Google Scholar 

  • Yutan, D., Jun, L., Fang, L., & Luying, C. (2014). A real-time anomalies detection system based on streaming technology. In 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 275–279).

    Google Scholar 

Download references

Acknowledgments

This work has been supported in part by European Commission and Generalitat Valenciana government (grants ACIF/2012/112 and BEFPI/2014/067).

Author information

Authors and Affiliations

  1. Institute of New Imaging Technologies, Universitat Jaume I, Castellón de la Plana, Spain

    Sergio Trilles, Óscar Belmonte & Joaquín Huerta

  2. Institute for Environment and Sustainability, European Commission—Joint Research Centre, Ispra, Italy

    Sven Schade

Authors
  1. Sergio Trilles
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven Schade
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Óscar Belmonte
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joaquín Huerta
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sergio Trilles .

Editor information

Editors and Affiliations

  1. NOVA IMS, Universidade Nova de Lisboa, Lisboa, Portugal

    Fernando Bacao

  2. School of Engineering, University of Minho, Guimarães, Portugal

    Maribel Yasmina Santos

  3. NOVA IMS, Universidade Nova de Lisboa, Lisboa, Portugal

    Marco Painho

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Trilles, S., Schade, S., Belmonte, Ó., Huerta, J. (2015). Real-Time Anomaly Detection from Environmental Data Streams. In: Bacao, F., Santos, M., Painho, M. (eds) AGILE 2015. Lecture Notes in Geoinformation and Cartography. Springer, Cham. https://doi.org/10.1007/978-3-319-16787-9_8

Download citation

Publish with us

Policies and ethics

Search

Navigation