Skip to main content
Book cover

AGILE 2015 pp 125–144Cite as

Real-Time Anomaly Detection from Environmental Data Streams

Part of the Lecture Notes in Geoinformation and Cartography book series (LNGC)

Abstract

Modern sensor networks monitor a wide range of phenomena. They are applied in environmental monitoring, health care, optimization of industrial processes, social media, smart city solutions, and many other domains. All in all, they provide a continuously pulse of the almost infinite activities that are happening in the physical space—and in cyber space. The handling of the massive amounts of generated measurements poses a series of (Big Data) challenges. Our work addresses one of these challenges: the detection of anomalies in real-time. In this paper, we propose a generic solution to this problem, and introduce a system that is capable of detecting anomalies, generating notifications, and displaying the recent situation to the user. We apply CUSUM a statistical control algorithm and adopt it so that it can be used inside the Storm framework—a robust and scalable real-time processing framework. We present a proof of concept implementation from the area of environmental monitoring.

Keywords

  • Big data and real-time analysis
  • Environmental sensor data
  • CUSUM
  • STORM

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-16787-9_8
  • Chapter length: 20 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-16787-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Hardcover Book
USD   179.99
Price excludes VAT (USA)
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Notes

  1. 1.

    Storm, Apache Incubator, http://storm.apache.org.

  2. 2.

    Apache S4, Apache Incubator, http://incubator.apache.org/s4.

  3. 3.

    Samza, Apache Incubator, http://samza.incubator.apache.org.

  4. 4.

    Spark Streaming, Apache Incubator, http://spark.apache.org/streaming.

  5. 5.

    Storm Trident, Apache Incubator, http://storm.apache.org/documentation/Trident-API-Overview.html.

  6. 6.

    ActiveMQ framework, Apache Software Foundation, http://activemq.apache.org.

  7. 7.

    OpenWire protocol, Apache Incubator, http://activemq.apache.org/openwire.html.

  8. 8.

    STOMP protocol, Apache Incubator, http://activemq.apache.org/stomp.html.

  9. 9.

    http://www.citma.gva.es/web/calidad-ambiental/datos-on-line.

  10. 10.

    Leaflet: An Open-Source JavaScript Library for Mobile-Friendly Interactive Maps, http://leafletjs.com.

  11. 11.

    ESRI, http://www.esri.com.

  12. 12.

    Bootstrap, Twitter http://getbootstrap.com.

  13. 13.

    jQuery, jQuery Foundation http://jquery.com.

  14. 14.

    Highcharts JS, Highcharts AS http://www.highcharts.com.

References

  • Barratt, B., & Fuller, G. (2014). Intervention assessments in the control of PM10 emissions from an urban waste transfer station. Environmental Science: Processes and Impacts, 16(6), 1328–1337.

    Google Scholar 

  • Barratt, B., Atkinson, R., Anderson, H., Beevers, S., Kelly, F., Mudway, I., & Wilkinson, P. (2007). Investigation into the use of the cusum technique in identifying changes in mean air pollution levels following introduction of a traffic management scheme. Atmospheric Environment, 41(8), 1784–1791.

    CrossRef  Google Scholar 

  • Buschmann, F., Meunier, R., Rohnert, H., & Sommerlad, P. (1996). Pattern-oriented software architecture: A system of patterns (Vol. 1). New York: Wiley.

    Google Scholar 

  • Carslaw, D., Ropkins, K., & Bell, M. (2006). Change-point detection of gaseous and particulate traffic-related pollutants at a roadside location. Environmental Science and Technology, 40(22), 6912–6918.

    CrossRef  Google Scholar 

  • Charles, J., & Jeh-Nan, P. (2002). Evaluating environmental performance using statistical process control techniques. European Journal of Operational Research, 139(1), 68–83.

    CrossRef  Google Scholar 

  • Chelani, A. (2011). Change detection using cusum and modified cusum method in air pollutant concentrations at traffic site in Delhi. Stochastic Environmental Research and Risk Assessment, 25(6), 827–834.

    CrossRef  Google Scholar 

  • Chuen-Sheng, C. (1995). A multi-layer neural network model for detecting changes in the process mean. Computers and Industrial Engineering, 28(1), 51–61.

    CrossRef  Google Scholar 

  • Cox, S. (2007). Observations and measurements part 1—observation schema. Technical report, Open Geospatial Consortium (OGC).

    Google Scholar 

  • De Francisci Morales, G. (2013). Samoa: A platform for mining big data streams. In Proceedings of the 22nd International Conference on World Wide Web Companion, International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, WWW ’13 Companion (pp. 777–778).

    Google Scholar 

  • Fielding, R. (2000). Representational state transfer (rest) (Chap. 5). Fielding Dissertation.

    Google Scholar 

  • Grigg, O., Farewell, V., & Spiegelhalter, D. (2003). Use of risk-adjusted CUSUM and RSPRTcharts for monitoring in medical contexts. Statistical Methods in Medical Research, 12(2), 147–170.

    Google Scholar 

  • Guh, R., & Hsieh, Y. (1999). A neural network based model for abnormal pattern recognition of control charts. Computers and Industrial Engineering, 36(1), 97–108.

    CrossRef  Google Scholar 

  • Hapner, M., Burridge, R., Sharma, R., Fialli, J., & Stout, K. (2002). Java message service. Santa Clara, CA: Sun Microsystems Inc.

    Google Scholar 

  • Hickson, I. (2011). The WebSocket API. W3C Working Draft WD-websockets-20110929, September.

    Google Scholar 

  • Jeske, D., Montes De Oca, V., Bischoff, W., & Marvasti, M. (2009). Cusum techniques for timeslot sequences with applications to network surveillance. Computational Statistics Data Analysis, 53(12), 4332–4344.

    CrossRef  Google Scholar 

  • Kortuem, G., Kawsar, F., Fitton, D., & Sundramoorthy, V. (2010). Smart objects as building blocks for the internet of things. IEEE Internet Computing, 14(1), 44–51.

    CrossRef  Google Scholar 

  • Lucas, J. (1982). Combined Shewhart-CUSUM quality control schemes. Journal of Quality Technology, 14(2).

    Google Scholar 

  • Manovich, L. (2012). Trending: The promises and the challenges of big social data. In M. K. Gold (Ed.), Debates in the digital humanities (pp. 460–475). Minneapolis: U of Minnesota P.

    Google Scholar 

  • Mesnil, B., & Petitgas, P. (2009). Detection of changes in time-series of indicators using cusum control charts. Aquatic Living Resources, 22(02), 187–192.

    Google Scholar 

  • Morris, S., & Paradiso, J. (2002). Shoe-integrated sensor system for wireless gait analysis and real-time feedback. In Engineering in Medicine and Biology, 2002. 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society EMBS/BMES Conference, 2002. Proceedings of the Second Joint, IEEE (Vol. 3, pp. 2468–2469).

    Google Scholar 

  • Nativi, S., Craglia, M., & Pearlman, J. (2012). The brokering approach for multidisciplinary interoperability: A position paper. International Journal of Spatial Data Infrastructures Research, 7, 1–15.

    Google Scholar 

  • Osanaiye, P., & Talabi, C. (1989). On some non-manufacturing applications of counted data cumulative sum (CUSUM) control chart schemes. The Statistician, 38(4), 251–257.

    Google Scholar 

  • Page, E. (1954). Continuous inspection schemes. Biometrika, 41(1/2), 100–115.

    CrossRef  Google Scholar 

  • Shvachko, K., Kuang, H., Radia, S., & Chansler, R. (2010). The Hadoop distributed file system. In 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), IEEE (pp. 1–10).

    Google Scholar 

  • Simoncelli, D., Dusi, M., Gringoli, R., & Niccolini, S. (2013). Stream-monitoring with blockmon: Convergence of network measurements and data analytics platforms. SIGCOMM Computer Communication Review, 43(2), 29–36.

    CrossRef  Google Scholar 

  • Sitaram, D., Srinivasaraghavan, H., Agarwal, K., Agrawal, A., Joshi, N., & Ray, D. (2013). Pipelining acoustic model training for speech recognition using storm. In 2013 Fifth International Conference on Computational Intelligence, Modelling and Simulation (CIMSim) (pp. 219–224).

    Google Scholar 

  • Sunderrajan, A., Aydt, H., & Knoll, A. (2014). Real time load prediction and outliers detection using storm. In Proceedings of the 8th ACM International Conference on Distributed Event-Based Systems, ACM, New York, NY, USA, DEBS ’14 (pp. 294–297).

    Google Scholar 

  • Trilles, S., Belmonte, O., Diaz, L., & Huerta, J. (2014). Mobile access to sensor networks by using GIS standards and restful services. IEEE Sensors Journal, 14(12), 4143–4153.

    CrossRef  Google Scholar 

  • Yutan, D., Jun, L., Fang, L., & Luying, C. (2014). A real-time anomalies detection system based on streaming technology. In 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics (IHMSC) (Vol. 2, pp. 275–279).

    Google Scholar 

Download references

Acknowledgments

This work has been supported in part by European Commission and Generalitat Valenciana government (grants ACIF/2012/112 and BEFPI/2014/067).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Sergio Trilles .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Trilles, S., Schade, S., Belmonte, Ó., Huerta, J. (2015). Real-Time Anomaly Detection from Environmental Data Streams. In: Bacao, F., Santos, M., Painho, M. (eds) AGILE 2015. Lecture Notes in Geoinformation and Cartography. Springer, Cham. https://doi.org/10.1007/978-3-319-16787-9_8

Download citation