Skip to main content
SpringerLink
Log in

Online Anomaly Energy Consumption Detection Using Lambda Architecture

  • Conference paper
  • First Online:
Big Data Analytics and Knowledge Discovery (DaWaK 2016)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 9829))

Included in the following conference series:

Abstract

With the widely use of smart meters in the energy sector, anomaly detection becomes a crucial mean to study the unusual consumption behaviors of customers, and to discover unexpected events of using energy promptly. Detecting consumption anomalies is, essentially, a real-time big data analytics problem, which does data mining on a large amount of parallel data streams from smart meters. In this paper, we propose a supervised learning and statistical-based anomaly detection method, and implement a Lambda system using the in-memory distributed computing framework, Spark and its extension Spark Streaming. The system supports not only iterative refreshing the detection models from scalable data sets, but also real-time anomaly detection on scalable live data streams. This paper empirically evaluates the system and the detection algorithm, and the results show the effectiveness and the scalability of the lambda detection system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Adnan, R., Setan, H., Mohamad, M.N.: Multiple outliers detection procedures in linear regression. Matematika 19, 29–45 (2003)

    Google Scholar 

  2. Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor networks: a survey. J. Comput. Netw. 38(4), 393–422 (2002)

    Article  Google Scholar 

  3. Ardakanian, O., Koochakzadeh, N., Singh, R.P., Golab, L., Keshav, S.: Computing electricity consumption profiles from household smart meter data. In: EDBT/ICDT Workshops, vol. 14, pp. 140–147 (2014)

    Google Scholar 

  4. Brown, M., Barrington-Leigh, C., Brown, Z.: Kernel regression for real-time building energy analysis. J. Build. Perform. Simul. 5(4), 263–276 (2011)

    Article  Google Scholar 

  5. Casado, R., Younas, M.: Emerging trends and technologies in big data processing. Concurrency Comput. Pract. Exp. 27(8), 2078–2091 (2015)

    Article  Google Scholar 

  6. Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 15 (2009)

    Article  Google Scholar 

  7. Cheng, B., Longo, S., Cirillo, F., Bauer, M., Kovacs, E.: Building a big data platform for smart cities: experience and lessons from santander. In: IEEE International Congress on Big Data, pp. 592–599. IEEE Press, New York (2015)

    Google Scholar 

  8. Chou, J.S., Telaga, A.S.: Real-time detection of anomalous power consumption. Renew. Sustain. Energ. Rev. 33, 400–411 (2014)

    Article  Google Scholar 

  9. De Nadai, M., van Someren, M.: Short-term anomaly detection in gas consumption through arima and artificial neural network forecast. In: IEEE Workshop on Environmental, Energy and Structural Monitoring Systems, pp. 250–255. IEEE Press, New York (2015)

    Google Scholar 

  10. Frigge, M., Hoaglin, D.C., Iglewicz, B.: Some implementations of the boxplot. Am. Stat. 43(1), 50–54 (1989)

    Google Scholar 

  11. Gao, X.: Scalable Architecture for Integrated Batch and Streaming Analysis of Big Data. Doctoral dissertation, Indiana University (2015)

    Google Scholar 

  12. Hasani, Z., Kon-Popovska, M., Velinov, G.: Lambda architecture for real time big data analytic. In: ICT Innovations (2014)

    Google Scholar 

  13. Jakkula, V., Cook, D.: Outlier detection in smart environment structured power datasets. In: 6th International Conference on Intelligent Environments, pp. 29–33. IEEE Press, New York (2010)

    Google Scholar 

  14. Janetzko, H., Stoffel, F., Mittelstdt, S., Keim, D.A.: Anomaly detection for visual analytics of power consumption data. Comput. Graph. 38, 27–37 (2014)

    Article  Google Scholar 

  15. Kroß, J., Brunnert, A., Prehofer, C., Runkler, T.A., Krcmar, H.: Stream processing on demand for lambda architectures. In: Beltrain, M., et al. (eds.) EPEW 2015. LNCS, vol. 9272, pp. 243–257. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  16. Lee, A.H., Fung, W.K.: Confirmation of multiple outliers in generalized linear and nonlinear regressions. J. Comput. Stat. Data Anal. 25(1), 55–65 (1997)

    Article  MATH  Google Scholar 

  17. Lee, W., Stolfo, S.J., Chan, P.K., Eskin, E., Fan, W., Miller, M., Zhang, J.: Real time data mining-based intrusion detection. In: DARPA Information Survivability Conference and Exposition II, DISCEX 2001, vol. 1, pp. 89–100. IEEE Press, New York (2001)

    Google Scholar 

  18. Liu, F., Jiang, H., Lee, Y.M., Snowdon, J., Bobker, M.: Statistical modeling for anomaly detection, forecasting and root cause analysis of energy consumption for a portfolio of buildings. In: 12th International Conference of the International Building Performance Simulation Association (2011)

    Google Scholar 

  19. Liu, G., Zhu, W., Saunders, C., Gao, F., Yu, Y.: Real-time complex event processing and analytics for smart grid. Procedia Comput. Sci. 61, 113–119 (2015)

    Article  Google Scholar 

  20. Liu, X., Iftikhar, N., Xie, X.: Survey of real-time processing systems for big data. In: 18th International Database Engineering & Applications Symposium, pp. 356–361. ACM, New York (2014)

    Google Scholar 

  21. Liu, X., Nielsen, P.S.: Streamlining smart meter data analytics. In: Proceedings of the 10th Conference on Sustainable Development of Energy, Water and Environment Systems, SDEWES 2015.0558, pp. 1–14 (2015)

    Google Scholar 

  22. Liu, X., Nielsen, P.S.: A hybrid ICT-solution for smart meter data analytics. J. Energy (2016). doi:10.1016/j.energy.2016.05.068

    Google Scholar 

  23. Liu, X., Golab, L., Ilyas, I.F.: SMAS: a smart meter data analytics system. In: Proceedings of the ICDE, pp. 1476–1479 (2015)

    Google Scholar 

  24. Liu, X., Golab, L., Golab, W., Ilyas, I.F.: Benchmarking smart meter data analytics. In: Proceedings of the EDBT, pp. 385–396 (2015)

    Google Scholar 

  25. Magld, K.W.: Features extraction based on linear regression technique. J. Comput. Sci. 8(5), 701–704 (2012)

    Article  Google Scholar 

  26. Martnez-Prieto, M.A., Cuesta, C.E., Arias, M., Fernnde, J.D.: The solid architecture for real-time management of big semantic data. Future Gener. Comput. Syst. 47, 62–79 (2015)

    Article  Google Scholar 

  27. Marz, N., Warren, J.: Big Data: Principles and Best Practices of Scalable Realtime Data Systems, 1st edn. Manning Publications Co., Greenwich (2013)

    Google Scholar 

  28. Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Xin, D.: MLlib: Machine Learning in Apache Spark (2015). arXiv preprint: arXiv:1505.06807

  29. Preuveneers, D., Berbers, Y., Joosen, W.: SAMURAI: a batch and streaming context architecture for large-scale intelligent applications and environments. J. Ambient Intell. Smart Environ. 8(1), 63–78 (2016)

    Article  Google Scholar 

  30. Schneider, M., Ertel, W., Ramos, F.: Expected Similarity Estimation for Large-Scale Batch and Streaming Anomaly Detection (2016). arXiv preprint: arXiv:1601.06602

    Google Scholar 

  31. Sequeira, H., Carreira, P., Goldschmidt, T., Vorst, P.: Energy cloud: real-time cloud-native energy management system to monitor and analyze energy consumption in multiple industrial sites. In: 7th IEEE/ACM International Conference on Utility and Cloud Computing, pp. 529–534. IEEE Press, New York (2014)

    Google Scholar 

  32. Villari, M., Celesti, A., Fazio, M., Puliafito, A.: Alljoyn lambda: an architecture for the management of smart environments in IOT. In: IEEE International Conference on Smart Computing Workshops, pp. 9–14. IEEE Press, New York (2014)

    Google Scholar 

  33. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Stoica, I.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: 9th USENIX Conference on Networked Systems Design and Implementation, p. 2. USENIX Association (2012)

    Google Scholar 

  34. Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In: 4th USENIX Conference on Hot Topics in Cloud Computing, p. 10. USENIX Association (2012)

    Google Scholar 

  35. Zhang, Y., Chen, W., Black, J.: Anomaly detection in premise energy consumption data. In: Power and Energy Society General Meeting, pp. 1–8. IEEE Press, New York (2011)

    Google Scholar 

Download references

Acknowledgements

This research was supported by the CITIES project (NO. 1035-00027B) funded by Innovation Fund Denmark.

Author information

Authors and Affiliations

  1. Technical University of Denmark, Kongens Lyngby, Denmark

    Xiufeng Liu, Per Sieverts Nielsen & Alfred Heller

  2. University College of Northern Denmark, Aalborg, Denmark

    Nadeem Iftikhar

Authors
  1. Xiufeng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nadeem Iftikhar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Per Sieverts Nielsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alfred Heller
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiufeng Liu .

Editor information

Editors and Affiliations

  1. University of Science and Technology , Rolla, Missouri, USA

    Sanjay Madria

  2. Osaka University , Osaka, Japan

    Takahiro Hara

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Liu, X., Iftikhar, N., Nielsen, P.S., Heller, A. (2016). Online Anomaly Energy Consumption Detection Using Lambda Architecture. In: Madria, S., Hara, T. (eds) Big Data Analytics and Knowledge Discovery. DaWaK 2016. Lecture Notes in Computer Science(), vol 9829. Springer, Cham. https://doi.org/10.1007/978-3-319-43946-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-43946-4_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-43945-7

  • Online ISBN: 978-3-319-43946-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation