Online Anomaly Energy Consumption Detection Using Lambda Architecture

  • Xiufeng LiuEmail author
  • Nadeem Iftikhar
  • Per Sieverts Nielsen
  • Alfred Heller
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9829)


With the widely use of smart meters in the energy sector, anomaly detection becomes a crucial mean to study the unusual consumption behaviors of customers, and to discover unexpected events of using energy promptly. Detecting consumption anomalies is, essentially, a real-time big data analytics problem, which does data mining on a large amount of parallel data streams from smart meters. In this paper, we propose a supervised learning and statistical-based anomaly detection method, and implement a Lambda system using the in-memory distributed computing framework, Spark and its extension Spark Streaming. The system supports not only iterative refreshing the detection models from scalable data sets, but also real-time anomaly detection on scalable live data streams. This paper empirically evaluates the system and the detection algorithm, and the results show the effectiveness and the scalability of the lambda detection system.


Anomaly detection Real-time Lambda architecture Data mining 



This research was supported by the CITIES project (NO. 1035-00027B) funded by Innovation Fund Denmark.


  1. 1.
    Adnan, R., Setan, H., Mohamad, M.N.: Multiple outliers detection procedures in linear regression. Matematika 19, 29–45 (2003)Google Scholar
  2. 2.
    Akyildiz, I.F., Su, W., Sankarasubramaniam, Y., Cayirci, E.: Wireless sensor networks: a survey. J. Comput. Netw. 38(4), 393–422 (2002)CrossRefGoogle Scholar
  3. 3.
    Ardakanian, O., Koochakzadeh, N., Singh, R.P., Golab, L., Keshav, S.: Computing electricity consumption profiles from household smart meter data. In: EDBT/ICDT Workshops, vol. 14, pp. 140–147 (2014)Google Scholar
  4. 4.
    Brown, M., Barrington-Leigh, C., Brown, Z.: Kernel regression for real-time building energy analysis. J. Build. Perform. Simul. 5(4), 263–276 (2011)CrossRefGoogle Scholar
  5. 5.
    Casado, R., Younas, M.: Emerging trends and technologies in big data processing. Concurrency Comput. Pract. Exp. 27(8), 2078–2091 (2015)CrossRefGoogle Scholar
  6. 6.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: a survey. ACM Comput. Surv. 41(3), 15 (2009)CrossRefGoogle Scholar
  7. 7.
    Cheng, B., Longo, S., Cirillo, F., Bauer, M., Kovacs, E.: Building a big data platform for smart cities: experience and lessons from santander. In: IEEE International Congress on Big Data, pp. 592–599. IEEE Press, New York (2015)Google Scholar
  8. 8.
    Chou, J.S., Telaga, A.S.: Real-time detection of anomalous power consumption. Renew. Sustain. Energ. Rev. 33, 400–411 (2014)CrossRefGoogle Scholar
  9. 9.
    De Nadai, M., van Someren, M.: Short-term anomaly detection in gas consumption through arima and artificial neural network forecast. In: IEEE Workshop on Environmental, Energy and Structural Monitoring Systems, pp. 250–255. IEEE Press, New York (2015)Google Scholar
  10. 10.
    Frigge, M., Hoaglin, D.C., Iglewicz, B.: Some implementations of the boxplot. Am. Stat. 43(1), 50–54 (1989)Google Scholar
  11. 11.
    Gao, X.: Scalable Architecture for Integrated Batch and Streaming Analysis of Big Data. Doctoral dissertation, Indiana University (2015)Google Scholar
  12. 12.
    Hasani, Z., Kon-Popovska, M., Velinov, G.: Lambda architecture for real time big data analytic. In: ICT Innovations (2014)Google Scholar
  13. 13.
    Jakkula, V., Cook, D.: Outlier detection in smart environment structured power datasets. In: 6th International Conference on Intelligent Environments, pp. 29–33. IEEE Press, New York (2010)Google Scholar
  14. 14.
    Janetzko, H., Stoffel, F., Mittelstdt, S., Keim, D.A.: Anomaly detection for visual analytics of power consumption data. Comput. Graph. 38, 27–37 (2014)CrossRefGoogle Scholar
  15. 15.
    Kroß, J., Brunnert, A., Prehofer, C., Runkler, T.A., Krcmar, H.: Stream processing on demand for lambda architectures. In: Beltrain, M., et al. (eds.) EPEW 2015. LNCS, vol. 9272, pp. 243–257. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  16. 16.
    Lee, A.H., Fung, W.K.: Confirmation of multiple outliers in generalized linear and nonlinear regressions. J. Comput. Stat. Data Anal. 25(1), 55–65 (1997)CrossRefzbMATHGoogle Scholar
  17. 17.
    Lee, W., Stolfo, S.J., Chan, P.K., Eskin, E., Fan, W., Miller, M., Zhang, J.: Real time data mining-based intrusion detection. In: DARPA Information Survivability Conference and Exposition II, DISCEX 2001, vol. 1, pp. 89–100. IEEE Press, New York (2001)Google Scholar
  18. 18.
    Liu, F., Jiang, H., Lee, Y.M., Snowdon, J., Bobker, M.: Statistical modeling for anomaly detection, forecasting and root cause analysis of energy consumption for a portfolio of buildings. In: 12th International Conference of the International Building Performance Simulation Association (2011)Google Scholar
  19. 19.
    Liu, G., Zhu, W., Saunders, C., Gao, F., Yu, Y.: Real-time complex event processing and analytics for smart grid. Procedia Comput. Sci. 61, 113–119 (2015)CrossRefGoogle Scholar
  20. 20.
    Liu, X., Iftikhar, N., Xie, X.: Survey of real-time processing systems for big data. In: 18th International Database Engineering & Applications Symposium, pp. 356–361. ACM, New York (2014)Google Scholar
  21. 21.
    Liu, X., Nielsen, P.S.: Streamlining smart meter data analytics. In: Proceedings of the 10th Conference on Sustainable Development of Energy, Water and Environment Systems, SDEWES 2015.0558, pp. 1–14 (2015)Google Scholar
  22. 22.
    Liu, X., Nielsen, P.S.: A hybrid ICT-solution for smart meter data analytics. J. Energy (2016). doi: 10.1016/
  23. 23.
    Liu, X., Golab, L., Ilyas, I.F.: SMAS: a smart meter data analytics system. In: Proceedings of the ICDE, pp. 1476–1479 (2015)Google Scholar
  24. 24.
    Liu, X., Golab, L., Golab, W., Ilyas, I.F.: Benchmarking smart meter data analytics. In: Proceedings of the EDBT, pp. 385–396 (2015)Google Scholar
  25. 25.
    Magld, K.W.: Features extraction based on linear regression technique. J. Comput. Sci. 8(5), 701–704 (2012)CrossRefGoogle Scholar
  26. 26.
    Martnez-Prieto, M.A., Cuesta, C.E., Arias, M., Fernnde, J.D.: The solid architecture for real-time management of big semantic data. Future Gener. Comput. Syst. 47, 62–79 (2015)CrossRefGoogle Scholar
  27. 27.
    Marz, N., Warren, J.: Big Data: Principles and Best Practices of Scalable Realtime Data Systems, 1st edn. Manning Publications Co., Greenwich (2013)Google Scholar
  28. 28.
    Meng, X., Bradley, J., Yavuz, B., Sparks, E., Venkataraman, S., Liu, D., Xin, D.: MLlib: Machine Learning in Apache Spark (2015). arXiv preprint: arXiv:1505.06807
  29. 29.
    Preuveneers, D., Berbers, Y., Joosen, W.: SAMURAI: a batch and streaming context architecture for large-scale intelligent applications and environments. J. Ambient Intell. Smart Environ. 8(1), 63–78 (2016)CrossRefGoogle Scholar
  30. 30.
    Schneider, M., Ertel, W., Ramos, F.: Expected Similarity Estimation for Large-Scale Batch and Streaming Anomaly Detection (2016). arXiv preprint: arXiv:1601.06602
  31. 31.
    Sequeira, H., Carreira, P., Goldschmidt, T., Vorst, P.: Energy cloud: real-time cloud-native energy management system to monitor and analyze energy consumption in multiple industrial sites. In: 7th IEEE/ACM International Conference on Utility and Cloud Computing, pp. 529–534. IEEE Press, New York (2014)Google Scholar
  32. 32.
    Villari, M., Celesti, A., Fazio, M., Puliafito, A.: Alljoyn lambda: an architecture for the management of smart environments in IOT. In: IEEE International Conference on Smart Computing Workshops, pp. 9–14. IEEE Press, New York (2014)Google Scholar
  33. 33.
    Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Stoica, I.: Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: 9th USENIX Conference on Networked Systems Design and Implementation, p. 2. USENIX Association (2012)Google Scholar
  34. 34.
    Zaharia, M., Das, T., Li, H., Shenker, S., Stoica, I.: Discretized streams: an efficient and fault-tolerant model for stream processing on large clusters. In: 4th USENIX Conference on Hot Topics in Cloud Computing, p. 10. USENIX Association (2012)Google Scholar
  35. 35.
    Zhang, Y., Chen, W., Black, J.: Anomaly detection in premise energy consumption data. In: Power and Energy Society General Meeting, pp. 1–8. IEEE Press, New York (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Xiufeng Liu
    • 1
    Email author
  • Nadeem Iftikhar
    • 2
  • Per Sieverts Nielsen
    • 1
  • Alfred Heller
    • 1
  1. 1.Technical University of DenmarkKongens LyngbyDenmark
  2. 2.University College of Northern DenmarkAalborgDenmark

Personalised recommendations