# Deep Online Hierarchical Unsupervised Learning for Pattern Mining from Utility Usage Data

## Abstract

Machine learning approaches for non-intrusive load monitoring (NILM) have focused on supervised algorithms. Unsupervised approaches can be more interesting and of more practical use in real case scenarios. More specifically, they do not require labelled training data to be collected from individual appliances and the algorithm can be deployed to operate on the measured aggregate data directly. In this paper, we propose a fully unsupervised NILM framework based on Deep Belief network (DBN) and online Latent Dirichlet Allocation (LDA). Firstly, the raw signals of the house utilities are fed into DBN to extract low-level generic features in an unsupervised fashion, and then the hierarchical Bayesian model, LDA, learns high-level features that capture the correlations between the low-level ones. Thus, the proposed method (DBN-LDA) harnesses the DBN’s ability of learning distributed hierarchies of features to extract sophisticated appliances-specific features without the need of precise human-crafted input representations. The clustering power of the hierarchical Bayesian models helps further summarise the input data by extracting higher-level information representing the residents’ energy consumption patterns. Using Deep-Hierarchical models reduces the computational complexity since LDA is not directly applied to the raw data. The computational efficiency is crucial as our application involves massive data from different types of utility usages. Moreover, we develop a novel online inference algorithm to cope with this big data. Another novelty of this work is that the data is a combination of different utilities (e.g., electricity, water and gas) and some sensors measurements. Finally, we propose different methods to evaluate the results and preliminary experiments show that the DBN-LDA is promising to extract useful patterns.

## Keywords

Unsupervised non-intrusive load monitoring Pattern recognition Online Latent Dirichlet Allocation Deep belief network## Notes

### Acknowledgment

This work was supported by the Energy Technology Institute (UK) as part of the project: *High Frequency Appliance Disaggregation Analysis (HFADA)*. A. Bouchachia was supported by the European Commission under the Horizon 2020 Grant 687691 related to the project: *PROTEUS: Scalable Online Machine Learning for Predictive Analytics and Real-Time Interactive Visualization*.

## References

- 1.Bonfigli, R., Squartini, S., Fagiani, M., Piazza, F.: Unsupervised algorithms for non-intrusive load monitoring: an up-to-date overview. In: 2015 IEEE 15th International Conference on Environment and Electrical Engineering (EEEIC). IEEE (2015)Google Scholar
- 2.Salakhutdinov, R., Tenenbaum, J.B., Torralba, A.: Learning with hierarchical-deep models. IEEE Trans. Pattern Anal. Mach. Intell.
**35**(8), 1958–1971 (2013)CrossRefGoogle Scholar - 3.LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature
**521**(7553), 436 (2015)CrossRefGoogle Scholar - 4.Deng, L.: A tutorial survey of architectures, algorithms, and applications for deep learning. APSIPA Trans. Sig. Inf. Process. (2014)Google Scholar
- 5.Bengio, Y., et al.: Learning deep architectures for AI. Found. Trends\(\textregistered \) Mach. Learn.
**2**(1), 1–127 (2009)Google Scholar - 6.Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput.
**18**(7), 1527–1554 (2006)MathSciNetCrossRefGoogle Scholar - 7.Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res.
**3**(Jan), 993–1022 (2003)MATHGoogle Scholar - 8.Fischer, C.: Feedback on household electricity consumption: a tool for saving energy? Energ. Effi.
**1**(1), 79–104 (2008)CrossRefGoogle Scholar - 9.Kelly, J., Knottenbelt, W.: The UK-DALE dataset, domestic appliance-level electricity demand and whole-house demand from five UK homes. Sci. Data
**2**, 150007 (2015)CrossRefGoogle Scholar - 10.Filip, A.: BLUED: a fully labeled public dataset for event-based non-intrusive load monitoring research. In: 2nd Workshop on Data Mining Applications in Sustainability (SustKDD) (2011)Google Scholar
- 11.Kolter, J.Z., Johnson, M.J.: REDD: a public data set for energy disaggregation research. In: Workshop on Data Mining Applications in Sustainability (SIGKDD). San Diego, CA (2011)Google Scholar
- 12.Hoffman, M.D., Blei, D.M., Wang, C., Paisley, J.: Stochastic variational inference. J. Mach. Learn. Res.
**4**(1), 1303–1347 (2013)MathSciNetMATHGoogle Scholar - 13.Makonin, S., Popowich, F., Bartram, L., Gill, B., Bajic, I.V.: AMPds: a public dataset for load disaggregation and eco-feedback research. In: 2013 IEEE Electrical Power & Energy Conference (EPEC). IEEE (2013)Google Scholar
- 14.Makonin, S., Ellert, B., Bajić, I.V., Popowich, F.: Electricity, water, and natural gas consumption of a residential house in Canada from 2012 to 2014. Sci. Data
**3**, 160037 (2016)CrossRefGoogle Scholar - 15.Hart, G.W.: Nonintrusive appliance load monitoring. In: Proceedings of the IEEE (1992)Google Scholar
- 16.Liang, J., Ng, S.K., Kendall, G., Cheng, J.W.: Load signature study part I: basic concept, structure, and methodology. IEEE Trans. Power Delivery
**25**(2), 551–560 (2010)CrossRefGoogle Scholar - 17.Kolter, J.Z., Batra, S., Ng, A.Y.: Energy disaggregation via discriminative sparse coding. In: Advances in Neural Information Processing Systems (2010)Google Scholar
- 18.Srinivasan, D., Ng, W., Liew, A.: Neural-network-based signature recognition for harmonic source identification. IEEE Trans. Power Delivery
**21**(1), 398–405 (2006)CrossRefGoogle Scholar - 19.Berges, M., Goldman, E., Matthews, H.S., Soibelman, L.: Learning systems for electric consumption of buildings. In: Computing in Civil Engineering (2009)Google Scholar
- 20.Ruzzelli, A.G., Nicolas, C., Schoofs, A., O’Hare, G.M.: Real-time recognition and profiling of appliances through a single electricity sensor. In: 2010 7th Annual IEEE Communications Society Conference on Sensor Mesh and Ad Hoc Communications and Networks (SECON). IEEE (2010)Google Scholar
- 21.Kelly, J., Knottenbelt, W.: Neural nilm: deep neural networks applied to energy disaggregation. In: Proceedings of the 2nd ACM International Conference on Embedded Systems for Energy-Efficient Built Environments. ACM (2015)Google Scholar
- 22.Lai, Y.-X., Lai, C.-F., Huang, Y.-M., Chao, H.-C.: Multi-appliance recognition system with hybrid SVM/GMM classifier in ubiquitous smart home. Inf. Sci.
**230**, 39–55 (2013)CrossRefGoogle Scholar - 23.Kim, H., Marwah, M., Arlitt, M., Lyon, G., Han, J.: Unsupervised disaggregation of low frequency power measurements. In: Proceedings of the 2011 SIAM International Conference on Data Mining. SIAM (2011)Google Scholar
- 24.Kolter, J.Z., Jaakkola, T.: Approximate inference in additive factorial HMMs with application to energy disaggregation. In: Artificial Intelligence and Statistics (2012)Google Scholar
- 25.Johnson, M.J., Willsky, A.S.: Bayesian nonparametric hidden semi-Markov models. J. Mach. Learn. Res.
**14**(Feb), 673–701 (2013)MathSciNetMATHGoogle Scholar - 26.Wytock, M., Kolter, J.Z.: Contextually supervised source separation with application to energy disaggregation. In: AAAI (2014)Google Scholar
- 27.Saad, M., Abdelhamid, B.: Online Gaussian LDA for unsupervised pattern mining from utility usage data. In: ECML-PKDD (2018, submitted)Google Scholar
- 28.Hoffman, M., Bach, F.R., Blei, D.M.: Online learning for latent Dirichlet allocation. In: Advances in Neural Information Processing Systems (2010)Google Scholar
- 29.Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Stat.
**22**, 400–407 (1951)MathSciNetCrossRefGoogle Scholar - 30.Mohamad, S., Bouchachia, A., Sayed-Mouchaweh, M.: Asynchronous stochastic variational inference. arXiv preprint. arXiv:1801.04289 (2018)
- 31.Mohamad, S., Sayed-Mouchaweh, M., Bouchachia, A.: Active learning for classifying data streams with unknown number of classes. Neural Netw.
**98**, 1–5 (2018)CrossRefGoogle Scholar - 32.Mohamad, S., Bouchachia, A., Sayed-Mouchaweh, M.: A bi-criteria active learning algorithm for dynamic data streams. IEEE Trans. Neural Netw. Learn. Syst. (2016)Google Scholar