Abstract
This paper is devoted to solving the problem of reducing the time costs of the process of data quality assessment. The data describe energy resources consumption of various enterprises and institutions. The first part of the paper contains a review of recent data quality assessment studies was made. The analysis describes the problems of this process and the characteristics of the data, metadata and the algorithms used in it. The next part of the paper shows a new approach to reduce the time consumption of the process of assessing the data quality, which differs from the existing ones by the presence of a data-packaging and decision-making support using the oDMN+ notation. Finally, this paper presents an implementation example of the oDMN+ model for data on the energy consumption of the Volgograd hardware plant. The results showed that the use of data packaging and modeling the assessment process is a promising approach for modeling and reducing time costs in the process of data quality assessment for energy management systems used in the enterprises and institutions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tyukov, A., Brebels, A., Shcherbakov, M., Kamaev, V.: A concept of web-based energy data quality assurance and control system. In: 14th International Conference on Information Integration and Web-based Applications & Services (IIWAS 2012), pp. 267–271. ACM, New York, NY, USA (2012)
Fu, Y., Li, Z., Feng, F., Xu, P.: Data-quality detection and recovery for building energy management and control systems: case study on submetering. Sci. Technol. Built. Environ. 22(6), 798–809 (2016)
Mohammadi, M., Al-Fuqaha, A., Sorour, S., Guizani, M.: Deep learning for IoT big data and streaming analytics: a survey. IEEE Commun. Surv. Tutorials 20(4), 2923–2960 (2018)
Tyukov, A., Khrzhanovskaya, O., Sokolov, A., Shcherbakov, M., Kamaev, V.: Fast access to large timeseries datasets in SCADA systems. Res. J. Appl. Sci. 10(1), 12–16 (2015)
Aljumaili, M., Karim, R., Tretten, P.: Metadata-based data quality assessment. VINE J. Inf. Knowl. Manag. Syst. 46(2), 232–250 (2016)
Aquino, G., Farias, C.M., Pirmez, L.: Data Quality Assessment and Enhancement on Social and Sensor Data. BiDu-Posters@VLDB (2018)
Kuemper, D., Iggena, T., Toenjes, R., Pulvermueller, E.: Valid.IoT: a framework for sensor data quality analysis and interpolation. In: 9th ACM Multimedia Systems Conference (MMSys 2018), pp. 294–303. ACM, New York, NY, USA (2018)
do Nascimento, N.M., de Lucena, C.J.P.: FIoT: an agent-based framework for self-adaptive and self-organizing applications based on the Internet of Things. Inf. Sci. 378, 161–176 (2017)
Timms, G.P., de Souza, P.A., Reznik Jr., L., Smith, D.V.: Automated data quality assessment of marine sensors. Sensors 11(10), 9589–9602 (2011)
Campbell, J.L., et al.: Quantity is nothing without quality: automated QA/QC for streaming environmental sensor data. Bioscience 63(7), 574–585 (2013)
Sokolov, A., Scherbakov, M., Tyukov, A., Janovsky, T.: Data quality and assurance framework for sensor-driven applications. In: Mathematical Methods in Engineering and Technology (MMTT-31), pp. 87–97. St. Petersburg State Technological Institute, St. Petersburg (2017)
Dai, W., Yoshigoe, K., Parsley, W.: Improving data quality through deep learning and statistical models. In: Latifi, S. (ed.) Information Technology - New Generations. AISC, vol. 558, pp. 515–522. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-54978-1_66
Using Machine Learning for Data Quality. https://www.talend.com/blog/2017/03/20/machine-learning-impact-data-quality-matching-transfered/. Accessed 06 May 2019
Rahman, A., Smith, D.V., Timms, G.: A novel machine learning approach toward quality assessment of sensor data. IEEE Sens. J. 14(4), 1035–1047 (2014)
Kontokosta, C.E.: DataIQ – a machine learning approach to anomaly detection for energy performance data quality and reliability. In: ACEEE Summer Study on Energy Efficiency in Buildings (2016)
Vale, S.: Statistical Data Quality in the UNECE. United Nations Statistics Division (2010)
Vetrò, A., Canova, L., Torchiano, M., Minotas, C.O., Iemma, R., Morando, F.: Open data quality measurement framework: definition and application to open government data. Govern. Inf. Q. 33(2), 325–337 (2016)
Horita, F.E.A., de Albuquerque, J.P., Marchezini, V., Mendiondo, E.M.: Bridging the gap between decision-making and emerging big data sources: an application of a model-based framework to disaster management in Brazil. Decis. Support Syst. 97, 12–22 (2017)
Envelope Design Pattern. https://www.marklogic.com/blog/envelope-design-pattern/. Accessed 06 May 2019
Dunning, T., Friedman, E.: Time Series Databases: New Ways to Store and Access Data, 1st edn. O’Reilly Media Inc, Sebastopol (2014)
Friendly Enough Expression Language (FEEL). https://docs.camunda.org/manual/7.4/reference/dmn11/feel/. Accessed 06 May 2019
Camunda Modeler. https://camunda.com/download/modeler/. Accessed 06 May 2019
Acknowledgments
The reported study was supported by RFBR, research project No. 19-47-340010/19.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Sokolov, A., Shcherbakov, M.V., Tyukov, A., Janovsky, T. (2019). A New Approach to Reduce Time Consumption of Data Quality Assessment in the Field of Energy Consumption. In: Kravets, A., Groumpos, P., Shcherbakov, M., Kultsova, M. (eds) Creativity in Intelligent Technologies and Data Science. CIT&DS 2019. Communications in Computer and Information Science, vol 1083. Springer, Cham. https://doi.org/10.1007/978-3-030-29743-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-29743-5_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-29742-8
Online ISBN: 978-3-030-29743-5
eBook Packages: Computer ScienceComputer Science (R0)