Abstract
Errors are ubiquitous in time series data as sensors are often unstable. Existing approaches based on constraints can achieve good data repair effect on abnormal values. The constraint typically refers to the speed range of data changes. If the speed of data changes is not in the range, it is identified as abnormal data violating the constraint and needs repair, like if the oil consumption per hour of a sedan is negative or greater than 15 gallons, it is probably abnormal data. However, existing methods are only limited to specific type of data whose value change speed is stable. They will be inefficient when handling the data stream with sharp fluctuation because their constraints based on priori, fixed speed range might miss most abnormal data. To make up the gap in this scenario, an online cleaning approach based on dynamic speed constraints is proposed for time series data with fluctuating value change speed. The dynamic constraints proposed is not determined in advance but self-adaptive as data changes over time. A dual window mechanism is devised to transform the global optimum of data repair problem to local optimum problem. The classic minimum change principle and median principle are introduced for data repair. With respect to repair invalidation of minimum change principle facing consecutive data points violating constraints, we propose to use the boundary of the corresponding candidate repair set as repair strategy. Extensive experiments on real datasets demonstrate that the proposed approach can achieve higher repair accuracy than traditional approaches.
Supported by the National Natural Science Foundation of China under Grant 61303016.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Broeck, J.V.D., Fadnes, L.T.: Data Cleaning (2005). https://doi.org/10.1371/journal.pmed.0020267
Welch, G., Bishop, G.: An Introduction to the Kalman Filter (1995). https://doi.org/10.1007/978-0-387-31439-6_716
Challa, S., Barshalom, Y.: Nonlinear filter design using Fokker-Planck-Kolmogorov probability density evolutions. IEEE Trans. Aerosp. Electron. Syst. 36(1), 309–315 (2000). https://doi.org/10.1109/7.826335
Brillinger, D.R.: Time series: data analysis and theory. IEEE Signal Process. Mag. 19(2), 94–94 (1975). https://doi.org/10.1137/1.9780898719246
Zhao, J., Wang, Z., Dong, S.P.: Online sequential extreme learning machine with forgetting mechanism. Neurocomputing 87(none), 79–89 (2012). https://doi.org/10.1016/j.neucom.2012.02.003
Song, S., Zhang, A., Wang, J., Yu, P.S.: SCREEN: stream data cleaning under speed constraint. In: SIGMOD 2015, vol. 06, no. 3, pp. 827–841 (2015). https://doi.org/10.1145/2723372.2723730
Keogh, E.J., Chu, S., Hart, D.M., Pazzani, M.J.: An online algorithm for segmenting time series. In: ICDM, pp. 289–296 (2001). https://doi.org/10.1109/ICDM.2001.989531
AI-Shammari, A., Zhou, R., Naseriparsa, M., Liu, C.: An effective density-based clustering and dynamic maintenance framework for evolving medical data streams. Int. J. Med. Inform. 126, 176–186 (2019). https://doi.org/10.1016/j.ijmedinf.2019.03.016
Han, D., Xiao, C., Zhou, R., Wang, G., Huo, H., Hui, X.: Load shedding for window joins over streams. In: Yu, J.X., Kitsuregawa, M., Leong, H.V. (eds.) WAIM 2006. LNCS, vol. 4016, pp. 472–483. Springer, Heidelberg (2006). https://doi.org/10.1007/11775300_40
Volkovs, M., Fei, C., Szlichta, J., et al.: Continuous data cleaning. In: IEEE International Conference on Data Engineering (2014). https://doi.org/10.1109/ICDE.2014.6816655
Hong, L., Tk, A.K., Thomas, J.P., et al.: Cleaning Framework for BigData: an interactive approach for data cleaning. In: IEEE Second International Conference on Big Data Computing Service & Applications (2016). https://doi.org/10.1109/BigDataService.2016.41
Huang, G.B., Zhu, Q.Y., Siew, C.K.: Extreme learning machine: a new learning scheme of feedforward neural networks. In: IEEE International Joint Conference on Neural Networks (2005). https://doi.org/10.1109/IJCNN.2004.1380068
Zhang, A., Song, S., Wang, J.: Sequential data cleaning: a statistical approach. In: ACM SIGMOD International Conference on Management of Data (2016). https://doi.org/10.1145/2882903.2915233
Zhang, A., Song, S., Wang, J., Yu, P.S.: Time series data cleaning: from anomaly detection to anomaly repairing. Proc. VLDB Endow. 10(10), 1046–1057 (2017). https://doi.org/10.14778/3115404.3115410
Bohannon, P.. Flaster, M., Fan, W., Rastogi. R.: A cost-based model and effective heuristic for repairing constraints by value modification. In: SIGMOD Conference, pp. 143–154 (2005). https://doi.org/10.1145/1066157.1066175
Al-Shammari, A., Zhou, R., Liu, C., Naseriparsa, M., Vo, B.Q.: A framework for processing cumulative frequency queries over medical data streams. In: Hacid, H., Cellary, W., Wang, H., Paik, H.-Y., Zhou, R. (eds.) WISE 2018. LNCS, vol. 11234, pp. 121–131. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-02925-8_9
Zhou, R., Wang, G., Han, D., Gong, P., Xiao, C., Li, H.: Buffer-Preposed QoS adaptation framework and load shedding techniques over streams. In: Aberer, K., Peng, Z., Rundensteiner, E.A., Zhang, Y., Li, X. (eds.) WISE 2006. LNCS, vol. 4255, pp. 234–246. Springer, Heidelberg (2006). https://doi.org/10.1007/11912873_25
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Ding, G., Li, C., Wei, R., Sun, S., Liu, Z., Fan, C. (2020). Time Series Data Cleaning Based on Dynamic Speed Constraints. In: Huang, Z., Beek, W., Wang, H., Zhou, R., Zhang, Y. (eds) Web Information Systems Engineering – WISE 2020. WISE 2020. Lecture Notes in Computer Science(), vol 12343. Springer, Cham. https://doi.org/10.1007/978-3-030-62008-0_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-62008-0_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62007-3
Online ISBN: 978-3-030-62008-0
eBook Packages: Computer ScienceComputer Science (R0)