Improving Data Quality Through Deep Learning and Statistical Models
Traditional data quality control methods are based on users’ experience or previously established business rules, and this limits performance in addition to being a very time consuming process with lower than desirable accuracy. Utilizing deep learning, we can leverage computing resources and advanced techniques to overcome these challenges and provide greater value to users.
In this paper, we, the authors, first review relevant works and discuss machine learning techniques, tools, and statistical quality models. Second, we offer a creative data quality framework based on deep learning and statistical model algorithm for identifying data quality. Third, we use data involving salary levels from an open dataset published by the state of Arkansas to demonstrate how to identify outlier data and how to improve data quality via deep learning. Finally, we discuss future work.
KeywordsData quality Data clean Deep learning Statistical quality control Weka
- 5.Natarajan, B. K. (2014). Machine learning: A theoretical approach. San Mateo: Morgan Kaufmann.Google Scholar
- 8.Deng, L., Hinton, G., & Kingsbury, B. (2013). New types of deep neural network learning for speech recognition and related applications: An overview. In IEEE international conference on acoustics, speech and signal processing (ICASSP), 2013 (pp. 8599–8603). IEEE.Google Scholar
- 10.Aggarwal, C. C. (2015). Outlier analysis. In Data mining (pp. 237–263). Springer International Publishing.Google Scholar
- 13.DeVor, R. E., Chang, T.-h., & Sutherland, J. W. (2007). Statistical quality design and control: Contemporary concepts and methods. Upper Saddle River: Prentice Hall.Google Scholar
- 14.Bluman, A. G. (2009). Elementary statistics: A step by step approach. New York: McGraw-Hill Higher Education.Google Scholar
- 15.Berthold, M. R., Cebron, N., Dill, F., Gabriel, T. R., Kötter, T., Meinl, T., Ohl, P., Sieb, C., Thiel, K., & Wiswedel, B. (2008). KNIME: The Konstanz information miner. Berlin Heidelberg: Springer.Google Scholar