Advertisement

Data-Driven Machine Learning Approach for Predicting Missing Values in Large Data Sets: A Comparison Study

  • Ogerta Elezaj
  • Sule Yildirim
  • Edlira KalemiEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10710)

Abstract

Pre-processing of large scale datasets in order to ensure data quality is a very important task in data mining. One of the serious threats to data quality is the lack of data collected during field experiments, which negatively affects the data quality. The missing data usually have significant effects in many real-life pattern classification scenarios, especially when it leads to biased parameter estimates but also disqualify for analysis purposes. The process of filling in the missing data based on other valid values of rest of the variables of a data set is known as the imputation process. In this paper, we present a new data-driven machine learning approach for imputing the missing data. Even though Machine Learning methods are used in order to impute missing data in the literature, it is difficult to decide on a single method to apply on a given data set for imputation. This is because imputation process is not considered as science but as art that focuses on choosing the best method with the least biased value. For this reason, we compare different machine learning methods, such as decision tree (C4.5), Bayesian network, clustering algorithm and artificial neural networks in this work. The comparison of the algorithms indicates that, for predicting categorical and numerical missing information in large survey data sets, clustering method is the most efficient out of the others methods found in literature. A hybrid method is introduced which combines unsupervised learning methods with supervised ones based on the missing ratio, for achieving a data imputation with higher accuracy. Additionally, some statistical imputation methods such as Mean\Mode, Hot-Deck have been applied emphasizing their limitations in large scale datasets in comparison to the machine learning methods. A comparison of all above mentioned methods, traditional statistical methods and machine learning methods has been made and conclusions are drawn for achieving data imputation with higher accuracy in data sets of large scale survey. Also, another objective of these experiments is to discover the effect of balancing the training data set in the performance of classifiers. All methods are tested to a real world data set, population and housing census.

Keywords

Imputation Data pre-processing Clustering Decision tree Neural networks Hot-Deck 

References

  1. 1.
    Mannila, H.: Data mining: machine learning, statistics, and databases. In: 8th International Conference on Scientific and Statistical Database Management (SSDBM 1996), p. 2 (1996)Google Scholar
  2. 2.
    Groves, R.M.: Survey Errors and Survey Costs. Wiley, New York (1989)CrossRefGoogle Scholar
  3. 3.
  4. 4.
    Allan, F.G., Wishart, J.: A method of estimating the yield of missing plot in field experiments. J. AgricSci. 20, 399–406 (1930)CrossRefGoogle Scholar
  5. 5.
    Barlett, M.S.: Some examples of statistical methods of research in agriculture and applied biology. J. R. Stat. Soc. B 4, 137–185 (1973)Google Scholar
  6. 6.
    Berglund, P.A.: An Introduction to Multiple Imputation of Complex Sample Data using SAS® v9.2. SAS Global Forum 2010-Statistics and Data Analysis (2010)Google Scholar
  7. 7.
    Rubin, D.B.: Inference and missing data. Biometrika 63, 581–592 (1976)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Rubin, D.B.: Multiple Imputation for Nonresponse in Surveys. Wiley, New York (1987)CrossRefGoogle Scholar
  9. 9.
    Little, R.J., Rubin, D.B.: Statistical Analysis with Missing Data, 2nd edn. Wiley, New York (2002)CrossRefGoogle Scholar
  10. 10.
    Schneider, T.: Analysis of incomplete climate data: estimation of mean values and covariance matrices and imputation of missing values. J. Clim. 14, 853–871 (2001). American Meteorological SocietyCrossRefGoogle Scholar
  11. 11.
    Di Zio, M., Scanu, M., Coppola, L., Luzi, O., Ponti, P.: Bayesian networks for imputation. J. R. Stat. Soc. Ser. A 167(2), 309–322 (2004a)Google Scholar
  12. 12.
    Purwar, A., Singh, S.K.: Empirical evaluation of algorithms to impute missing values for financial dataset. IEEE (2014)Google Scholar
  13. 13.
  14. 14.
    Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, vol. 2, pp. 1137–1143 (1995)Google Scholar
  15. 15.
    Li, Y., Sun, G., Zhu, Y.: Data imbalance problem in text classification. In: 2010 Third International Symposium on Information Processing (ISIP). IEEE (2010)Google Scholar
  16. 16.
    Chawla, N.V., et al.: SMOTE: synthetic minority over-sampling technique. arXiv preprint arXiv:1106.1813 (2002)
  17. 17.
    Visa, S.: Fuzzy classifiers for imbalanced data sets. Department of Electrical and Computer Engineering and Computer Science, University of Cincinnati, Cincinnati (2006)Google Scholar
  18. 18.
  19. 19.
    Han, J., Kamber, M.: Data Mining: Concepts and Techniques. Morgan Kaufmann Publishers, San Francisco (2001)zbMATHGoogle Scholar
  20. 20.
    Liu, Y., Salvendy, G.: Visualization support to better comprehend and improve decision tree classification modeling process: a survey and appraisal. Theor. Issues Ergon. Sci. 8(1), 63–92 (2007)CrossRefGoogle Scholar
  21. 21.
    Fujikawa, Y., Ho, T.B.: Cluster-based algorithms for dealing with missing values. In: Chen, M.-S., Yu, P.S., Liu, B. (eds.) PAKDD 2002. LNCS (LNAI), vol. 2336, pp. 549–554. Springer, Heidelberg (2002).  https://doi.org/10.1007/3-540-47887-6_54CrossRefGoogle Scholar
  22. 22.
    Westin, L.: Missing data and the preprocessing perceptron. Department of Computing Science, Umeå University (2002)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Faculty of Computer Science and Media TechnologyNorwegian University of Science and TechnologyGjøvikNorway
  2. 2.Faculty of Engineering and Physical SciencesUniversity of SurreyGuildfordUK

Personalised recommendations