Abstract
Missing value imputation (MVI) has been studied for several decades being the basic solution method for incomplete dataset problems, specifically those where some data samples contain one or more missing attribute values. This paper aims at reviewing and analyzing related studies carried out in recent decades, from the experimental design perspective. Altogether, 111 journal papers published from 2006 to 2017 are reviewed and analyzed. In addition, several technical issues encountered during the MVI process are addressed, such as the choice of datasets, missing rates and missingness mechanisms, and the MVI techniques and evaluation metrics employed, are discussed. The results of analysis of these issues allow limitations in the existing body of literature to be identified based upon which some directions for future research can be gleaned.
Similar content being viewed by others
References
Acuna E, Rodriguez C (2004) The treatment of missing values and its effect in the classifier accuracy. In: Banks D et al (eds) Classification, clustering and data mining applications. Springer, Berlin, pp 639–648
Aittokallio T (2009) Dealing with missing values in large-scale studies: microarray data imputation and beyond. Brief Bioinform 11(2):253–264
Armitage EG, Godzien J, Alonso-Herranz V, Lopez-Gonzalvez A, Barbas C (2015) Missing value imputation strategies for metabolomics data. Electrophoresis 36:3050–3060
Aussem A, de Morais SR (2010) A conservative feature subset selection algorithm with missing data. Neurocomputing 73:585–590
Aydilek IB, Arslan A (2012) A novel hybrid approach to estimating missing values in databases using k-nearest neighbors and neural networks. Int J Innov Comput Inf Control 8(7):4705–4717
Aydilek IB, Arslan A (2013) A hybrid method for imputation of missing values using optimized fuzzy c-means with support vector regression and a genetic algorithm. Inf Sci 233:25–35
Baraldi AN, Enders CK (2010) An introduction to modern missing data analyses. J Sch Psychol 48:5–37
Bras LP, Menezes JC (2007) Improving cluster-based missing value estimation of DNA microarray data. Biomol Eng 24:273–282
Brock GN, Shaffer JR, Blakesley RE, Lotz MJ, Tseng GC (2008) Which missing value imputation method to use in expression profiles: a comparative study and two selection schemes. BMC Bioinform 9:12–23
Burgette LF, Reiter JP (2014) Multiple imputation for missing data via sequential regression trees. Am J Epidemiol 172(9):1070–1076
Celton M, Malpertuy A, Lelandais G, de Brevern AG (2010) Comparative analysis of missing value imputation methods to improve clustering and interpretation of microarray experiments. BMC Genom 11:15–30
Chen X, Wei Z, Li Z, Liang J, Cai Y, Zhang B (2017) Ensemble correlation-based low-rank matrix completion with applications to traffic data imputation. Knowl Based Syst 132:249–262
Cheng KO, Law NF, Siu WC (2012) Iterative bicluster-based least square framework for estimation of missing values in microarray gene expression data. Pattern Recogn 45:1281–1289
Chiu C-C, Chan S-Y, Wang C-C, Wu W-S (2013) Missing value imputation for microarray data: a comprehensive comparison study and a web tool. BMC Syst Biol 7:S12
Clark PG, Grzymala-Busse JW, Rzasa W (2014) Mining incomplete data with singleton, subset and concept probabilistic approximations. Inf Sci 280:368–384
Conroy B, Eshelman L, Potes C, Xu-Wilson M (2016) A dynamic ensemble approach to robust classification in the presence of missing data. Mach Learn 102:443–463
De Leeuw ED (2001) Reducing missing data in surveys: an overview of methods. Qual Quant 35:147–160
De Souto MCP, Jaskowiak PA, Costa IG (2015) Impact of missing data imputation methods on gene expression clustering and classification. Bioinformatics 16:64–72
Di Nuovo AG (2011) Missing data analysis with fuzzy c-means: a study of its application in a psychological scenario. Expert Syst Appl 38:6793–6797
Di Zio M, Guarnera U, Luzi O (2007) Imputation through finite Gaussian mixture models. Comput Stat Data Anal 51:5305–5316
Ding Y, Ross A (2012) A comparison of imputation methods for handling missing scores in biometric fusion. Pattern Recogn 45:919–933
Ding Y, Simonoff JS (2010) An investigation of missing data methods for classification trees applied to binary response data. J Mach Learn Res 11:131–170
Donders ART, van der Heijden GJMG, Stijnen T, Moons KGM (2006) Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 59:1087–1091
Doove LL, Van Buuren S, Dusseldorp E (2014) Recursive partitioning for missing data imputation in the presence of interaction effects. Comput Stat Data Anal 72:92–104
Doquire G, Verleysen M (2012) Feature selection with missing data using mutual information estimators. Neurocomputing 90:3–11
Eirola E, Doquire G, Verleysen M, Lendasse A (2013) Distance estimation in numerical data sets with missing values. Inf Sci 240:115–128
Eirola E, Lendasse A, Vandewalle V, Biernacki C (2014) Mixture of Gaussians for distance estimation with missing data. Neurocomputing 131:32–42
Farhangfar A, Kurgan LA, Pedrycz W (2007) A novel framework for imputation of missing values in databases. IEEE Trans Syst Man Cybern A Syst Humans 37(5):692–709
Farhangfar A, Kurgan LA, Dy J (2008) Impact of imputation of missing values on classification error for discrete data. Pattern Recogn 41:3692–3705
Folino G, Pisani FS (2016) Evolving meta-ensemble of classifiers for handling incomplete and unbalanced datasets in the cyber security domain. Appl Soft Comput 47:179–190
Fortes I, Mora-Lopez L, Morales R, Triguero F (2006) Inductive learning models with missing values. Math Comput Model 44:790–806
Gan X, Liew AW-C, Yan H (2006) Microarray missing data imputation based on a set theoretic framework and biological knowledge. Nucleic Acids Res 34(5):1608–1619
Garcia JCF, Kalenatic D, Bello CAL (2011) Missing data imputation in multivariate data by evolutionary algorithms. Comput Hum Behav 27:1468–1474
Garcia-Laencina PJ, Sancho-Gomez J-L, Figueiras-Vidal AR, Verleysen M (2009) K nearest neighbours with mutual information for simultaneous classification and missing data imputation. Neurocomputing 72:1483–1493
Garcia-Laencina PJ, Sancho-Gomez J-L, Figueiras-Vidal AR (2010) Pattern classification with missing data: a review. Neural Comput Appl 19:263–282
Garcia-Laencina PJ, Sancho-Gomez J-L, Figueiras-Vidal AR (2013) Classifying patterns with missing values using multi-task learning perceptrons. Expert Syst Appl 40:1333–1341
Garciarena U, Santana R (2017) An extensive analysis of the interaction between missing data types, imputation methods, and supervised classifiers. Expert Syst Appl 89:52–65
Gautam C, Ravi V (2015) Data imputation via evolutionary computation, clustering and a neural network. Neurocomputing 156:134–142
Ghanad-Rezaie M, Soltanian-Zadeh H, Ying H, Dong M (2010) Selection-fusion approach for classification of datasets with missing values. Pattern Recogn 43:2340–2350
Ghorbani S, Desmarais MC (2017) Performance comparison of recent imputation methods for classification tasks over binary data. Appl Artif Intell 31(1):1–22
Graham JW, Olchowski AE, Gilreath TD (2007) How many imputations are really needed? Some practical clarifications of multiple imputation theory. Prev Sci 8:206–213
Hapfelmeier A, Ulm K (2014) Variable selection by random forests using data with missing values. Comput Stat Data Anal 80:129–139
Hapfelmeier A, Hothorn T, Ulm K (2012) Recursive partitioning on incomplete data using surrogate decisions and multiple imputation. Comput Stat Data Anal 56:1552–1565
Harel O, Zhou X-H (2007) Multiple imputation: review of theory, implementation and software. Stat Med 26:3057–3077
He Y, Zaslavsky AM, Harrington DP, Catalano HP, Landrum MB (2009) Multiple imputation in a large-scale complex survey: a practical guide. Stat Methods Med Res 19(6):653–670
Hron K, Templ M, Filzmoser P (2010) Imputation of missing values for compositional data using classical and robust methods. Comput Stat Data Anal 54:3095–3107
Hruschka ER Jr, Hruschka ER, Ebecken NFF (2007) Bayesian networks for imputation in classification problems. J Intell Inf Syst 29:231–252
Hu J, Li H, Waterman MS, Zhou XJ (2006) Integrative missing value estimation for microarray data. BMC Bioinform 7:449–462
Huang MW, Lin W-C, Chen C-W, Ke S-W, Tsai C-F, Eberle W (2016) Data preprocessing issues for incomplete medical datasets. Expert Syst 33(5):432–438
Huang J, Keung JW, Sarro F, Li Y-F, Yu YT, Chan WK, Sun H (2017) Cross-validation based K nearest neighbor imputation for software quality datasets: an empirical study. J Syst Softw 132:226–252
Iacus SM, Porro G (2007) Missing data imputation, matching and other applications of random recursive partitioning. Comput Stat Data Anal 52:773–789
Janssen KJM, Donders ART, Harrell FE Jr, Vergouwe Y, Chen Q, Grobbee DE, Moons KGM (2010) Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 63:721–727
Jerez JM, Molina I, Garcia-Laencina PJ, Alba E, Ribelles N, Martin M, Franco L (2010) Missing data imputation using statistical and machine learning methods in real breast cancer problem. Artif Intell Med 50:105–115
Kang P (2013) Locally linear reconstruction based missing value imputation for supervised learning. Neurocomputing 118:65–78
Kapelner A, Bleich J (2015) Prediction with missing data via Bayesian additive regression trees. Can J Stat 43(2):224–239
Khoshgoftaar TM, Van Hulse J (2008) Imputation techniques for multivariate missingness in software measurement data. Softw Qual J 16:563–600
Kiasari MA, Jang G-J, Lee M (2017) Novel iterative approach using generative ad discriminative models for classification with missing features. Neurocomputing 225:23–30
Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. Int Joint Conf Artif Intell 2:1137–1143
Leung KC, Leung CH (2013) Dynamic discriminant functions with missing feature values. Pattern Recogn Lett 34:1548–1556
Li YY, Parker LE (2014) Nearest neighbor imputation using spatial-temporal correlations in wireless sensor networks. Inf Fusion 15:64–79
Li D, Gu H, Zhang L (2010) A fuzzy c-means clustering algorithm based on nearest-neighbor intervals for incomplete data. Expert Syst Appl 37:6942–6947
Li Z, Sharaf MA, Sitbon L, Sadiq S, Indulska M, Zhou X (2014) A web-based approach to data imputation. World Wide Web 17:873–897
Liao S, Lin Y, Kang DD, Chandra D, Bon J, Kaminski N, Sciurba FC, Tseng GC (2014) Missing value imputation in high-dimensional phenomic data: imputable or not, and how? BMC Bioinform 15:346–357
Liew AW-C, Law N-F, Yan H (2011) Missing value imputation for gene expression data: computation techniques to recover missing data from available information. Brief Bioinform 12(5):498–513
Lin T, Lee JC, Ho HJ (2006) On fast supervised learning for normal mixture models with missing information. Pattern Recogn 39:1177–1187
Little RJA, Rubin DB (1987) Statistical analysis with missing data. Wiley, Hoboken
Liu C-C, Dai D-Q, Yan H (2010) The theoretic framework of local weighted approximation for microarray missing value estimation. Pattern Recogn 43:2993–3002
Liu J, Musialski P, Wonka P, Ye J (2013) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35(1):208–220
Luengo J, Garcia S, Herrera F (2012) On the choice of the best imputation methods for missing values considering three groups of classification methods. Knowl Inf Syst 32:77–108
Merlin P, Sorjamaa A, Maillet B, Lendasse A (2010) X-SOM and L-SOM: a double classification approach for missing value imputation. Neurocomputing 73:1103–1108
Mesquite DPP, Gomes JPP, Junior AHS, Nobre JS (2017) Euclidean distance estimation in incomplete datasets. Neurocomputing 248:11–18
Moons KGM, Donders RART, Stijnen T, Harrell FE Jr (2006) Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 59:1092–1101
Munoz JF, Rueda M (2009) New imputation methods for missing data using quantiles. J Comput Appl Math 232:305–317
Nishanth KJ, Ravi V (2016) Probabilistic neural network based categorical data imputation. Neurocomputing 218:17–25
Nishanth KJ, Ravi V, Ankaiah N, Bose I (2012) Soft computing based imputation and hybrid data and text mining: the case of predicting the severity of phishing alerts. Expert Syst Appl 39:10583–10589
Oh S, Kang DD, Brock GN, Tseng GC (2011) Biological impact of missing-value imputation on downstream analyses of gene expression profiles. Bioinformatics 27(1):78–86
Pan R, Yang T, Cao J, Lu K, Zhang Z (2015) Missing data imputation by K nearest neighbours based on grey relational structure and mutual information. Appl Intell 43:614–632
Pati SK, Das AK (2017) Missing value estimation for microarray data through cluster analysis. Knowl Inf Syst 52(3):709–750
Paul A, Sil J, Mukhopadhyay CD (2017) Gene selection for designing optimal fuzzy rule base classifier by estimating missing value. Appl Soft Comput 55:276–288
Peng C-Y, Zhu J (2008) Comparison of two approaches for handling missing covariates in logistic regression. Educ Psychol Measur 68:58–77
Polikar R, DePasquale J, Mohammed HS (2010) Learn++.MF: a random subspace approach for the missing feature problem. Pattern Recogn 43:3817–3832
Purwar A, Singh SK (2015) Hybrid prediction model with missing value imputation for medical data. Expert Syst Appl 42:5621–5631
Qin Y, Zhang S, Zhu X, Zhang J, Zhang C (2007) Semi-parametric optimization for missing data imputation. Appl Intell 27(1):79–88
Qin Y, Zhang S, Zhu X, Zhang J, Zhang C (2009) POP algorithm: kernel-based imputation to treat missing values in knowledge discovery from databases. Expert Syst Appl 36:2794–2804
Rahman MdG, Islam MdZ (2013) Missing value imputation using decision trees and decision forests by splittling and merging records: two novel techniques. Knowl Based Syst 53:51–65
Rao SSS, Shepherd LA, Bruno AE, Liu S, Miecznikowski JC (2013) Comparing imputation procedures for affymetrix gene expression datasets using MAQC datasets. Adv Bioinform 2013:790567
Raymond M, Roberts D (1987) A comparison of methods for treating incomplete data in selection research. Educ Psychol Meas 47:13–26
Saar-Tsechansky M, Provost F (2007) Handling missing values when applying classification models. J Mach Learn Res 8:1625–1657
Saha B, Gupta S, Phung D, Venkatesh S (2017) Effective sparse imputation of patient conditions in electronic medical records for emergency risk predictions. Knowl Inf Syst 53(1):179–206
Sehgal MSB, Gondal I, Dooley LS, Coppel R (2008) Ameliorative missing value imputation for robust biological knowledge inference. J Biomed Inform 41:499–514
Sehgal MSB, Gondal I, Dooley LS, Coppel R (2009) How to improve postgenomic knowledge discovery using imputation. EURASIP J Bioinform Syst Biol 2009:717136
Shah AD, Bartlett JW, Carpenter J, Nicholas O, Hemingway H (2014) Comparison of random forest and parametric imputation models for imputing missing data using MICE: a caliber study. Am J Epidemiol 179(6):764–774
Shao J, Meng W, Sun G (2017) Evaluation of missing value imputation methods for wireless soil datasets. Pers Ubiquit Comput 21(1):113–123
Silva-Ramirez E-L, Pino-Mejias R, Lopez-Coello M, Cubiles-de-la-Vega M-D (2011) Missing value imputation on missing completely at random data using multilayer perceptrons. Neural Netw 24:121–129
Silva-Ramirez E-L, Pino-Mejias R, Lopez-Coello M (2015) Single imputation with multilayer perceptron and multiple imputation combining multilayer perceptron and k-nearest neighbours for monotone patterns. Appl Soft Comput 29:65–74
Somasundaram RS, Nedunchezhian R (2011) Evaluation of three simple imputation methods for enhancing preprocessing of data with missing values. Int J Comput Appl 12(10):14–19
Song Q, Shepperd M, Chen X, Liu J (2008) Can k-NN imputation improve the performance of C4.5 with small software project datasets? A comparative evaluation. J Syst Softw 81:2361–2370
Stekhoven DJ, Buhlmann P (2012) MissForest—non-parametric missing value imputation for mixed-type data. Bioinformatics 28(1):112–118
Strike K, Emam KE, Madhavji N (2001) Software cost estimation with incomplete data. IEEE Trans Softw Eng 27(10):890–908
Subasi MM, Subasi E, Anthony M, Hammer PL (2011) A new imputation method for incomplete binary data. Discrete Appl Math 159:1040–1047
Sun Y, Braga-Neto U, Dougherty ER (2009) Impact of missing value imputation on classification for DNA microarray gene expression data—a model-based study. EURASIP J Bioinform Syst Biol 2009:504069
Tian J, Yu B, Yu D, Ma S (2014) Missing data analyses: a hybrid multiple imputation algorithm using gray system theory and entropy based on clustering. Appl Intell 40:376–388
Tsai C-F, Chang F-Y (2016) Combining instance selection for better missing value imputation. J Syst Softw 122:63–71
Tsikriktsis N (2005) A review of techniques for treating missing data in OM survey research. J Oper Manag 24:53–62
Tuikkala J, Elo LL, Nevalainen OS, Aittokallio T (2008) Missing value imputation improves clustering and interpretation of gene expression microarray data. BMC Bioinform 9:202–215
Twala B (2009) An empirical comparison of techniques for handling incomplete data using decision trees. Appl Artif Intell 23(5):373–405
Twala BETH, Jones MC, Hand DJ (2008) Good methods for coping with missing data in decision trees. Pattern Recogn Lett 29:950–956
Valdiviezo HC, Van Aelst S (2015) Tree-based prediction on incomplete data using imputation or surrogate decision. Inf Sci 311:163–181
Van Ginkel JR, Kroonenberg PM (2014) Using generalized procrustes analysis for multiple imputation in principal component analysis. J Classif 31:242–269
Van Ginkel JR, Van der Ark LA, Sijtsma K, Vermunt JK (2007) Two-way imputation: a Bayesian method for estimating missing scores in tests and questionnaires, and an accurate approximation. Comput Stat Data Anal 51:4013–4027
Van Hulse J, Khoshgoftaar TM (2014) Incomplete-case nearest neighbor imputation in software measurement data. Inf Sci 259:596–610
Wang X, Li A, Jiang Z, Feng H (2006) Missing value estimation for DNA microarray gene expression data by support vector regression imputation and orthogonal coding scheme. BMC Bioinform 7:32–41
Xia J, Zhang S, Cai G, Li L, Pan Q, Yan J, Ning G (2017) Adjusted weight voting algorithm for random forests in handling missing values. Pattern Recogn 69:52–60
Yan Y-T, Zhang Y-P, Zhang Y-W, Du X-Q (2017) A selective neural network ensemble classification for incomplete data. Int J Mach Learn Cybern 8(5):1513–1524
Yu T, Peng H, Sun W (2011) Incorporating nonlinear relationships in microarray missing value imputation. IEEE/ACM Trans Comput Biol Bioinf 8(3):723–731
Zhang S (2008) Parimputation: from imputation and null-imputation to partially imputation. IEEE Intell Inform Bull 9(1):32–38
Zhang S (2011) Shell-neighbor method and its application in missing data imputation. Appl Intell 35:123–133
Zhang S (2012) Nearest neighbor selection for iteratively kNN imputation. J Syst Softw 85:2541–2552
Zhang Y, Liu Y (2009) Data imputation using least squares support vector machines in urban arterial streets. IEEE Signal Process Lett 16(5):414–417
Zhang X, Song X, Wang H, Zhang H (2008) Sequential local least squares imputation estimating missing value of microarray data. Comput Biol Med 38:1112–1120
Zhang S, Jin Z, Zhu X (2011) Missing data imputation by utilizing information within incomplete instances. J Syst Softw 84:452–459
Zhang L, Bing Z, Zhang L (2015) A hybrid clustering algorithm based on missing attribute interval estimation for incomplete data. Pattern Anal Appl 18:377–384
Zhu X, Zhang S, Jin Z, Zhang Z, Xu Z (2011) Missing value estimation for mixed-attribute data sets. IEEE Trans Knowl Data Eng 23(1):110–121
Zhu B, He C, Liatsis P (2012) A robust missing value imputation method for noisy data. Appl Intell 36:61–74
Zuccolotto P (2012) Principal component analysis with interval imputed missing values. AStA Adv Stat Anal 96:1–23
Acknowledgements
The work of the first author was supported in part in part by the Healthy Aging Research Center, Chang Gung University from the Featured Areas Research Center Program within the Framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan under Grants EMRPD1I0481 and EMRPD1I0501, and in part by Chang Gung Memorial Hospital, Linkou under Grant CMRPD3I0031. This research of the second author was supported by the Ministry of Science and Technology of Taiwan (MOST 105-2410-H-008-043-MY3).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lin, WC., Tsai, CF. Missing value imputation: a review and analysis of the literature (2006–2017). Artif Intell Rev 53, 1487–1509 (2020). https://doi.org/10.1007/s10462-019-09709-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10462-019-09709-4