Advertisement

Annals of Operations Research

, Volume 265, Issue 2, pp 205–222 | Cite as

Optimal column subset selection for image classification by genetic algorithms

  • Pavel Krömer
  • Jan Platoš
  • Jana Nowaková
  • Václav Snášel
CS and OR in Big Data and Cloud Com

Abstract

Many problems in operations research can be solved by combinatorial optimization. Fixed-length subset selection is a family of combinatorial optimization problems that involve selection of a set of unique objects from a larger superset. Feature selection, p-median problem, and column subset selection problem are three examples of hard problems that involve search for fixed-length subsets. Due to their high complexity, exact algorithms are often infeasible to solve real-world instances of these problems and approximate methods based on various heuristic and metaheuristic (e.g. nature-inspired) approaches are often employed. Selecting column subsets from massive data matrices is an important technique useful for construction of compressed representations and low rank approximations of high-dimensional data. Search for an optimal subset of exactly k columns of a matrix, \(A^{m\times n}\), \(k < n\), is a well-known hard optimization problem with practical implications for data processing and mining. It can be used for unsupervised feature selection, dimensionality reduction, data visualization, and so on. A compressed representation of raw real-world data can contribute, for example, to reduction of algorithm training times in supervised learning, to elimination of overfitting in classification and regression, to facilitation of better data understanding, and to many other benefits. This paper proposes a novel genetic algorithm for the column subset selection problem and evaluates it in a series of computational experiments with image classification. The evaluation shows that the proposed modifications improve the results obtained by artificial evolution.

Keywords

Column subset selection Galgorithms Dimensionality reduction Feature selection Classification 

References

  1. Avron, H., & Boutsidis, C. (2013). Faster subset selection for matrices and applications. SIAM Journal on Matrix Analysis and Applications, 34(4), 1464–1499. doi: 10.1137/120867287.CrossRefGoogle Scholar
  2. Balzano, L., Nowak, R., & Bajwa, W.U. (2010). Column subset selection with missing data. In NIPS workshop on low-rank methods for large-scale machine learning.Google Scholar
  3. Boutsidis, C. (2009). An improved approximation algorithm for the column subset selection problem. In Proceedings of the twentieth annual ACM-SIAM symposium on discrete algorithms, ser. SODA ’09. Philadelphia, PA, USA: Society for Industrial and Applied Mathematics (pp. 968–977). http://dl.acm.org/citation.cfm?id=1496770.1496875
  4. Boutsidis, C., & Magdon-Ismail, M. (2011). Deterministic feature selection for k-means clustering. arXiv:1109.5664.
  5. Boutsidis, C., Mahoney, M.W., & Drineas, P. (2009). An improved approximation algorithm for the column subset selection problem. In Proceedings of the 20th annual ACM-SIAM symposium on discrete algorithms (SODA), SIAM, Philadelphia (pp. 968–977).Google Scholar
  6. Boutsidis, C., Zouzias, A., Mahoney, M. W., & Drineas, P. (2013). Stochastic dimensionality reduction for k-means clustering. arXiv:1110.2897.
  7. Boutsidis, C., & Magdon-Ismail, M. (2014). A note on sparse leastsquares regression. Information Processing Letter, 114(5), 273–276.Google Scholar
  8. Boyce, D., Farhi, A., & Weischedel, R. (2013). Optimal subset selection: Multiple regression, interdependence and optimal network algorithms (Vol. 103). New York: Springer Science & Business Media.Google Scholar
  9. Chan, T. F., & Hansen, P. C. (1992). Some applications of the rank revealing QR factorization. SIAM Journal on Scientific and Statistical Computing, 13, 727–741.CrossRefGoogle Scholar
  10. Cicirello, V. A. (2006). Non-wrapping order crossover: An order preserving crossover operator that respects absolute position. In Proceedings of the 8th annual conference on genetic and evolutionary computation, GECCO ’06, New York, NY, USA. ACM (pp. 1125–1132).Google Scholar
  11. Çivril, A., & Magdon-Ismail, M. (2012) Column subset selection via sparse approximation of SVD. Theoretical Computer Science, 421(0), 1–14. http://www.sciencedirect.com/science/article/pii/S0304397511009388
  12. Çivril, A. (2014). Column subset selection problem is UG-hard. Journal of Computer and System Sciences, 80(4), 849–859.CrossRefGoogle Scholar
  13. Couvreur, C., & Bresler, Y. (2000). On the optimality of the backward greedy algorithm for the subset selection problem. SIAM Journal on Matrix Analysis and Applications, 21(3), 797–808. doi: 10.1137/S0895479898332928.CrossRefGoogle Scholar
  14. Czarn, A., MacNish, C., Vijayan, K., & Turlach, B. A. (2004). Statistical exploratory analysis of genetic algorithms: The influence of gray codes upon the difficulty of a problem. In Australian conference on artificial intelligence, ser. Lecture Notes in Computer Science, G. I. Webb and X. Yu, Eds., vol. 3339 (pp. 1246–1252). Springer.Google Scholar
  15. Das, A., & Kempe, D. (2008). Algorithms for subset selection in linear regression. In Proceedings of the fortieth annual ACM symposium on theory of computing, ser. STOC ’08. New York, NY, USA: ACM (pp. 45–54). doi: 10.1145/1374376.1374384
  16. de Hoog, F. R., & Mattheij, R. M. M. (2007). Subset selection for matrices. Linear Algebra and its Applications, 422, 349–359.CrossRefGoogle Scholar
  17. de Hoog, F. R., & Mattheij, R. M. M. (2011). A note on subset selection for matrices. Linear Algebra and its Applications, 434, 1845–1850.CrossRefGoogle Scholar
  18. Deshpande, A., & Rademacher, L. (2010). Efficient volume sampling for row/column subset selection. In Proceedings of the 2010 IEEE 51st annual symposium on foundations of computer science, ser. FOCS ’10. Washington, DC, USA: IEEE Computer Society (pp. 329–338). doi: 10.1109/FOCS.2010.38
  19. Diao, R., & Shen, Q. (2015). Nature inspired feature selection meta-heuristics. Artificial Intelligence Review, 44(3), 311–340.CrossRefGoogle Scholar
  20. Farahat, A. K. (2013). Distributed column subset selection on mapreduce. In 2013 IEEE 13th international conference on data mining (ICDM) (pp. 171–180).Google Scholar
  21. Farahat, A. K., Elgohary, A., Ghodsi, A., & Kamel, M. S. (2013). Greedy column subset selection for large-scale data sets, CoRR. arXiv:1312.6838.
  22. Farahat, A. K., Ghodsi, A., & Kamel, M. S. (2013). A fast greedy algorithm for generalized column subset selection. CoRR. arXiv:1312.6820.
  23. Foster, L., & Kommu, R. (2006). Algorithm 853: An efficient algorithm for solving rank-deficient least squares problems. ACM Transactions on Mathematical Software, 32, 157–165.CrossRefGoogle Scholar
  24. Friedberg, S. (2003). Linear algebra, 4th edn. Prentice-Hall Of India Pvt. Limited. http://books.google.cz/books?id=yLCLMQAACAAJ
  25. Garris, M. D. (1994). Design, collection, and analysis of handwriting sample image databases. Encyclopedia of Computer Science and Technology, 31(supplement 16), 189–213.Google Scholar
  26. Golub, G., & Van Loan, C. (1996). Matrix computations, 3rd edn. In Johns Hopkins studies in the mathematical sciences. Johns Hopkins University Press.Google Scholar
  27. Hastie, T., Tibshirani, R., & Friedman, J. (2013). The elements of statistical learning: data mining, inference, and prediction. Springer series in statistics. New York: Springer.Google Scholar
  28. Ipsen, I. C. F., Kelley, C. T., & Pope, S. R. (2011). Rank-deficient nonlinear least squares problems and subset selection. SIAM Journal on Numerical Analysis, 49, 1244–1266.CrossRefGoogle Scholar
  29. Jongen, H. Th, Meer, K., & Triesch, E. (2004). Optimization theory. Berlin: Kluwer Academic Publishers.Google Scholar
  30. Joshi, S., & Boyd, S. (2009). Sensor selection via convex optimization. IEEE Transactions on Signal Processing, 57, 451–462.CrossRefGoogle Scholar
  31. Krömer, P., & Platoš, J. (2014). Genetic algorithm for sampling from scale-free data and networks. In Proceedings of the 2014 annual conference on genetic and evolutionary computation, GECCO ’14 (pp. 793–800), New York, NY, USA, ACM.Google Scholar
  32. Krömer, P., & Platoš, J. (2016). A comparison of differential evolution and genetic algorithms for the column subset selection problem. In 9th international conference on computer recognition systems, CORES 2015; Wrocaw; Poland; Advances in Intelligent Systems and Computing, Vol 403 (pp. 223–232).Google Scholar
  33. Maynard, H. B., Zandin, K. B., & Zandin, K. B. (2001). Maynard’s industrial engineering handbook. McGraw-Hill, New York. No. Sirsi i9780070411029.Google Scholar
  34. Mitchell, M. (1996). An introduction to genetic algorithms. Cambridge, MA: MIT Press.Google Scholar
  35. Mladenovic, N., Brimberg, J., Hansen, P., & Moreno-Pérez, J. A. (2007). The p-median problem: a survey of metaheuristic approaches. European Journal of Operational Research, 179(3), 927–939.CrossRefGoogle Scholar
  36. Pan, C.-T., & Tang, P. (1999). Bounds on singular values revealed by QR factorizations. BIT Numerical Mathematics, 39(4), 740–756.CrossRefGoogle Scholar
  37. Ravindran, A. (2008). Operations Research and Management Science Handbook. New York: CRC Press. ISBN 978-0-8493-9721-9.Google Scholar
  38. Sabeti, M., Boostani, R., & Zoughi, T. (2012). Using genetic programming to select the informative eeg-based features to distinguish schizophrenic patients. Neural Network World, 22(1), 3–20.CrossRefGoogle Scholar
  39. Santana, L. E. A. S., & de Paula Canuto, A. M. (2014). Filter-based optimization techniques for selection of feature subsets in ensemble systems. Expert Systems with Applications, 41(4), 1622–1631. http://www.sciencedirect.com/science/article/pii/S0957417413006805
  40. Shen, J., Ju, B., Jiang, T., Ren, J., Zheng, M., Yao, C., et al. (2011). Column subset selection for active learning in image classification. Neurocomputing, 74(18), 3785–3792.CrossRefGoogle Scholar
  41. Tropp, J. A. (2009). Column subset selection, matrix factorization, and eigenvalue optimization. In: 20th annual ACM-SIAM symposium on discrete algorithms location, New York (pp. 978–986).Google Scholar
  42. Wang, Y., & Singh, A. (2015). An empirical comparison of sampling techniques for matrix column subset selection. In 2015 53rd annual allerton conference on communication, control, and computing, Allerton (pp. 1069–1074).Google Scholar
  43. Wu, A. S., Lindsay, R. K., & Riolo, R. (1997). Empirical observations on the roles of crossover and mutation. In Bäck, T. (ed.), Proceedings of the seventh international conference on genetic algorithms, San Francisco, CA (pp. 362–369). Morgan Kaufmann. http://citeseer.ist.psu.edu/wu97empirical.html

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of Computer Science, Faculty of Electrical Engineering and Computer ScienceVŠB-Technical University of OstravaOstrava-PorubaCzech Republic

Personalised recommendations