Abstract
Feature selection is an important factor of accurately classifying high dimensional data sets by identifying relevant features and improving classification accuracy. The use of feature selection in operations research allows for the identification of relevant features and the creation of optimal subsets of features for improved predictive performance. This paper proposes a novel feature selection algorithm inspired from ensemble pruning which involves the use of second-order conic programming modeled as an embedded feature selection technique with neural networks, named feature selection via second order cone programming (FSOCP). The proposed FSOCP algorithm trains features individually on a neural network and generates a probability class distribution and prediction, allowing the second-order conic programming model to determine the most important features for improved classification accuracies. The algorithm is evaluated on multiple synthetic data sets and compared with other feature selection techniques, demonstrating its promising potential as a feature selection approach.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akadi AE, Ouardighi AE, Aboutajdine D (2008) A powerful feature selection approach based on mutual information
Alelyani S, Tang J, Liu H (2018) Feature selection for clustering: a review. In: Data clustering: algorithms and applications. https://api.semanticscholar.org/CorpusID:7044218
Battiti R (1994) Using mutual information for selecting features in supervised neural net learning. IEEE Trans Neural Netw 5(4):537–50
Brown G, Pocock AC, Zhao M-J, Luján M (2012) Conditional likelihood maximisation: a unifying framework for information theoretic feature selection. J Mach Learn Res 13:27–66
Cheng G, Peddinti V, Povey D, Manohar V, Khudanpur S, Yan Y (2017) An exploration of dropout with LSTMS. Interspeech, Dublin
Dobos I, Vörösmarty G (2020) Supplier selection: comparison of DEA models with additive and reciprocal data. Cent Eur J Oper Res 29:447–462
Dougherty J, Kohavi R, Sahami M (1995) Supervised and unsupervised discretization of continuous features. In: Proceedings of the twelfth international conference on international conference on machine learning. Morgan Kaufmann Publishers Inc, San Francisco, CA, USA, pp 194–202
Duda R, Hart P, Stork DG (2001) Pattern classification, vol xx. Wiley, Hoboken
Duda J, Gurgul H, Syrek R (2021) Multi-feature evaluation of financial contagion. Cent Eur J Oper Res 30:1167–1194
El Aboudi N, Benhlima L (2016) Review on wrapper feature selection approaches. In: 2016 International conference on engineering and MIS (ICEMIS), pp 1–5
Ewertowski T, Çisil Güldoǧuş B, Kuter S, Akyüz S, Weber G-W, Sadłowska-Wrzesińska J, Racek E (2023) The use of machine learning techniques for assessing the potential of organizational resilience. Cent Eur J Oper Res. https://doi.org/10.1007/s10100-023-00875-z
Fleuret F (2004) Fast binary feature selection with conditional mutual information. J Mach Learn Res 5:1531–1555
Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19(1):1–67. https://doi.org/10.1214/aos/1176347963
Fukunaga K (1972) Introduction to statistical pattern recognition. Elsevier, Amsterdam
Gini CW (1971) Variability and mutability, contribution to the study of statistical distributions and relations. Stud Econ Giuridici R Univ Cagliari (1912). Reviewed in: Light RJ, Margolin BH: An analysis of variance for categorical data. J Am Stat Assoc 66:534–544. https://cir.nii.ac.jp/crid/1573950399715818496
Güldoğuş BC, Abdullah AN, Ali MA, Özöğür Akyüz S (2023) Autoselection of the ensemble of convolutional neural networks with second-order cone programming
Guo B, Nixon MS (2007) Gait feature subset selection by mutual information. In: 2007 First IEEE international conference on biometrics: theory, applications, and systems, pp 1–6
Guyon IM, Gunn SR, Nikravesh M, Zadeh LA (2006) Feature extraction—foundations and applications. Feature extraction
Hall MA, Smith LA (1999) Feature selection for machine learning: comparing a correlation-based filter approach to the wrapper. The Florida AI Research Society
He X, Cai D, Niyogi P (2005) Laplacian score for feature selection. NIPS
Hoque N, Singh M, Bhattacharyya DK (2018) EFS-MI: an ensemble feature selection method for classification. Complex Intell Syst 4:105–118
Kotsiantis SB, Kanellopoulos DN (2006) Discretization techniques: a recent survey
Kuncová M, Seknickova J (2021) Two-stage weighted Promethee II with results’ visualization. Cent Eur J Oper Res 30:547–571
Kuter S, Weber G-W, Akyürek Z (2017) A progressive approach for processing satellite data by operational research. Oper Res 17(2):371–393. https://doi.org/10.1007/s12351-016-0229-x
Lal TN, Chapelle O, Weston J, Elisseeff A (2006) Embedded methods. In: Guyon I, Nikravesh M, Gunn S, Zadeh LA (eds) Feature extraction: foundations and applications. Springer, Berlin Heidelberg, pp 137–165
Li J, Cheng K, Wang S, Morstatter F, Trevino R, Tang J, Liu H (2016) Feature selection: a data perspective. ACM Comput Surv. https://doi.org/10.1145/3136625
Li J, Cheng K, Wang S, Morstatter F, Trevino RP, Tang J, Liu H (2017) Feature selection: a data perspective. ACM Comput Surv. https://doi.org/10.1145/3136625
Li Y, Li T, Liu H (2017) Recent advances in feature selection and its applications. Knowl Inf Syst 53:551–577
Lin D, Tang X (2006) Conditional infomax learning: an integrated framework for feature extraction and fusion. In: European conference on computer vision
Marill T, Green DM (1963) On the effectiveness of receptors in recognition systems. IEEE Trans Inf Theory 9:11–17
Meyer PE, Bontempi G (2006) On the use of variable complementarity for feature selection in cancer classification. In: Evoworkshops
Neumann U, Genze N, Heider D (2017) EFS: an ensemble feature selection tool implemented as R-package and web-application. BioData Min 10:1–9
Nie F, Xiang S, Jia Y, Zhang C, Yan S (2008) Trace ratio criterion for feature selection. AAAI 2:671–676
Özmen A, Weber GW (2014) RMARS: robustification of multivariate adaptive regression spline under polyhedral uncertainty. J Comput Appl Math 259:914–924. https://doi.org/10.1016/j.cam.2013.09.055
Radovic MD, Ghalwash MF, Filipovic ND, Obradovic Z (2017) Minimum redundancy maximum relevance feature selection approach for temporal gene expression data. BMC Bioinform 18:1–14
Robnik-Sikonja M, Kononenko I (2003) Theoretical and empirical analysis of ReliefF and RReliefF. Mach Learn 53:23–69
Sánchez-Maroño N, Alonso-Betanzos A, Tombilla-Sanromán M (2007) Filter methods for feature selection—a comparative study. Ideal
Seijo-Pardo B, Bolón-Canedo V, Porto-Díaz I, Alonso-Betanzos A (2015) Ensemble feature selection for rankings of features. In: International work-conference on artificial and natural neural networks
Vidal-Naquet M, Ullman S (2003) Object recognition with informative features and linear classification. In: Proceedings ninth IEEE international conference on computer vision, vol 1, pp281–288
Weber G-W, Batmaz I, Köksal G, Taylan P, Yerlikaya-Özkurt F (2012) CMARS: a new contribution to nonparametric regression with multivariate adaptive regression splines supported by continuous optimization. Inverse Probl Sci Eng 20:371–400. https://doi.org/10.1080/17415977.2011.624770
Wright S (1965) The interpretation of population structure by f-statistics with special regard to systems of mating. Evolution 19:395–420
Yang HH, Moody JE (1999) Feature selection based on joint mutual information
Yu L, Liu H (2003) Feature selection for high-dimensional data: a fast correlation-based filter solution. In: International conference on machine learning
Zhao M, Lin M, Chiu B, Zhang Z, Song Tang X (2018) Trace ratio criterion based discriminative feature selection via l2, p-norm regularization for supervised learning. Neurocomputing 321:1–16. https://doi.org/10.1016/j.neucom.2018.08.040
Acknowledgements
This study is the part of the 1001—The Scientific and Technological Research Projects Funding Program, the project number is 119E100 and the study is supported by Scientific and Technological Research Council of Turkey. We also would like to thank to our colleague Muhammad Ammar Ali for his feedbacks.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Güldoğuş, B.Ç., Özögür-Akyüz, S. FSOCP: feature selection via second-order cone programming. Cent Eur J Oper Res (2024). https://doi.org/10.1007/s10100-023-00903-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s10100-023-00903-y