Skip to main content
SpringerLink
Log in

Regularized Information Loss for Improved Model Selection

  • Conference paper
  • First Online:
Intelligent Communication Technologies and Virtual Mobile Networks (ICICV 2023)

Part of the book series: Lecture Notes on Data Engineering and Communications Technologies ((LNDECT,volume 171))

Included in the following conference series:

  • 349 Accesses

Abstract

Information criteria are used in many applications including statistical model selection and intelligent systems. The traditional information criteria such as the Akaike information criterion (AIC) do not always provide an adequate penalty on the number of model covariates. To address this issue, we propose a novel method for evaluating statistical models based on information criterion. The proposed method, called regularized information criterion (RIL), modifies the penalty term in AIC to reduce model overfitting. The results of numerical experiments show that RIL provides a better reflection of model predictive error than AIC. Thus, RIL can be a useful tool in model selection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Akaike H (1998) Information theory and an extension of the maximum likelihood principle. In: Selected papers of Hirotugu Akaike. Springer, New York, NY, pp 199–213

    Google Scholar 

  2. Altinisik Y, Van Lissa CJ, Hoijtink H, Oldehinkel AJ, Kuiper RM (2021) Evaluation of inequality constrained hypotheses using a generalization of the AIC. Psychol Methods 26(5):599. Chicago

    Google Scholar 

  3. Bai Z, Choi KP, Fujikoshi Y (2018) Consistency of AIC and BIC in estimating the number of significant components in high-dimensional principal component analysis. Ann Stat 46(3):1050–1076

    Article  MathSciNet  MATH  Google Scholar 

  4. Barron AR (2020) Predicted squared error: a criterion for automatic model selection. In: Self-organizing methods in modeling. CRC Press, pp 87–103

    Google Scholar 

  5. Bozdogan H (1987) Model selection and Akaike’s information criterion (AIC): the general theory and its analytical extensions. Psychometrika 52(3):345–370

    Article  MathSciNet  MATH  Google Scholar 

  6. Burnham KP, Anderson DR, Huyvaert KP (2011) AIC model selection and multimodel inference in behavioral ecology: some background, observations, and comparisons. Behavioral Ecol Sociobiol 65(1):23–35

    Article  Google Scholar 

  7. Chen J, Chen Z (2012) Extended BIC for small-n-large-p sparse GLM. Stat Sin 22(2):555–574. http://www.jstor.org/stable/24310025

  8. Ding J, Tarokh V, Yang Y (2018) Model selection techniques: an overview. IEEE Sign Process Mag 35(6):16–34

    Article  Google Scholar 

  9. Dormann CF, Calabrese JM, Guillera-Arroita G, Matechou E, Bahn V, Bartoń K, Hartig F (2018) Model averaging in ecology: a review of Bayesian, information-theoretic, and tactical approaches for predictive inference. Ecol Monogr 88(4):485–504

    Google Scholar 

  10. Dziak JJ, Coffman DL, Lanza ST, Li R, Jermiin LS (2020) Sensitivity and specificity of information criteria. Briefings Bioinform 21(2):553–565

    Article  Google Scholar 

  11. Heinze G, Wallisch C, Dunkler D (2018) Variable selection-a review and recommendations for the practicing statistician. Biometrical J 60(3):431–449

    Article  MathSciNet  MATH  Google Scholar 

  12. Kalyaanamoorthy S, Minh BQ, Wong TK, Von Haeseler A, Jermiin LS (2017) ModelFinder: fast model selection for accurate phylogenetic estimates. Nat methods 14(6):587–589

    Article  Google Scholar 

  13. Kamalov F, Thabtah F (2017) A feature selection method based on ranked vector scores of features for classification. Ann Data Sci 4(4):483–502

    Article  Google Scholar 

  14. Kamalov F (2021) Orthogonal variance decomposition based feature selection. Expert Syst Appl 182:115191

    Article  Google Scholar 

  15. Khan FM, Gupta R (2020) ARIMA and NAR based prediction model for time series analysis of COVID-19 cases in India. J Saf Sci Resilience 1(1):12–18

    Article  Google Scholar 

  16. Kuiper R (2022) AIC-type theory-based model selection for structural equation models. Struct Eq Model Multidisc J 29(1):151–158

    Article  MathSciNet  Google Scholar 

  17. Lefort V, Longueville JE, Gascuel O (2017) SMS: smart model selection in PhyML. Mol Biol Evol 34(9):2422–2424

    Article  Google Scholar 

  18. Li H, Yang Z, Yan W (2022) An improved AIC onset-time picking method based on regression convolutional neural network. Mech Syst Sign Process 171:108867

    Article  Google Scholar 

  19. Li Y, Zhang Q, Wang L, Liang L (2021) An AIC-based approach to identify the most influential variables in eco-efficiency evaluation. Expert Syst Appl 167:113883

    Article  Google Scholar 

  20. Liu W, Rioul O, Beaudouin-Lafon M (2023) Bayesian information gain to design interaction

    Google Scholar 

  21. Mahmud N, Fricker Z, Hubbard RA, Ioannou GN, Lewis JD, Taddei TH, Kaplan DE (2021) Risk prediction models for post-operative mortality in patients with cirrhosis. Hepatology 73(1):204–218

    Article  Google Scholar 

  22. Mulder J, Raftery AE (2022) BIC extensions for order-constrained model selection. Sociol Methods Res 51(2):471–498

    Article  MathSciNet  Google Scholar 

  23. Piironen J, Vehtari A (2017) Comparison of Bayesian predictive methods for model selection. Stat Comput 27(3):711–735

    Article  MathSciNet  MATH  Google Scholar 

  24. Pham H (2019) A new criterion for model selection. Mathematics 7(12):1215

    Article  Google Scholar 

  25. Qasim OS, Algamal ZY (2018) Feature selection using particle swarm optimization-based logistic regression model. Chemometr. Intell Lab Syst 182:41–46

    Article  Google Scholar 

  26. Raschka S (2018) Model evaluation, model selection, and algorithm selection in machine learning. arXiv preprint arXiv:1811.12808

  27. Rajab K, Kamalov F (2021) Finite sample based mutual information. IEEE Access 9:118871–118879

    Article  Google Scholar 

  28. Schnapp S Sabato S (2021) Active feature selection for the mutual information criterion. In: Proceedings of the AAAI conference on artificial intelligence, vol 35, no 11, pp 9497–9504

    Google Scholar 

  29. Schwarz G (1978) Estimating the dimension of a model. Ann Stat 461–464

    Google Scholar 

  30. Shafiq A, Lone SA, Sindhu TN, Al-Mdallal QM, Rasool G (2021) Statistical modeling for bioconvective tangent hyperbolic nanofluid towards stretching surface with zero mass flux condition. Sci Rep 11(1):1–11

    Article  Google Scholar 

  31. Sharma PN, Shmueli G, Sarstedt M, Danks N, Ray S (2021) Prediction-oriented model selection in partial least squares path modeling. Decis Sci 52(3):567–607

    Article  Google Scholar 

  32. Solorio-Fernández S, Carrasco-Ochoa JA, Martínez-Trinidad JF (2020) A review of unsupervised feature selection methods. Artif Intell Rev 53(2):907–948

    Article  Google Scholar 

  33. Taylor DC, Snipes M, Barber NA (2018) Indicators of hotel profitability: model selection using Akaike information criteria. Tour Hosp Res 18(1):61–71

    Article  Google Scholar 

  34. Thabtah F, Kamalov F, Hammoud S, Shahamiri SR (2020) Least loss: a simplified filter method for feature selection. Inf Sci 534:1–15

    Article  MathSciNet  MATH  Google Scholar 

  35. Tredennick AT, Hooker G, Ellner SP, Adler PB (2021) A practical guide to selecting models for exploration, inference, and prediction in ecology. Ecology 102(6):e03336

    Article  Google Scholar 

  36. Wagenmakers EJ, Farrell S (2004) AIC model selection using Akaike weights. Psychon Bull Rev 11(1):192–196

    Article  Google Scholar 

  37. Yang W, Zhang D, Peng L, Zhuge C, Hong L (2021) Rational evaluation of various epidemic models based on the COVID-19 data of China. Epidemics 37:100501

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Canadian University Dubai, Dubai, UAE

    Firuz Kamalov & Sherif Moussa

  2. Faculty of Engineering, Canadian University Dubai, Dubai, UAE

    Jorge Avante Reyes

Authors
  1. Firuz Kamalov
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sherif Moussa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jorge Avante Reyes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Firuz Kamalov .

Editor information

Editors and Affiliations

  1. Department of Electronics and Communication Engineering, Francis Xavier Engineering College, Tirunelveli, Tamil Nadu, India

    G. Rajakumar

  2. Department of Electrical and Computer Engineering, Concordia University, Montreal, QC, Canada

    Ke-Lin Du

  3. ISEG—University of Lisbon, Lisbon, Portugal

    Álvaro Rocha

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kamalov, F., Moussa, S., Reyes, J.A. (2023). Regularized Information Loss for Improved Model Selection. In: Rajakumar, G., Du, KL., Rocha, Á. (eds) Intelligent Communication Technologies and Virtual Mobile Networks. ICICV 2023. Lecture Notes on Data Engineering and Communications Technologies, vol 171. Springer, Singapore. https://doi.org/10.1007/978-981-99-1767-9_58

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-1767-9_58

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-1766-2

  • Online ISBN: 978-981-99-1767-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation