Pattern Recognition and Image Analysis

, Volume 19, Issue 3, pp 447–455 | Cite as

General solutions for information-based and Bayesian approaches to model selection in linear regression and their equivalence

  • D. KropotovEmail author
  • D. Vetrov
Mathematical Theory of Pattern Recognition


In the paper we consider the problem of model selection for linear regression within Bayesian and information-based frameworks. For both cases we generalize known approaches (evidence-based and Akaike information criterion) and derive criterion functions in terms of (in general case non-factorial) weight priors which are assumed to be Gaussian. Optimization of these criterion functions leads to two semidefinite optimization problems which can be solved analytically. We present a method that finds best priors in both approaches and show their equivalence. Surprisingly it appears that optimal prior has rank one covariance matrix. We derive explicit condition of degenerative decision rule, i.e., regression with all weights equal to zero. We conclude with experiments that show that the proposed approach significantly reduces the time needed for model selection in comparison with alternatives based on cross-validation and iterative evidence maximization while keeping generalization ability


Akaike Information Criterion Relevance Vector Machine BAYESIAN Model Selection Regularization Matrix Regularization Coefficient 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    D. Spiegelhalter, N. Best, B. Carlin, and A. van der Linde, “Bayesian Measures of Model Complexity and Fit, J. Royal Statistical Soc., Ser. B 64, 583–640 (2002).zbMATHCrossRefGoogle Scholar
  2. 2.
    H. Akaike, “A New Look at Statistical Model Identification,” IEEE Trans. Automatic Control 25, 461–464 (1974).Google Scholar
  3. 3.
    C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006).Google Scholar
  4. 4.
    T. G. Dietterich, “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms,” Neural Computation 10, 1895–1924 (1998).CrossRefGoogle Scholar
  5. 5.
    D. J. C. MacKay, “The Evidence Framework Applied to Classification Networks,” Neural Computation 4, 720–736 (1992).CrossRefGoogle Scholar
  6. 6.
    R. M. Neal, Bayesian Learning for Neural Networks (Springer, New York, 1996).zbMATHGoogle Scholar
  7. 7.
    G. Schwarz, “Estimating the Dimension of a Model,” Annals Statistics 6, 461–464 (1978).zbMATHCrossRefGoogle Scholar
  8. 8.
    M. E. Tipping, “The Relevance Vector Machine,” Adv. Neural Inform. Proc. Systems 12, 652–658 (2000).Google Scholar
  9. 9.
    M. E. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine,” J. Mach. Learn. Res. 1, 211–244 (2001).zbMATHCrossRefMathSciNetGoogle Scholar
  10. 10.
    D. A. Kropotov and D. P. Vetrov, “On One Method of Non-Diagonal Regularization in Sparse Bayesian Learning,” in Proceedings of the 24th Annual International Conference on Machine Learning (ICML 2007), 2007, pp. 457–464.Google Scholar
  11. 11.
    D. Wipf and S. Nagarajan, “A New View of Automatic Relevance Determination,” Adv. Neural Inform. Proc. Systems 20, 1625–1632 (2007).Google Scholar
  12. 12.
    A. Asuncion and D. J. Newman, UCI Machine Learning Repository, Univ. California, Irvine, School of Information and Computer Sciences, 2007.
  13. 13.
    D. Kropotov, N. Ptashko, and D. Vetrov, “The Application of Continuous Akaike Information Criterion for Automatic Selection of Relevant Regressors,” in Proc. of 9th International Conference “Pattern Recognition and Image Analysis: New Information Technologies” (Nizhni Novgorod, 2008), Vol. 1, pp. 423–426.Google Scholar

Copyright information

© Pleiades Publishing, Ltd. 2009

Authors and Affiliations

  1. 1.Dorodnicyn Computing Centre of the Russian Academy of SciencesMoscowRussia
  2. 2.Lomonosov Moscow State UniversityMoscowRussia

Personalised recommendations