Incremental Relevance Vector Machine with Kernel Learning

  • Dimitris Tzikas
  • Aristidis Likas
  • Nikolaos Galatsanos
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5138)


Recently, sparse kernel methods such as the Relevance Vector Machine (RVM) have become very popular for solving regression problems. The sparsity and performance of these methods depend on selecting an appropriate kernel function, which is typically achieved using a cross-validation procedure. In this paper we propose a modification to the incremental RVM learning method, that also learns the location and scale parameters of Gaussian kernels during model training. More specifically, in order to effectively model signals with different characteristics at various locations, we learn different parameter values for each kernel, resulting in a very flexible model. In order to avoid overfitting we use a sparsity enforcing prior that controls the effective number of parameters of the model. Finally, we apply the proposed method to one-dimensional and two-dimensional artificial signals, and evaluate its performance on two real-world datasets.


Basis Function Mean Square Error Marginal Likelihood Incremental Algorithm Relevance Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Tipping, M.E.: Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1, 211–244 (2001)zbMATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  3. 3.
    Lanckriet, G.R.G., Cristianini, N., Bartlett, P., Ghaoui, L.E., Jordan, M.I.: Learning the kernel matrix with semidefinite programming. J. Mach. Learn. Res. 5, 27–72 (2004)Google Scholar
  4. 4.
    Girolami, M., Rogers, S.: Hierarchic Bayesian models for kernel learning. In: ICML 2005: Proceedings of the 22nd international conference on machine learning, pp. 241–248. ACM, New York (2005)CrossRefGoogle Scholar
  5. 5.
    Sonnenburg, S., Rätsch, G., Schäfer, C., Schölkopf, B.: Large scale multiple kernel learning. J. Mach. Learn. Res. 7, 1531–1565 (2006)MathSciNetGoogle Scholar
  6. 6.
    Quiñonero-Candela, J., Hansen, L.K.: Time series prediction based on the relevance vector machine with adaptive kernels. In: Proceedings of the International Conference on Acoustics, Speech, and Signal Processing, Piscataway, New Jersey, vol. 1, pp. 985–988. IEEE, Los Alamitos (2002)Google Scholar
  7. 7.
    Krishnapuram, B., Hartemink, A.J., Figueiredo, M.A.T.: A Bayesian approach to joint feature selection and classifier design. IEEE Trans. Pattern Anal. Mach. Intell. 26(9), 1105–1111 (2004)CrossRefGoogle Scholar
  8. 8.
    Schmolck, A., Everson, R.: Smooth relevance vector machine: a smoothness prior extension of the RVM. Machine Learning 68(2), 107–135 (2007)CrossRefGoogle Scholar
  9. 9.
    Tipping, M., Faul, A.: Fast marginal likelihood maximisation for sparse Bayesian models. In: Proc. of the Ninth International Workshop on Artificial Intelligence and Statistics (2003)Google Scholar
  10. 10.
    Faul, A.C., Tipping, M.E.: Analysis of sparse Bayesian learning. In: Advances in Neural Information Processing Systems, pp. 383–389. MIT Press, Cambridge (2001)Google Scholar
  11. 11.
    Holmes, C.C., Denison, D.G.T.: Bayesian wavelet analysis with a model complexity prior. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics 6: Proceedings of the Sixth Valencia International Meeting. Oxford University Press, Oxford (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Dimitris Tzikas
    • 1
  • Aristidis Likas
    • 1
  • Nikolaos Galatsanos
    • 2
  1. 1.Department of Computer ScienceUniversity of IoanninaIoanninaGreece
  2. 2.Department of Electrical EngineeringUniversity of PatrasRioGreece

Personalised recommendations