Generalized Augmentation of Multiple Kernels

  • Wan-Jui Lee
  • Robert P. W. Duin
  • Marco Loog
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6713)


Kernel combination is meant to improve the performance of single kernels and avoid the difficulty of kernel selection. The most common way of combining kernels is to compute their weighted sum. Usually, the kernels are assumed to exist in independent empirical feature spaces and therefore were combined without considering their relationships.

To take these relationships into consideration in kernel combination, we propose the generalized augmentation kernel which is extended by all the single kernels considering their correlations. The generalized augmentation kernel, unlike the weighted sum kernel, does not need to find out the weight of each kernel, and also would not suffer from information loss due to the average of kernels.

In the experiments, we observe that the generalized augmentation kernel usually can achieve better performances than other combination methods that do not consider relationship between kernels.


Support Vector Machine Radial Basis Function Kernel Multiple Kernel Machine Learn Research Empirical Feature 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bennett, K.P., Momma, M., Embrechts, M.J.: MARK: A Boosting Algorithm for Heterogeneous Kernel Models. In: Proc. 8th ACMSIGKDD Int. Conf. Knowledge Discovery and Data Mining, pp. 24–31 (2002)Google Scholar
  2. 2.
    Bousquet, O., Herrmann, D.: On the Complexity of Learning the Kernel Matrix. In: Proc. Advances in Neural Information Processing Systems, pp. 415–422 (2003)Google Scholar
  3. 3.
    Burges, C.J.C.: A Tutorial on Support Vector Machines for Pattern Recognition. Knowledge Discovery and Data Mining 2(2), 1–43 (1998)CrossRefGoogle Scholar
  4. 4.
    Camps-Valls, G., Gomez-Chova, L., Muñoz-Marí, J., Vila-Francés, J., Calpe-Maravilla, J.: Composite Kernels for Hyperspectral Image Classification. IEEE Geoscience and Remote Sensing Letters 3(1), 93–97 (2006)CrossRefGoogle Scholar
  5. 5.
    Chang, C.C., Lin, C.J.: LIBSVM: A Library for Support Vector Machines, Taiwan, National Taiwan University (2001),
  6. 6.
    Crammer, K., Keshet, J., Singer, Y.: Kernel Design Using Boosting. In: Proc. of the Fifteenth Annual Conference on Neural Information Processing Systems (2002)Google Scholar
  7. 7.
    Cristianini, N., Kandola, J., Elisseeff, A., Shawe-Taylor, J.: On Kernel Target Alignment. Technical Report NeuroColt, pp. 2001–2099. Royal Holloway University, London (2001)Google Scholar
  8. 8.
    de Diego, I.M., Moguerza, J.M., Mu noz, A.: Combining Kernel Information for Support Vector Classification. In: Proc. Multiple Classifier Systems, pp. 102–111 (2004)Google Scholar
  9. 9.
    Fung, G., Dundar, M., Bi, J., Rao, B.: A Fast Iterative Algorithm for Fisher Discriminant Using Heterogeneous Kernels. In: Proc. 21st Int. Conf. Machine Learning (2004)Google Scholar
  10. 10.
    Lanckriet, G.R.G., Cristianini, N., Bartlett, P., Ghaoui, L.E., Jordan, M.I.: Learning the Kernel Matrix with Semidefinite Programming. Journal of Machine Learning Research 5, 27–72 (2004)zbMATHGoogle Scholar
  11. 11.
    Lee, W.-J., Verzakov, S.A., Duin, R.P.W.: Kernel Combination Versus Classifier Combination. In: Proc. Multiple Classifier Systems, pp. 22–31 (2007)Google Scholar
  12. 12.
    Lin, C.T., Yeh, C.M., Liang, S.F., Chung, J.F., Kumar, N.: Support-Vector-Based Fuzzy Neural Network for Pattern Classification. IEEE Trans. on Fuzzy Systems 14(1), 31–41 (2006)CrossRefGoogle Scholar
  13. 13.
    Micchelli, C.A., Pontil, M.: Learning the Kernel Function via Regularization. Journal of Machine Learning Research 6, 1099–1125 (2005)zbMATHGoogle Scholar
  14. 14.
    Moguerza, J.M., Munoz, A., de Diego, I.M.: Improving Support Vector Classification via the Combination of Multiple Sources of Information. In: SSPR/SPR, pp. 592–600 (2004)Google Scholar
  15. 15.
    Asuncion, A., Newman, D.J.: UCI Machine Learning Repository. University of California, Department of Information and Computer Science, Irvine, CA (2007),
  16. 16.
    Jain, A.K., Ramaswami, M.D.: Classifier design with Parzen window. Pattern Recogition and Artificial Intelligence (1988)Google Scholar
  17. 17.
    Duin, R.P.W., Juszczak, P., Paclik, P., Pȩkalska, E., de Ridder, D., Tax, D.M.J.: PRTOOLS4, A Matlab Toolbox for Pattern Recognition, Delft University of Technology, Pattern Recognition Laboratory, The Netherlands (2004),
  18. 18.
    Ong, C.S., Smola, A.J., Williamson, R.C.: Learning the Kernel with Hyperkernels. Journal of Machine Learning Research 6, 1043–1071 (2005)zbMATHGoogle Scholar
  19. 19.
    Tsang, I.W.H., Kwok, J.T.Y.: Efficient Hyperkernel Learning Using Second-Order Cone Programming. IEEE Trans. on Neural Networks 17(1), 48–58 (2006)CrossRefzbMATHGoogle Scholar
  20. 20.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)CrossRefzbMATHGoogle Scholar
  21. 21.
    Yan, F., Mikolajczyk, K., Kittler, J., Tahir, M.A.: Combining Multiple Kernels by Augmenting the Kernel Matrix. In: Proc. Multiple Classifier Systems, pp. 175–184 (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Wan-Jui Lee
    • 1
  • Robert P. W. Duin
    • 1
  • Marco Loog
    • 1
  1. 1.Pattern Recognition LaboratoryDelft University of TechnologyThe Netherlands

Personalised recommendations