Abstract
In many learning problems prior knowledge about pattern variations can be formalized and beneficially incorporated into the analysis system. The corresponding notion of invariance is commonly used in conceptionally different ways. We propose a more distinguishing treatment in particular in the active field of kernel methods for machine learning and pattern analysis. Additionally, the fundamental relation of invariant kernels and traditional invariant pattern analysis by means of invariant representations will be clarified. After addressing these conceptional questions, we focus on practical aspects and present two generic approaches for constructing invariant kernels. The first approach is based on a technique called invariant integration. The second approach builds on invariant distances. In principle, our approaches support general transformations in particular covering discrete and non-group or even an infinite number of pattern-transformations. Additionally, both enable a smooth interpolation between invariant and non-invariant pattern analysis, i.e. they are a covering general framework. The wide applicability and various possible benefits of invariant kernels are demonstrated in different kernel methods.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Abe, S. (2003). On invariance of support vector machines. In Proceedings of the 4th international conference on intelligent data engineering and automated learning.
Alt, H., & Guibas, L. J. (1999). Discrete geometric shapes: Matching, interpolation, and approximation—a survey. In Handbook of computational geometry (pp. 121–153). North-Holland: Elsevier.
Bahlmann, C., Haasdonk, B., & Burkhardt, H. (2002). On-line handwriting recognition with support vector machines—a kernel approach. In Proceedings of the 8th international workshop on frontiers in handwriting recognition (pp. 49–54).
Berg, C., Christensen, J. P. R., & Ressel, P. (1984). Harmonic analysis on semigroups. Theory of positive definite and related functions. In Graduate texts in mathematics. New York: Springer.
Binford, T. O., & Levitt, T. S. (1993). Quasi-invariants: Theory and explotation. In Proceedings of DARPA image understanding workshop (pp. 819–829).
Burges, C. J. C. (1999). Geometry and invariance in kernel based methods. In B. Schölkopf, C. J. C. Burges & A. J. Smola (Eds.), Advances in kernel methods—support vector learning (pp. 89–116).
Burkhardt, H., & Siggelkow, S. (2001). Invariant features in pattern recognition—fundamentals and applications. In Nonlinear model-based image/video processing and analysis (pp. 269–307). New York: Wiley.
Canterakis, N. (1999). 3D Zernike moments and Zernike affine invariants for 3D image analysis and recognition. In Proceedings of the 11th Scandinavian conference on image analysis.
Chapelle, O., & Schölkopf, B. (2002). Incorporating invariances in nonlinear support vector machines. In Advances in neural information processing systems (Vol. 14, pp. 609–616).
Cortes, C., Haffner, P., & Mohri, M. (2003). Rational kernels. In Advances in neural information processing systems 15.
DeCoste, D., & Schölkopf, B. (2002). Training invariant support vector machines. Machine Learning, 46(1), 161–190.
Duda, R. O., Hart, P. E., & Stork, D. G. (2001). Pattern classification (2nd ed.). New York: Wiley.
Fitzgibbon, A. W., & Zisserman, A. (2003). Joint manifold distance: A new approach to appearance based clustering. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition.
Fung, G. M., Mangasarian, O. L., & Shavlik, J. W. (2004). Knowledge-based support vector machine classifiers. In Advances in neural information processing systems (Vol. 16).
Genton, M. G. (2001). Classes of kernels for machine learning: A statistics perspective. Journal of Machine Learning Research, 2, 299–312.
Graepel, T., & Herbrich, R. (2004). Invariant pattern recognition by semidefinite programming machines. In Advances in neural information processing systems (Vol. 16).
Haasdonk, B. (2005a). Feature space interpretation of svms with indefinite kernels. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4), 482–492.
Haasdonk, B. (2005b). KerMet-tools: A MATLAB invariant kernel method toolbox. Software available at http://lmb.informatik.uni-freiburg.de/people/haasdonk/KerMet-Tools.
Haasdonk, B. (2005c). Transformation knowledge in pattern analysis with kernel methods—distance and integration kernels. PhD thesis, Computer Science Department, University of Freiburg, Germany.
Haasdonk, B., & Keysers, D. (2002). Tangent distance kernels for support vector machines. In Proceedings of the 16th international conference on pattern recognition (Vol. 2, pp. 864–868).
Haasdonk, B., Vossen, A., & Burkhardt, H. (2005). Invariance in kernel methods by Haar-integration kernels. In Proceedings of the 14th Scandinavian conference on image analysis (pp. 841–851).
Haussler, D. (1999). Convolution kernels on discrete structures. Technical report UCS-CRL-99-10, UC Santa Cruz.
Herbrich, R. (2002). Learning kernel classifiers. Cambridge: MIT.
Joachims, T. (1999). Estimating the generalization performance of a SVM efficiently. Technical Report LS8 Report 25, University of Dortmund, Germany.
Keysers, D., Dahmen, J., Theiner, T., & Ney, H. (2000). Experiments with an extended tangent distance. In Proceedings of the 15th international conference on pattern recognition (Vol. 2, pp. 38–42).
Keysers, D., Gollan, C., & Ney, H. (2004). Local context in non-linear deformation models for handwritten character recognition. In Proceedings of the 17th international conference on pattern recognition.
Leen, T. K. (1995). From data distributions to regularization in invariant learning. In Advances in neural information processing systems (Vol. 7).
Lenz, R. (1991). Group theoretical feature extraction: Weighted invariance and texture analysis. In Proceedings of the 7th Scandinavian conference on image analysis (pp. 63–70).
Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., & Watkins, C. (2002). Text classification using string kernels. Journal of Machine Learning Research, 2, 419–444.
Loosli, G., Canu, S., Vishwanathan, S., & Smola, A. (2005). Invariances in classification: An efficient SVM implementation. In Proceedings of the 11th international symposium on applied stochastic models and data analysis.
Mika, S., Rätsch, G., Schölkopf, B., Smola, A., Weston, J., & Müller, K.-R. (2000). Invariant feature extraction and classification in kernel spaces. In Advances in neural information processing systems (Vol. 12).
Mundy, J. L., Zisserman, A., & Forsyth, D. (1994). Applications of invariance in computer vision. In Proceedings of the of 2nd joint European–US workshop, 1993. New York: Springer.
Nachbin, L. (1965). The Haar integral. Princeton: Van Nostrand.
Ong, C. S., Mary, X., Canu, S., & Smola, A. J. (2004). Learning with non-positive kernels. In Proceedings of the 21st international conference on machine learning (pp. 639–646).
Peschke, K., Haasdonk, B., Ronneberger, O., Burkhardt, H., Rösch, P., Harz, M., & Popp, J. (2006). Using transformation knowledge for the classification of Raman spectra of biological samples. In Proceedings of the 4th IASTED international conference on biomedical engineering (pp. 288–293).
Pozdnoukhov, A., & Bengio, S. (2004). Tangent vector kernels for invariant image classification with SVMs. In Proceedings of the 17th international conference on pattern recognition.
Rabiner, L., & Juang, B. (1993). Fundamentals of speech recognition. New York: Prentice Hall.
Sahbi, H., & Fleuret, F. (2002). Scale-invariance of support vector machines based on the triangular kernel. Technical Report RR-4601, INRIA.
Schölkopf, B. (2001). The kernel trick for distances. In Advances in neural information processing systems (Vol. 13, pp. 301–307).
Schölkopf, B., & Smola, A. J. (2002). Learning with kernels: Support vector machines, regularization, optimization and beyond. Cambridge: MIT.
Schölkopf, B., Burges, C., & Vapnik, V. (1996). Incorporating invariances in support vector learning machines. In Proceedings of the 6th international conference on artificial neural networks (pp. 47–52).
Schölkopf, B., Simard, P., Smola, A., & Vapnik, V. (1998). Prior knowledge in support vector kernels. In Advances in neural information processing systems (Vol. 10, pp. 640–646).
Schölkopf, B., Smola, A., Williamson, R., & Bartlett, P. (2000). New support vector algorithms. Neural Computation, 12, 1083–1121.
Schulz-Mirbach, H. (1994). Constructing invariant features by averaging techniques. In Proceedings of the 12th international conference on pattern recognition (Vol. 2, pp. 387–390).
Schulz-Mirbach, H. (1995). Anwendung von Invarianzprinzipien zur Merkmalgewinnung in der Mustererkennung. PhD thesis, Technical University Hamburg-Harburg, Germany.
Schur, I. (1968). Vorlesungen über Invariantentheorie. Berlin: Springer.
Shawe-Taylor, J., & Cristianini, N. (2004). Kernel methods for pattern analysis. Cambridge: Cambridge University Press.
Simard, P. Y., LeCun, Y. A., & Denker, J. S. (1993). Efficient pattern recognition using a new transformation distance. In Advances in neural information processing systems (Vol. 5, pp. 50–58).
Simard, P. Y., LeCun, Y. A., Denker, J. S., & Victorri, B. (1998). Transformation invariance in pattern recognition—tangent distance and tangent propagation. In Neural networks: Tricks of the trade (pp. 239–274).
Vasconcelos, N., & Lippman, A. (1998). Multiresolution tangent distance for affine-invariant classification. In Advances in neural information processing systems (Vol. 10).
Veltkamp, R. C. (2001). Shape matching: Similarity measures and algorithms. Technical Report UU-CS-2001-03, Department of Computing Science, Utrecht University, The Netherlands.
Vishwanathan, S. V. N., Vidal, R., & Smola, A. J. (2005). Binet–Cauchy kernels on dynamical systems. Technical report PA005796, National ICT Australia.
Watkins, C. (2000). Dynamic alignment kernels. In Advances in large margin classifiers (pp. 39–50). Cambridge: MIT Press.
Werman, M., & Weinshall, D. (1995). Similarity and affine invariant distances between 2D point sets. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(8), 810–814.
Wood, J. (1996). Invariant pattern recognition: A review. Pattern Recognition, 29(1), 1–17.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: Phil Long.
Rights and permissions
About this article
Cite this article
Haasdonk, B., Burkhardt, H. Invariant kernel functions for pattern analysis and machine learning. Mach Learn 68, 35–61 (2007). https://doi.org/10.1007/s10994-007-5009-7
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-007-5009-7