Advertisement

A Tale of Two Classifiers: SNoW vs. SVM in Visual Recognition

  • Ming-Hsuan Yang
  • Dan Roth
  • Narendra Ahuja
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2353)

Abstract

Numerous statistical learning methods have been developed for visual recognition tasks. Few attempts, however, have been made to address theoretical issues, and in particular, study the suitability of different learning algorithms for visual recognition. Large margin classifiers, such as SNoW and SVM, have recently demonstrated their success in object detection and recognition. In this paper, we present a theoretical account of these two learning approaches, and their suitability to visual recognition. Using tools from computational learning theory, we show that the main difference between the generalization bounds of SVM and SNoW depends on the properties of the data. We argue that learning problems in the visual domain have sparseness characteristics and exhibit them by analyzing data taken from face detection experiments. Experimental results exhibit good generalization and robustness properties of the SNoW-based method, and conform to the theoretical analysis.

Keywords

Object Recognition Object Detection Target Node Feature Representation Linear Feature 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    M. Aizerman, E. Braverman, and L. Rozonoer. Theoretical foundations of the potential function method in pattern recognition learning. Automation and Remote Control, 25:821–837, 1964.MathSciNetGoogle Scholar
  2. 2.
    Y. Amit and D. Geman. A computational model for visual selection. Neural Computation, 11(7):1691–1715, 1999.CrossRefGoogle Scholar
  3. 3.
    S. Ben-David and H. U. Simon. Efficient learning of linear perceptron. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 189–195. MIT Press, 2001.Google Scholar
  4. 4.
    A. Blum. Learning boolean functions in an infinite attribute space. Machine Learning, 9(4):373–386, 1992.zbMATHGoogle Scholar
  5. 5.
    B. E. Boser, I. M. Guyon, and V. Vapnik. A training algorithm for optimal margin classifiers. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, pages 144–152, 1992.Google Scholar
  6. 6.
    A. Carleson, C. Cumby, J. Rosen, and D. Roth. The SNoW learning architecture. Technical Report UIUCDCS-R-99-2101, UIUC Computer Science Department, 1999.Google Scholar
  7. 7.
    N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and other Kernel-based learning methods. Cambridge University Press, 2000.Google Scholar
  8. 8.
    J. De Bonet and P. Viola. Texture recognition using a non-parametric multi-scale statistical model. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 641–647, 1998.Google Scholar
  9. 9.
    G. Donato, M. S. Bartlett, J. C. Hager, P. Ekman, and T. J. Sejnowski. Classifying facial actions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(10):974–989, 2000.CrossRefGoogle Scholar
  10. 10.
    Y. Freund and R. Schapire. Large margin classification using the perceptron. Machine Learning, 37(3):277–296, 1999.zbMATHCrossRefGoogle Scholar
  11. 11.
    T. Graepel, R. Herbrich, and R. C. Williamson. From margin to sparsity. In Advances in Neural Information Processing Systems 13, pages 210–216. MIT Press, 2001.Google Scholar
  12. 12.
    B. Heisele, T. Poggio, and M. Pontil. Face detection in still gray images. Technical Report AI Memo 1687, MIT AI Lab, 2000.Google Scholar
  13. 13.
    J. Kivinen, M. K. Warmuth, and P. Auer. The Perceptron algorithm vs. Winnow: linear vs. logarithmic mistake bound when few input variables are relevant. Artificial Intelligence, 1–2:325–343, 1997.MathSciNetCrossRefGoogle Scholar
  14. 14.
    Y. Le Cun, L. Jackel, L. Bottou, A. Brunot, C. Cortes, J. Denker, H. Drucker, I. Guyon, U. Müller, E. Säckinger, P. Simard, and V. Vapnik. Comparison of learning algorithms for handwritten digit recognition. In Proceedings of International Conference on Artificial Neural Networks, pages 53–60, 1995.Google Scholar
  15. 15.
    N. Littlestone. Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm. Machine Learning, 2:285–318, 1988.Google Scholar
  16. 16.
    N. Littlestone. Redundant noisy attributes, attribute errors, and linear threshold learning using winnow. In Proceedings of the fourth Annual Workshop on Computational Learning Theory, pages 147–156, 1991.Google Scholar
  17. 17.
    B. W. Mel and J. Fiser. Minimizing binding errors using learned conjunctive features. Neural Computation, 12:247–278, 2000.CrossRefGoogle Scholar
  18. 18.
    A. Mohan, C. Papageorgiou, and T. Poggio. Example-based object detection in images by components. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(4):349–361, 2001.CrossRefGoogle Scholar
  19. 19.
    E. Osuna, R. Freund, and F. Girosi. Training support vector machines: An application to face detection. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 130–136, 1997.Google Scholar
  20. 20.
    C. Papageorgiou and T. Poggio. A trainable system for object detection. International Journal of Computer Vision, 38(1):15–33, 2000.zbMATHCrossRefGoogle Scholar
  21. 21.
    P. Penev and J. Atick. Local feature analysis: A general statistical theory for object representation. Network: Computation in Neural Systems, 7(3):477–500, 1996.zbMATHCrossRefGoogle Scholar
  22. 22.
    T. Poggio and S. Edelman. A network that learns to recognize 3D objects. Nature, 343:263–266, 1990.CrossRefGoogle Scholar
  23. 23.
    M. Pontil and A. Verri. Support vector machines for 3d object recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(6):637–646, 1998.CrossRefGoogle Scholar
  24. 24.
    T. Rikert, M. Jones, and P. Viola. A cluster-based statistical model for object detection. In Proceedings of the Seventh IEEE International Conference on Computer Vision, pages 1046–1053, 1999.Google Scholar
  25. 25.
    F. Rosenblatt. The Perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65:386–407, 1958.MathSciNetCrossRefGoogle Scholar
  26. 26.
    D. Roth. Learning to resolve natural language ambiguities: A unified approach. In Proceedings of the Fifteenth National Conference on Artificial Intelligence, pages 806–813, 1998.Google Scholar
  27. 27.
    D. Roth, M.-H. Yang, and N. Ahuja. Learning to recognize objects. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 724–731, 2000.Google Scholar
  28. 28.
    H. Rowley, S. Baluja, and T. Kanade. Neural network-based face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1):23–38, 1998.CrossRefGoogle Scholar
  29. 29.
    H. Schneiderman. A Statistical Approach to 3D Object Detection Applied to Faces and Cars. PhD thesis, Carnegie Mellon University, 2000.Google Scholar
  30. 30.
    H. Schneiderman and T. Kanade. Probabilistic modeling of local appearance and spatial relationships for object recognition. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 45–51, 1998.Google Scholar
  31. 31.
    K.-K. Sung and T. Poggio. Example-based learning for view-based human face detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(1):39–51, 1998.CrossRefGoogle Scholar
  32. 32.
    Y. W. Teh and G. E. Hinton. Rate-coded restricted Boltzmann machines for face recognition. In T. K. Leen, T. G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 908–914. MIT Press, 2001.Google Scholar
  33. 33.
    K. Tieu and P. Viola. Boosting image retrieval. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, volume 1, pages 228–235, 2000.Google Scholar
  34. 34.
    M. Turk and A. Pentland. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–86, 1991.CrossRefGoogle Scholar
  35. 35.
    L. G. Valiant. A theory of the learnable. Communications of the ACM, 27(11):1134–1142, Nov. 1984.Google Scholar
  36. 36.
    V. N. Vapnik. The Nature of Statistical Learning Theory. Springer-Verlag, New York, 1995.zbMATHCrossRefGoogle Scholar
  37. 37.
    M.-H. Yang, D. Roth, and N. Ahuja. Learning to recognize 3D objects with SNoW. In Proceedings of European Conference on Computer Vision, volume 1, pages 439–454, 2000.Google Scholar
  38. 38.
    M.-H. Yang, D. Roth, and N. Ahuja. A SNoW-based face detector. In S. A. Solla, T. K. Leen, and K.-R. Müller, editors, Advances of Neural Information Processing Systems, pages 855–861. MIT Press, 2000.Google Scholar
  39. 39.
    T. Zhang. Some theoretical results concerning the convergence of compositions of regularized linear functions. In S. A. Solla, T. K. Leen, and K.-R. Müller, editors, Advances in Neural Information Processing Systems 12, pages 370–376. MIT Press, 2000.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Ming-Hsuan Yang
    • 1
  • Dan Roth
    • 2
  • Narendra Ahuja
    • 3
  1. 1.Honda Fundamental Research Labs
  2. 2.Beckman Institute and Department of Computer ScienceUniversity of Illinois at Urbana-ChampaignUrbana
  3. 3.Beckman Institute and Department of Electrical and Computer EngineeringUniversity of Illinois at Urbana-ChampaignUrbana

Personalised recommendations