International Journal of Computer Vision

, Volume 70, Issue 1, pp 77–90 | Cite as

Unsupervised Learning of Image Manifolds by Semidefinite Programming

Article

Abstract

Can we detect low dimensional structure in high dimensional data sets of images? In this paper, we propose an algorithm for unsupervised learning of image manifolds by semidefinite programming. Given a data set of images, our algorithm computes a low dimensional representation of each image with the property that distances between nearby images are preserved. More generally, it can be used to analyze high dimensional data that lies on or near a low dimensional manifold. We illustrate the algorithm on easily visualized examples of curves and surfaces, as well as on actual images of faces, handwritten digits, and solid objects.

Keywords

manifold learning dimensionality reduction semidefinite programming kernel methods data analysis image manifolds semidefinite embedding 

References

  1. Belkin, M. and Niyogi, P. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396.CrossRefMATHGoogle Scholar
  2. Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522.CrossRefGoogle Scholar
  3. Bengio, Y., Paiement, J.F., and Vincent, P. 2004. Out-of-sample extensions for LLE, Isomap, MDS, eigenmaps, and spectral clustering. Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press.Google Scholar
  4. Beymer, D. and Poggio, T. 1996. Image representation for visual learning. Science, 272, 1905.Google Scholar
  5. Biswas, P. and Ye, Y. 2003. A distributed method for solving semidefinite programs arising from ad hoc wireless sensor network localization. Stanford University, Department of Electrical Engineering, working paper.Google Scholar
  6. Blitzer, J., Weinberger, K.Q., Saul, L.K., and Pereira, F.C.N. 2005. Hierarchical distributed representations for statistical language modeling. Advances in Neural and Information Processing Systems. Cambridge, MA: MIT Press.Google Scholar
  7. Borchers, B. 1999. CSDP, a C library for semidefinite programming. Optimization Methods and Software, 11(1):613–623.MATHMathSciNetGoogle Scholar
  8. Bowling, M., Ghodsi, A., and Wilkinson, D. 2005. Action respecting embedding. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). Bonn, Germany.Google Scholar
  9. Brand, M. 2003. Charting a manifold. Advances in Neural Information Processing Systems 15, Cambridge, MA: MIT Press, pp. 985–992.Google Scholar
  10. Burer, S. and Monteiro, R. 2003. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, Series, B95, pp. 329–357.Google Scholar
  11. Burges, C.J.C. 2005. Geometric methods for feature extraction and dimensional reduction. In L. Rokach and O. Maimon (eds.), Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers. Kluwer Academic Publishers.Google Scholar
  12. Cox, T. and Cox, M. 1994. Multidimensional Scaling. London: Chapman and Hall.Google Scholar
  13. Donoho, D.L. and Grimes, C.E. 2002. When Does Isomap Recover The Natural Parameterization of Families of Articulated Images? (Technical Report 2002–27). Department of Statistics. Stanford University.Google Scholar
  14. Donoho, D.L. and Grimes, C.E. 2003. Hessian eigenmaps: locally linear embedding techniques for high-dimensional data. In Proceedings of the National Academy of Arts and Sciences 100, pp. 5591–5596.Google Scholar
  15. Elgammal, A. and Lee, C. 2004. Separating style and content on a nonlinear manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2004), Washington DC, pp. 478–485.Google Scholar
  16. Fazel, M., Hindi, H., and Boyd, S.P. 2001. A rank minimization heuristic with application to minimum order system approximation. In Proceedings of the American Control Conference, pp. 4734–4739.Google Scholar
  17. Gordon, S., Goldberger, J., and Greenspan, H. 2003. Applying the information bottleneck principle to unsupervised clustering of discrete and continuous image representations. In Proceedings of the Ninth International Conference on Computer Vision (ICCV 2003), pp. 370–377.Google Scholar
  18. Ham, J., Lee, D.D., Mika, S., and Schölkopf, B. 2003. A kernel view of the dimensionality reduction of manifolds (Technical Report TR-110). Max-Planck-Institut für Biologische Kybernetik, Tübingen.Google Scholar
  19. Hull, J.J. 1994. A database for handwritten text recognition research. IEEE Transaction on Pattern Analysis and Machine Intelligence 16(5):550–554.CrossRefGoogle Scholar
  20. Jolliffe, I.T. 1986. Principal Component Analysis. New York: Springer-Verlag.Google Scholar
  21. Lanckriet, G.R.G., Christianini, N., Bartlett, P.L., Ghaoui, L.E., and Jordan, M.I. 2002. Learning the kernel matrix with semi-definite programming. In Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pp. 323–330.Google Scholar
  22. Lee, D.D., and Seung, H.S. 1999. Learning the parts of objects with nonnegative matrix factorization. Nature, 401:788–791.CrossRefGoogle Scholar
  23. Lee, K., Ho, J., Yang, M.-H., and Kriegman, D. 2003. Video-based face recognition using probabilistic appearance manifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2003), pp. 313–320.Google Scholar
  24. Lu, H., Fainman, Y., and Hecht-Nielsen, R. 1998. Image manifolds. Applications of Artificial Neural Networks in Image Processing III, Proceedings of SPIE, Bellingham, WA: SPIE, pp. 52–63.Google Scholar
  25. Nash, J. 1956. The imbedding problem for riemannian manifolds. Annals of Mathematics pp. 20–63.Google Scholar
  26. Ng, A.Y., Jordan, M., and Weiss, Y. 2002. On spectral clustering: analysis and an algorithm. Advances in Neural Information Processing Systems14 Cambridge, MA: MIT Press, pp. 849–856.Google Scholar
  27. Pless, R. 2004. Differential structure in non-linear image embedding functions. Proceedings of the IEEE Workshop on Articulated and Non-Rigid Motion, Washington, D.C. pp. 10–17.Google Scholar
  28. Pless, R. and Simon, I. 2002. Embedding images in non-flat spaces. In Proceedings of the Conference on Imaging Science Systems and Technology pp. 182–188.Google Scholar
  29. Roweis, S.T. and Saul, L.K. 2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323–2326.Google Scholar
  30. Saul, L.K. and Roweis, S.T. 2003. Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research, 4:119–155.MathSciNetCrossRefGoogle Scholar
  31. Schölkopf, B. and Smola, A.J. 2002. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA: MIT Press.Google Scholar
  32. Schölkopf, B., Smola, A.J., and Müller, K.R. 1998. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299–1319.CrossRefGoogle Scholar
  33. Seung, H.S. and Lee, D.D. 2000. The manifold ways of perception. Science, 290:2268–2269.CrossRefGoogle Scholar
  34. Sha, F. and Saul, L.K. 2005. Analysis and extension of spectral methods for nonlinear dimensionality reduction. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). Bonn, Germany.Google Scholar
  35. Shi, J. and Malik, J. 2000. Normalized cuts and image segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), pp. 888–905.Google Scholar
  36. Simard, P.Y., LeCun, Y., and Decker, J. 1993. Efficient pattern recognition using a new transformation distance. Advances in Neural Information Processing Systems, San Mateo, CA: Morgan Kaufman, pp. 50–58.Google Scholar
  37. Souvenir, R. and Pless, R. 2005. Isomap and non-parametric models of image deformation. In Proceedings of the IEEE Workshop on Motion and Video Computing, Breckenridge, CO., pp. 195–200.Google Scholar
  38. Sun, J., Boyd, S., Xiao, L., and Diaconis, P. 2004. The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem. Submitted to SIAM Review.Google Scholar
  39. Tenenbaum, J.B., de Silva, V., and Langford, J.C. 2000. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319–2323.CrossRefGoogle Scholar
  40. Turk, M. and Pentland, A. 1991. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–86.CrossRefGoogle Scholar
  41. Vandenberghe, L. and Boyd, S.P. 1996. Semidefinite programming. SIAM Review, 38(1):49–95.MathSciNetCrossRefMATHGoogle Scholar
  42. Vapnik, V. 1998. Statistical Learning Theory. N.Y.: Wiley.Google Scholar
  43. Weinberger, K.Q., Packer, B.D., and Saul, L.K. 2005. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics. Barbados, West Indies.Google Scholar
  44. Weinberger, K.Q. and Saul, L.K. 2004. Unsupervised learning of image manifolds by semidefinite programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), Washington D.C, pp. 988–995.Google Scholar
  45. Weinberger, K.Q., Sha, F., and Saul, L.K. 2004. Learning a kernel matrix for nonlinear dimensionality reduction. In Proceedings of the Twenty First International Conference on Machine Learning (ICML-04), Banff, Canada, pp. 839–846.Google Scholar
  46. Zha, H. and Zhang, Z. 2003. Isometric embedding and continuum Isomap. In Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 864–871.Google Scholar
  47. Zhang, Z. and Zha, H. 2004. Principal manifolds and nonlinear dimensionality reduction by local tangent space alignment. SIAM Journal of Scientific Computing, 26:313–338.MathSciNetCrossRefMATHGoogle Scholar

Copyright information

© Springer Science + Business Media, LLC 2006

Authors and Affiliations

  1. 1.Department of Computer and Information ScienceUniversity of PennsylvaniaPhiladelphia

Personalised recommendations