Skip to main content
SpringerLink
Log in

Unsupervised Learning of Image Manifolds by Semidefinite Programming

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Can we detect low dimensional structure in high dimensional data sets of images? In this paper, we propose an algorithm for unsupervised learning of image manifolds by semidefinite programming. Given a data set of images, our algorithm computes a low dimensional representation of each image with the property that distances between nearby images are preserved. More generally, it can be used to analyze high dimensional data that lies on or near a low dimensional manifold. We illustrate the algorithm on easily visualized examples of curves and surfaces, as well as on actual images of faces, handwritten digits, and solid objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Belkin, M. and Niyogi, P. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396.

    Article  MATH  Google Scholar 

  • Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522.

    Article  Google Scholar 

  • Bengio, Y., Paiement, J.F., and Vincent, P. 2004. Out-of-sample extensions for LLE, Isomap, MDS, eigenmaps, and spectral clustering. Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press.

  • Beymer, D. and Poggio, T. 1996. Image representation for visual learning. Science, 272, 1905.

  • Biswas, P. and Ye, Y. 2003. A distributed method for solving semidefinite programs arising from ad hoc wireless sensor network localization. Stanford University, Department of Electrical Engineering, working paper.

  • Blitzer, J., Weinberger, K.Q., Saul, L.K., and Pereira, F.C.N. 2005. Hierarchical distributed representations for statistical language modeling. Advances in Neural and Information Processing Systems. Cambridge, MA: MIT Press.

  • Borchers, B. 1999. CSDP, a C library for semidefinite programming. Optimization Methods and Software, 11(1):613–623.

    MATH  MathSciNet  Google Scholar 

  • Bowling, M., Ghodsi, A., and Wilkinson, D. 2005. Action respecting embedding. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). Bonn, Germany.

  • Brand, M. 2003. Charting a manifold. Advances in Neural Information Processing Systems 15, Cambridge, MA: MIT Press, pp. 985–992.

    Google Scholar 

  • Burer, S. and Monteiro, R. 2003. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, Series, B95, pp. 329–357.

  • Burges, C.J.C. 2005. Geometric methods for feature extraction and dimensional reduction. In L. Rokach and O. Maimon (eds.), Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers. Kluwer Academic Publishers.

  • Cox, T. and Cox, M. 1994. Multidimensional Scaling. London: Chapman and Hall.

  • Donoho, D.L. and Grimes, C.E. 2002. When Does Isomap Recover The Natural Parameterization of Families of Articulated Images? (Technical Report 2002–27). Department of Statistics. Stanford University.

  • Donoho, D.L. and Grimes, C.E. 2003. Hessian eigenmaps: locally linear embedding techniques for high-dimensional data. In Proceedings of the National Academy of Arts and Sciences 100, pp. 5591–5596.

  • Elgammal, A. and Lee, C. 2004. Separating style and content on a nonlinear manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2004), Washington DC, pp. 478–485.

  • Fazel, M., Hindi, H., and Boyd, S.P. 2001. A rank minimization heuristic with application to minimum order system approximation. In Proceedings of the American Control Conference, pp. 4734–4739.

  • Gordon, S., Goldberger, J., and Greenspan, H. 2003. Applying the information bottleneck principle to unsupervised clustering of discrete and continuous image representations. In Proceedings of the Ninth International Conference on Computer Vision (ICCV 2003), pp. 370–377.

  • Ham, J., Lee, D.D., Mika, S., and Schölkopf, B. 2003. A kernel view of the dimensionality reduction of manifolds (Technical Report TR-110). Max-Planck-Institut für Biologische Kybernetik, Tübingen.

  • Hull, J.J. 1994. A database for handwritten text recognition research. IEEE Transaction on Pattern Analysis and Machine Intelligence 16(5):550–554.

    Article  Google Scholar 

  • Jolliffe, I.T. 1986. Principal Component Analysis. New York: Springer-Verlag.

  • Lanckriet, G.R.G., Christianini, N., Bartlett, P.L., Ghaoui, L.E., and Jordan, M.I. 2002. Learning the kernel matrix with semi-definite programming. In Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pp. 323–330.

  • Lee, D.D., and Seung, H.S. 1999. Learning the parts of objects with nonnegative matrix factorization. Nature, 401:788–791.

    Article  Google Scholar 

  • Lee, K., Ho, J., Yang, M.-H., and Kriegman, D. 2003. Video-based face recognition using probabilistic appearance manifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2003), pp. 313–320.

  • Lu, H., Fainman, Y., and Hecht-Nielsen, R. 1998. Image manifolds. Applications of Artificial Neural Networks in Image Processing III, Proceedings of SPIE, Bellingham, WA: SPIE, pp. 52–63.

  • Nash, J. 1956. The imbedding problem for riemannian manifolds. Annals of Mathematics pp. 20–63.

  • Ng, A.Y., Jordan, M., and Weiss, Y. 2002. On spectral clustering: analysis and an algorithm. Advances in Neural Information Processing Systems14 Cambridge, MA: MIT Press, pp. 849–856.

  • Pless, R. 2004. Differential structure in non-linear image embedding functions. Proceedings of the IEEE Workshop on Articulated and Non-Rigid Motion, Washington, D.C. pp. 10–17.

  • Pless, R. and Simon, I. 2002. Embedding images in non-flat spaces. In Proceedings of the Conference on Imaging Science Systems and Technology pp. 182–188.

  • Roweis, S.T. and Saul, L.K. 2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323–2326.

    Google Scholar 

  • Saul, L.K. and Roweis, S.T. 2003. Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research, 4:119–155.

    Article  MathSciNet  Google Scholar 

  • Schölkopf, B. and Smola, A.J. 2002. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA: MIT Press.

  • Schölkopf, B., Smola, A.J., and Müller, K.R. 1998. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299–1319.

    Article  Google Scholar 

  • Seung, H.S. and Lee, D.D. 2000. The manifold ways of perception. Science, 290:2268–2269.

    Article  Google Scholar 

  • Sha, F. and Saul, L.K. 2005. Analysis and extension of spectral methods for nonlinear dimensionality reduction. In Proceedings of the Twenty-second International Conference on Machine Learning (ICML 2005). Bonn, Germany.

  • Shi, J. and Malik, J. 2000. Normalized cuts and image segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), pp. 888–905.

  • Simard, P.Y., LeCun, Y., and Decker, J. 1993. Efficient pattern recognition using a new transformation distance. Advances in Neural Information Processing Systems, San Mateo, CA: Morgan Kaufman, pp. 50–58.

  • Souvenir, R. and Pless, R. 2005. Isomap and non-parametric models of image deformation. In Proceedings of the IEEE Workshop on Motion and Video Computing, Breckenridge, CO., pp. 195–200.

  • Sun, J., Boyd, S., Xiao, L., and Diaconis, P. 2004. The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem. Submitted to SIAM Review.

  • Tenenbaum, J.B., de Silva, V., and Langford, J.C. 2000. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319–2323.

    Article  Google Scholar 

  • Turk, M. and Pentland, A. 1991. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–86.

    Article  Google Scholar 

  • Vandenberghe, L. and Boyd, S.P. 1996. Semidefinite programming. SIAM Review, 38(1):49–95.

    Article  MathSciNet  MATH  Google Scholar 

  • Vapnik, V. 1998. Statistical Learning Theory. N.Y.: Wiley.

  • Weinberger, K.Q., Packer, B.D., and Saul, L.K. 2005. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics. Barbados, West Indies.

  • Weinberger, K.Q. and Saul, L.K. 2004. Unsupervised learning of image manifolds by semidefinite programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR-04), Washington D.C, pp. 988–995.

  • Weinberger, K.Q., Sha, F., and Saul, L.K. 2004. Learning a kernel matrix for nonlinear dimensionality reduction. In Proceedings of the Twenty First International Conference on Machine Learning (ICML-04), Banff, Canada, pp. 839–846.

  • Zha, H. and Zhang, Z. 2003. Isometric embedding and continuum Isomap. In Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 864–871.

  • Zhang, Z. and Zha, H. 2004. Principal manifolds and nonlinear dimensionality reduction by local tangent space alignment. SIAM Journal of Scientific Computing, 26:313–338.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 19104-6389

    Kilian Q. Weinberger & Lawrence K. Saul

Authors
  1. Kilian Q. Weinberger
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lawrence K. Saul
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kilian Q. Weinberger.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Weinberger, K.Q., Saul, L.K. Unsupervised Learning of Image Manifolds by Semidefinite Programming. Int J Comput Vision 70, 77–90 (2006). https://doi.org/10.1007/s11263-005-4939-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-005-4939-z

Keywords

Search

Navigation