Find out how to access previewonly content
Date:
01 May 2006
Unsupervised Learning of Image Manifolds by Semidefinite Programming
 Kilian Q. Weinberger,
 Lawrence K. Saul
 … show all 2 hide
Rent the article at a discount
Rent now* Final gross prices may vary according to local VAT.
Get AccessAbstract
Can we detect low dimensional structure in high dimensional data sets of images? In this paper, we propose an algorithm for unsupervised learning of image manifolds by semidefinite programming. Given a data set of images, our algorithm computes a low dimensional representation of each image with the property that distances between nearby images are preserved. More generally, it can be used to analyze high dimensional data that lies on or near a low dimensional manifold. We illustrate the algorithm on easily visualized examples of curves and surfaces, as well as on actual images of faces, handwritten digits, and solid objects.
 Belkin, M. and Niyogi, P. 2003. Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15(6):1373–1396. CrossRef
 Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(4):509–522. CrossRef
 Bengio, Y., Paiement, J.F., and Vincent, P. 2004. Outofsample extensions for LLE, Isomap, MDS, eigenmaps, and spectral clustering. Advances in Neural Information Processing Systems 16. Cambridge, MA: MIT Press.
 Beymer, D. and Poggio, T. 1996. Image representation for visual learning. Science, 272, 1905.
 Biswas, P. and Ye, Y. 2003. A distributed method for solving semidefinite programs arising from ad hoc wireless sensor network localization. Stanford University, Department of Electrical Engineering, working paper.
 Blitzer, J., Weinberger, K.Q., Saul, L.K., and Pereira, F.C.N. 2005. Hierarchical distributed representations for statistical language modeling. Advances in Neural and Information Processing Systems. Cambridge, MA: MIT Press.
 Borchers, B. 1999. CSDP, a C library for semidefinite programming. Optimization Methods and Software, 11(1):613–623.
 Bowling, M., Ghodsi, A., and Wilkinson, D. 2005. Action respecting embedding. In Proceedings of the Twentysecond International Conference on Machine Learning (ICML 2005). Bonn, Germany.
 Brand, M. 2003. Charting a manifold. Advances in Neural Information Processing Systems 15, Cambridge, MA: MIT Press, pp. 985–992.
 Burer, S. and Monteiro, R. 2003. A nonlinear programming algorithm for solving semidefinite programs via lowrank factorization. Mathematical Programming, Series, B95, pp. 329–357.
 Burges, C.J.C. 2005. Geometric methods for feature extraction and dimensional reduction. In L. Rokach and O. Maimon (eds.), Data Mining and Knowledge Discovery Handbook: A Complete Guide for Practitioners and Researchers. Kluwer Academic Publishers.
 Cox, T. and Cox, M. 1994. Multidimensional Scaling. London: Chapman and Hall.
 Donoho, D.L. and Grimes, C.E. 2002. When Does Isomap Recover The Natural Parameterization of Families of Articulated Images? (Technical Report 2002–27). Department of Statistics. Stanford University.
 Donoho, D.L. and Grimes, C.E. 2003. Hessian eigenmaps: locally linear embedding techniques for highdimensional data. In Proceedings of the National Academy of Arts and Sciences 100, pp. 5591–5596.
 Elgammal, A. and Lee, C. 2004. Separating style and content on a nonlinear manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2004), Washington DC, pp. 478–485.
 Fazel, M., Hindi, H., and Boyd, S.P. 2001. A rank minimization heuristic with application to minimum order system approximation. In Proceedings of the American Control Conference, pp. 4734–4739.
 Gordon, S., Goldberger, J., and Greenspan, H. 2003. Applying the information bottleneck principle to unsupervised clustering of discrete and continuous image representations. In Proceedings of the Ninth International Conference on Computer Vision (ICCV 2003), pp. 370–377.
 Ham, J., Lee, D.D., Mika, S., and Schölkopf, B. 2003. A kernel view of the dimensionality reduction of manifolds (Technical Report TR110). MaxPlanckInstitut für Biologische Kybernetik, Tübingen.
 Hull, J.J. 1994. A database for handwritten text recognition research. IEEE Transaction on Pattern Analysis and Machine Intelligence 16(5):550–554. CrossRef
 Jolliffe, I.T. 1986. Principal Component Analysis. New York: SpringerVerlag.
 Lanckriet, G.R.G., Christianini, N., Bartlett, P.L., Ghaoui, L.E., and Jordan, M.I. 2002. Learning the kernel matrix with semidefinite programming. In Proceedings of the Nineteenth International Conference on Machine Learning (ICML 2002), pp. 323–330.
 Lee, D.D., and Seung, H.S. 1999. Learning the parts of objects with nonnegative matrix factorization. Nature, 401:788–791. CrossRef
 Lee, K., Ho, J., Yang, M.H., and Kriegman, D. 2003. Videobased face recognition using probabilistic appearance manifolds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2003), pp. 313–320.
 Lu, H., Fainman, Y., and HechtNielsen, R. 1998. Image manifolds. Applications of Artificial Neural Networks in Image Processing III, Proceedings of SPIE, Bellingham, WA: SPIE, pp. 52–63.
 Nash, J. 1956. The imbedding problem for riemannian manifolds. Annals of Mathematics pp. 20–63.
 Ng, A.Y., Jordan, M., and Weiss, Y. 2002. On spectral clustering: analysis and an algorithm. Advances in Neural Information Processing Systems14 Cambridge, MA: MIT Press, pp. 849–856.
 Pless, R. 2004. Differential structure in nonlinear image embedding functions. Proceedings of the IEEE Workshop on Articulated and NonRigid Motion, Washington, D.C. pp. 10–17.
 Pless, R. and Simon, I. 2002. Embedding images in nonflat spaces. In Proceedings of the Conference on Imaging Science Systems and Technology pp. 182–188.
 Roweis, S.T. and Saul, L.K. 2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290:2323–2326.
 Saul, L.K. and Roweis, S.T. 2003. Think globally, fit locally: unsupervised learning of low dimensional manifolds. Journal of Machine Learning Research, 4:119–155. CrossRef
 Schölkopf, B. and Smola, A.J. 2002. Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA: MIT Press.
 Schölkopf, B., Smola, A.J., and Müller, K.R. 1998. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation, 10:1299–1319. CrossRef
 Seung, H.S. and Lee, D.D. 2000. The manifold ways of perception. Science, 290:2268–2269. CrossRef
 Sha, F. and Saul, L.K. 2005. Analysis and extension of spectral methods for nonlinear dimensionality reduction. In Proceedings of the Twentysecond International Conference on Machine Learning (ICML 2005). Bonn, Germany.
 Shi, J. and Malik, J. 2000. Normalized cuts and image segmentation. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), pp. 888–905.
 Simard, P.Y., LeCun, Y., and Decker, J. 1993. Efficient pattern recognition using a new transformation distance. Advances in Neural Information Processing Systems, San Mateo, CA: Morgan Kaufman, pp. 50–58.
 Souvenir, R. and Pless, R. 2005. Isomap and nonparametric models of image deformation. In Proceedings of the IEEE Workshop on Motion and Video Computing, Breckenridge, CO., pp. 195–200.
 Sun, J., Boyd, S., Xiao, L., and Diaconis, P. 2004. The fastest mixing Markov process on a graph and a connection to a maximum variance unfolding problem. Submitted to SIAM Review.
 Tenenbaum, J.B., de Silva, V., and Langford, J.C. 2000. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319–2323. CrossRef
 Turk, M. and Pentland, A. 1991. Eigenfaces for recognition. Journal of Cognitive Neuroscience, 3(1):71–86. CrossRef
 Vandenberghe, L. and Boyd, S.P. 1996. Semidefinite programming. SIAM Review, 38(1):49–95. CrossRef
 Vapnik, V. 1998. Statistical Learning Theory. N.Y.: Wiley.
 Weinberger, K.Q., Packer, B.D., and Saul, L.K. 2005. Nonlinear dimensionality reduction by semidefinite programming and kernel matrix factorization. In Proceedings of the Tenth International Workshop on Artificial Intelligence and Statistics. Barbados, West Indies.
 Weinberger, K.Q. and Saul, L.K. 2004. Unsupervised learning of image manifolds by semidefinite programming. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR04), Washington D.C, pp. 988–995.
 Weinberger, K.Q., Sha, F., and Saul, L.K. 2004. Learning a kernel matrix for nonlinear dimensionality reduction. In Proceedings of the Twenty First International Conference on Machine Learning (ICML04), Banff, Canada, pp. 839–846.
 Zha, H. and Zhang, Z. 2003. Isometric embedding and continuum Isomap. In Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pp. 864–871.
 Zhang, Z. and Zha, H. 2004. Principal manifolds and nonlinear dimensionality reduction by local tangent space alignment. SIAM Journal of Scientific Computing, 26:313–338. CrossRef
 Title
 Unsupervised Learning of Image Manifolds by Semidefinite Programming
 Journal

International Journal of Computer Vision
Volume 70, Issue 1 , pp 7790
 Cover Date
 20061001
 DOI
 10.1007/s112630054939z
 Print ISSN
 09205691
 Online ISSN
 15731405
 Publisher
 Kluwer Academic Publishers
 Additional Links
 Topics
 Keywords

 manifold learning
 dimensionality reduction
 semidefinite programming
 kernel methods
 data analysis
 image manifolds
 semidefinite embedding
 Industry Sectors
 Authors

 Kilian Q. Weinberger ^{(1)}
 Lawrence K. Saul ^{(1)}
 Author Affiliations

 1. Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 191046389