We present a new manifold learning algorithm called Local Orthogonality Preserving Alignment (LOPA). Our algorithm is inspired by the Local Tangent Space Alignment (LTSA) method that aims to align multiple local neighborhoods into a global coordinate system using affine transformations. However, LTSA often fails to preserve original geometric quantities such as distances and angles. Although an iterative alignment procedure for preserving orthogonality was suggested by the authors of LTSA, neither the corresponding initialization nor the experiments were given. Procrustes Subspaces Alignment (PSA) implements the orthogonality preserving idea by estimating each rotation transformation separately with simulated annealing. However, the optimization in PSA is complicated and multiple separated local rotations may produce globally contradictive results. To address these difficulties, we first use the pseudo-inverse trick of LTSA to represent each local orthogonal transformation with the unified global coordinates. Second the orthogonality constraints are relaxed to be an instance of semi-definite programming (SDP). Finally a two-step iterative procedure is employed to further reduce the errors in orthogonal constraints. Extensive experiments show that LOPA can faithfully preserve distances, angles, inner products, and neighborhoods of the original datasets. In comparison, the embedding performance of LOPA is better than that of PSA and comparable to that of state-of-the-art algorithms like MVU and MVE, while the runtime of LOPA is significantly faster than that of PSA, MVU and MVE.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Tenenbaum J B, de Silva V, Langford J C. A global geometric framework for nonlinear dimensionality reduction. Science, 2000, 290(5500): 2319–2323.
Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(5500): 2323–2326.
Saul L K, Roweis S T. Think globally, fit locally: Unsupervised learning of low dimensional manifolds. The Journal of Machine Learning Research, 2003, 4: 119–155.
Seung H S, Lee D D. The manifold ways of perception. Science, 2000, 290(5500): 2268–2269.
Donoho D L. High-dimensional data analysis: The curses and blessings of dimensionality. In Proc. AMS Mathematical Challenges of the 21st Century, Aug. 2000.
Pei Y, Huang F, Shi F, Zha H. Unsupervised image matching based on manifold alignment. IEEE Trans. Pattern Analysis and Machine Intelligence (TPAMI), 2012, 34(8): 1658–1664.
Zhang J, Huang H, Wang J. Manifold learning for visualizing and analyzing high-dimensional data. IEEE Intelligent Systems, 2010, 25(4): 54–61.
Weinberger K Q, Saul L K. Unsupervised learning of image manifolds by semidefinite programming. International Journal of Computer Vision, 2006, 70(1): 77–90.
Weinberger K Q, Sha F, Saul L K. Learning a kernel matrix for nonlinear dimensionality reduction. In Proc. the 21st Int. Conf. Machine Learning, July 2004.
Shaw B, Jebara T. Minimum volume embedding. In Proc. the 11th Int. Conf. Artificial Intelligence and Statistics, Mar. 2007, pp.460-467.
Lin T, Zha H. Riemannian manifold learning. TPAMI, 2008, 30(5): 796–809.
Sha F, Saul L K. Analysis and extension of spectral methods for nonlinear dimensionality reduction. In Proc. the 22nd Int. Conf. Machine Learning, Aug. 2005, pp.784-791.
Donoho D L, Grimes C. Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data. Proc. the National Academy of Sciences, 2003, 100(10): 5591–5596.
Belkin M, Niyogi P. Laplacian Eigenmaps for dimensionality reduction and data representation. Neural Computation, 2003, 15(6): 1373–1396.
Zhang Z, Zha H. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM Journal of Scientific Computing, 2004, 26(1): 313–338.
Qiao H, Zhang P, Wang D, Zhang B. An explicit nonlinear mapping for manifold learning. IEEE Trans. Cybernetics, 2013, 43(1): 51–63.
Yang L. Alignment of overlapping locally scaled patches for multidimensional scaling and dimensionality reduction. TPAMI, 2008, 30(3): 438–450.
Wang R, Shan S, Chen X, Chen J, Gao W. Maximal linear embedding for dimensionality reduction. TPAMI, 2011, 33(9): 1776–1792.
Gashler M, Ventura D, Martinez T. Manifold learning by graduated optimization. IEEE Trans. Systems, Man, and Cybernetics – Part B: Cybernetics, 2011, 41(6): 1458–1470.
Venna J, Peltonen J, Nybo K, Aidos H, Kaski S. Information retrieval perspective to nonlinear dimensionality reduction for data visualization. The Journal of Machine Learning Research, 2010, 11: 451–490.
Goldberg Y, Ritov Y. Local Procrustes for manifold embedding: A measure of embedding quality and embedding algorithms. Machine Learning, 2009, 77(1): 1–25.
do Carmo M P. Riemannian Geometry (1st edition). Birkhäuser, Boston, 1992.
Perraul-Joncas D, Meila M. Non-linear dimensionality reduction: Riemannian metric estimation and the problem of geometric discovery. arXiv:1305.7255, 2013.
Kokiopoulou E, Saad Y. Orthogonal neighborhood preserving projections: A projection-based dimensionality reduction technique. TPAMI, 2007, 29(12): 2143–2156.
Wen Z, Yin W. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 2013, 142(1/2): 397–434.
Borchers B. CSDP, A C library for semidefinite programming. Optimization Methods and Software, 1999, 11(1/2/3/4): 613–623.
Toh K, Todd M, Tütüncü R. SDPT3 — A Matlab software package for semidefinite programming, version 1.3. Optimization Methods and Software, 1999, 11(1–4): 545–581.
Sturm J. Using SeDuMi 1.02, a Matlab toolbox for optimization over symmetric cones. Optimization Methods and Software, 1999, 11(1/2/3/4): 625–653.
Vandenberghe L, Boyd S. Semidefinite programming. SIAM Review, 1996, 38(1): 49–95.
Golub G H, van Van Loan C F. Matrix Computations (3rd edition). The Johns Hopkins University Press, 1996.
van der Maaten L, Postma E, van den Herik J. Dimensionality reduction: A comparative review. Technical Report, TiCC TR 2009–005, Tilburg Centre for Creative Computing, Tilburg University, Oct. 2009.
Special Section of CVM 2016
This work was supported by the National Basic Research 973 Program of China under Grant No. 2011CB302202, the National Natural Science Foundation of China under Grant Nos. 61375051 and 61075119, and the Seeding Grant for Medicine and Information Sciences of Peking University of China under Grant No. 2014-MI-21.
About this article
Cite this article
Lin, T., Liu, Y., Wang, B. et al. Nonlinear Dimensionality Reduction by Local Orthogonality Preserving Alignment. J. Comput. Sci. Technol. 31, 512–524 (2016). https://doi.org/10.1007/s11390-016-1644-4
- manifold learning
- dimensionality reduction
- semi-definite programming
- Procrustes measure