Abstract
Dimension reduction (DR) aims to learn low-dimensional representations of high-dimensional data with the preservation of essential information. In the context of manifold learning, we define that the representation after information-lossless DR preserves the topological and geometric properties of data manifolds formally, and propose a novel two-stage DR method, called invertible manifold learning (inv-ML) to bridge the gap between theoretical information-lossless and practical DR. The first stage includes a homeomorphic sparse coordinate transformation to learn low-dimensional representations without destroying topology and a local isometry constraint to preserve local geometry. In the second stage, a linear compression is implemented for the trade-off between the target dimension and the incurred information loss in excessive DR scenarios. Experiments are conducted on seven datasets with a neural network implementation of inv-ML, called i-ML-Enc. Empirically, i-ML-Enc achieves invertible DR in comparison with typical existing methods as well as reveals the characteristics of the learned manifolds. Through latent space interpolation on real-world datasets, we find that the reliability of tangent space approximated by the local neighborhood is the key to the success of manifold-based DR algorithms.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Behrmann, J., Grathwohl, W., Chen, R.T.Q., Duvenaud, D., Jacobsen, J.: Invertible residual networks. In: International Conference on Machine Learning (ICML) (2019)
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., Ha, D.: Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718 (2018)
Dinh, L., Krueger, D., Bengio, Y.: NICE: non-linear independent components estimation. In: International Conference on Learning Representations (ICLR) (2015)
Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. In: International Conference on Learning Representations (ICLR) (2017)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52, 1289–1306 (2006)
Duque, A.F., Morin, S., Wolf, G., Moon, K.R.: Extendable and invertible manifold learning with geometry regularized autoencoders. arXiv preprint arXiv:2007.07142 (2020)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hull, J.: Database for handwritten text recognition research. IEEE Trans. Pattern Anal. Mach. Intell. 16, 550–554 (1994)
Jacobsen, J., Smeulders, A.W.M., Oyallon, E.: i-revnet: deep invertible networks. In: International Conference on Learning Representations (ICLR) (2018)
Johnson, W.B., Lindenstrauss, J.: Extensions of lipschitz maps into a hilbert space. Contemp. Math. 26, 189–206 (1984)
Kaski, S., Venna, J.: Visualizing gene interaction graphs with local multidimensional scaling. In: European Symposium on Artificial Neural Networks, pp. 557–562 (2006)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (ICLR) (2014)
LeCun, Y., Bottou, L., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Li, S.Z., Zhang, Z., Wu, L.: Markov-lipschitz deep learning. arXiv preprint arXiv:2006.08256 (2020)
Maaten, L.V.D., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9, 2579–2605 (2008)
McQueen, J., Meila, M., Joncas, D.: Nearly isometric embedding by relaxation. In: Proceedings of the 29th Neural Information Processing Systems (NIPS), pp. 2631–2639 (2016)
Mei, J.: Introduction to Manifold and Geometry. Beijing Science Press, Beijing (2013)
Moor, M., Horn, M., Rieck, B., Borgwardt, K.: Topological autoencoders. In: International Conference on Machine Learning (ICML) (2020)
Nash, J.: The imbedding problem for riemannian manifolds. Ann. Math. 63, 20–63 (1956)
Nene, S.A., Nayar, S.K., Murase, H.: Columbia object image library (coil-20). Technical Report, Columbia University (1996). https://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php
Nguyen, T.-G.L., Ardizzone, L., Köthe, U.: Training Invertible Neural Networks as Autoencoders. In: Fink, G.A., Frintrop, S., Jiang, X. (eds.) DAGM GCPR 2019. LNCS, vol. 11824, pp. 442–455. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-33676-9_31
Pedregosa, F., et al.: Édouard Duchesnay: Scikit-learn: machine learning in python. J. Mach. Learn. Res. 12(85), 2825–2830 (2011)
Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)
Seshadri, H., Verma, K.: The embedding theorems of Whitney and Nash. Resonance 21(9), 815–826 (2016). https://doi.org/10.1007/s12045-016-0387-4
Tenenbaum, J.B., De Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Zhang, Z., Wang, J.: Mlle: modified locally linear embedding using multiple weights. In: Advances in Neural Information Processing systems, pp. 1593–1600 (2007)
Zhang, Z., Zha, H.: Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM J. Sci. Comput. 26(1), 313–338 (2004)
Acknowledgments
This work was done during the internship of Siyuan Li and Haitao Lin at Westlake University. We thank Di Wu for helpful insights on hyperparameters tuning.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, S., Lin, H., Zang, Z., Wu, L., Xia, J., Li, S.Z. (2021). Invertible Manifold Learning for Dimension Reduction. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12977. Springer, Cham. https://doi.org/10.1007/978-3-030-86523-8_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-86523-8_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86522-1
Online ISBN: 978-3-030-86523-8
eBook Packages: Computer ScienceComputer Science (R0)