Skip to main content
SpringerLink
Log in

Manifold learning by a deep Gaussian process autoencoder

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The paper presents a novel manifold learning algorithm, the deep Gaussian process autoencoder (DPGA), based on deep Gaussian processes. Deep Gaussian process autoencoder algorithm has the following two main characteristics. The former is a bottleneck structure, borrowed by variational autoencoders and the latter is based on the so-called doubly stochastic variational inference for deep Gaussian processes architecture (DSVI). The main novelties of the paper consist in DGPA algorithm and the experimental protocol for evaluating it. In fact, to the best of our knowledge, deep Gaussian processes algorithms have not been applied to manifold learning, yet. Besides, an experimental protocol is introduced, the so-called manifold learning performance protocol (MLPP), to compare quantitatively the geometric preserved properties of manifold learning projections of the proposed deep Gaussian process autoencoder with the ones of state-of-the-art manifold learning algorithms. Extensive experimental tests on eleven synthetic and five real datasets show that deep Gaussian process autoencoder compares favorably with the other manifold learning competitors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Camastra F, Staiano A (2016) Intrinsic dimension estimation: advances and open problems. Inf Sci 328:26–41. https://doi.org/10.1016/j.ins.2015.08.029

    Article  MATH  Google Scholar 

  2. Fukunaga K (1982) Intrinsic dimensionality extraction. In: Handbook of statistics, vol 2, pp 347–360

  3. Bellman R, Kalaba R (1959) On adaptive control processes. IRE Trans Autom Control 4(2):1–9

    Article  MATH  Google Scholar 

  4. Tenenbaum JB, Silva VD, Langford JC (2000) A global geometric framework for nonlinear dimensionality reduction. Science 290(5500):2319–2323. https://doi.org/10.1126/science.290.5500.2319

    Article  Google Scholar 

  5. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326. https://doi.org/10.1126/science.290.5500.2323

    Article  Google Scholar 

  6. Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput 15(6):1373–1396. https://doi.org/10.1162/089976603321780317

    Article  MATH  Google Scholar 

  7. Donoho DL, Grimes C (2003) Hessian eigenmaps: locally linear embedding techniques for high-dimensional data. Proc Natl Acad Sci 100(10):5591–5596. https://doi.org/10.1073/pnas.1031596100

    Article  MathSciNet  MATH  Google Scholar 

  8. Zhang Z, Zha H (2002) Principal manifolds and nonlinear dimension reduction via local tangent space alignment. arXiv preprint cs/0212008. https://doi.org/10.1137/S1064827502419154

  9. Jiang X, Gao J, Hong X, Cai Z (2014) In: Advances in knowledge discovery and data mining: 18th Pacific-Asia conference, PAKDD 2014, Tainan, Taiwan, May 13–16, 2014. Proceedings, Part II 18. Springer, pp 62–73

  10. Domingues R, Michiardi P, Zouaoui J, Filippone M (2018) Deep Gaussian process autoencoders for novelty detection. Mach Learn 107:1363–1383

    Article  MathSciNet  MATH  Google Scholar 

  11. Salimbeni H, Deisenroth M (2017) Doubly stochastic variational inference for deep Gaussian processes. Adv Neural Inf Process Syst. https://doi.org/10.48550/arXiv.1705.0893

    Article  Google Scholar 

  12. Pearson K (1901) Liii on lines and planes of closest fit to systems of points in space. Lond Edinb Dubl Philos Mag J Sci 2(11):559–572. https://doi.org/10.1080/14786440109462720

    Article  MATH  Google Scholar 

  13. Kruskal JB (1964) Nonmetric multidimensional scaling: a numerical method. Psychometrika 29(2):115–129. https://doi.org/10.1007/BF02289694

    Article  MathSciNet  MATH  Google Scholar 

  14. Leiserson CE, Rivest RL, Cormen TH, Stein C (1994) Introduction to algorithms, vol 3. MIT press, Cambridge

    MATH  Google Scholar 

  15. Zhang Z, Wang J (2006) Mlle: Modified locally linear embedding using multiple weights. Adv Neural Inf Process Syst 19

  16. Kingma DP, Welling M, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013). https://doi.org/10.48550/arXiv.1312.6114

  17. Casale FP, Dalca A, Saglietti L, Listgarten J, Fusi N, Gaussian process prior variational autoencoders. Adv Neural Inf Process Syst 31

  18. Park E (2015) Manifold learning with variational auto-encoder for medical image analysis. Technical Report, University of North Carolina at Chapel Hill, Tech. rep

  19. Williams CK, Rasmussen CE (2006) Gaussian processes for machine learning, vol 2. MIT press Cambridge, MA

    MATH  Google Scholar 

  20. Berg C, Christensen JPR, Ressel P (1984) Harmonic analysis on semigroups: theory of positive definite and related functions, vol 100. Springer, Berlin

    MATH  Google Scholar 

  21. Snelson E, Ghahramani Z (2005) Sparse gaussian processes using pseudo-inputs. Adv Neural Inf Process Syst 18

  22. Hensman J, Fusi N, Lawrence ND (2013) Gaussian processes for big data. arXiv preprint arXiv:1309.6835. https://doi.org/10.48550/arXiv.1309.6835

  23. Titsias M (2009) Artificial intelligence and statistics. In: PMLR, pp 567–574

  24. Lawrence N, Hyvärinen A (2005) Probabilistic non-linear principal component analysis with gaussian process latent variable models. J Mach Learn Res 6(11)

  25. Tipping ME, Bishop CM (1999) Probabilistic principal component analysis. J R Stat Soc Ser B (Stat Methodol) 61(3):611–622. https://doi.org/10.1111/1467-9868.00196

    Article  MathSciNet  MATH  Google Scholar 

  26. Titsias M M, Lawrence ND (2010) In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, pp 844–851

  27. Damianou A, Lawrence ND (2013) Artificial intelligence and statistics. In: PMLR, 2013, pp 207–215. https://doi.org/10.48550/arXiv.1211.0358

  28. He K, Zhang X, Ren S, Sun J (2016) In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778. https://doi.org/10.48550/arXiv.1512.03385

  29. Dai Z, Damianou A, González J, Lawrence N (2015) Variational auto-encoded deep gaussian processes. arXiv preprint arXiv:1511.06455

  30. Hübner U, Weiss C, Abraham N, Tang D (2005) In: Time series prediction. Forecasting the Future and Understanding the Past. Addison Wesley, pp 73–104

  31. Valle M, Oganov AR (2010) Crystal fingerprint space-a novel paradigm for studying crystal-structure sets. Acta Crystallogr A 66(5):507–517

    Article  Google Scholar 

  32. Hein M, Audibert JY (2005) In: ICML’05 Proc. of 22nd international conference on Machine Learning, pp. 289–296. https://doi.org/10.1145/1102351.1102388

  33. Van der Maaten L, Hinton G (2008) Visualizing data using t-sne. J Mach Learn Res 9(11)

  34. Lloyd S (1982) Least squares quantization in PCM. IEEE Trans Inf Theory 28(2):129–137. https://doi.org/10.1109/TIT.1982.1056489

    Article  MathSciNet  MATH  Google Scholar 

  35. Ng A, Jordan M, Weiss Y (2001) On spectral clustering: analysis and an algorithm. Adv Neural Inf Process Syst 14

  36. Rosenberg A, Hirschberg J (2007) In: Proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning (EMNLP-CoNLL) (2007), pp 410–420

  37. Vinh NX Epps, J, Bailey J (2009) In: Proceedings of the 26th annual international conference on machine learning, pp 1073–1080. https://doi.org/10.1145/1553374.1553511

  38. Yeung KY, Ruzzo WL (2001) Details of the adjusted rand index and clustering algorithms, supplement to the paper an empirical study on principal component analysis for clustering gene expression data. Bioinformatics 17(9):763–774

    Article  Google Scholar 

  39. Fowlkes EB, Mallows CL (1983) A method for comparing two hierarchical clusterings. J Am Stat Assoc 78(383):553–569

    Article  MATH  Google Scholar 

  40. Ester M, Kriegel HP, Sander J, Xu X, et al. In: kdd, vol 96, pp 226–231

  41. Lawrence N (2003) Gaussian process latent variable models for visualisation of high dimensional data. Advances Neural Inf Process Syst 16

  42. Havasi M, Hernández-Lobato JM, Murillo-Fuentes JJ (2018) Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo. Adv Neural Inf Process Syst. https://doi.org/10.48550/arXiv.1806.05490

    Article  Google Scholar 

  43. Lee J, Bahri Y, Novak R, Schoenholz SS, Pennington J, Sohl-Dickstein J (2017) Deep neural networks as gaussian processes. arXiv preprint arXiv:1711.00165 (2017). https://doi.org/10.48550/arXiv.1711.00165

Download references

Acknowledgements

Firstly, we would like to thank the anonymous reviewers for the useful comments. Angelo Casolaro and Gennaro Iannuzzo developed parts of the work as their M. Sc. theses in Applied Computer Science (machine learning and Big Data), under the supervision of Francesco Camastra, at University Parthenope of Naples.

Author information

Author notes

  1. Angelo Casolaro and Gennaro Iannuzzo have contributed equally to this work.

Authors and Affiliations

  1. Department of Science and Technology, Parthenope University of Naples, Centro Direzionale Isola C4, 80143, Naples, Italy

    Francesco Camastra, Angelo Casolaro & Gennaro Iannuzzo

Authors
  1. Francesco Camastra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Angelo Casolaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gennaro Iannuzzo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francesco Camastra.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Camastra, F., Casolaro, A. & Iannuzzo, G. Manifold learning by a deep Gaussian process autoencoder. Neural Comput & Applic 35, 15573–15582 (2023). https://doi.org/10.1007/s00521-023-08536-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08536-7

Keywords

Search

Navigation