Skip to main content
SpringerLink
Log in

Improving Deep Learning Projections by Neighborhood Analysis

  • Conference paper
  • First Online:
Book cover Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1474))

Abstract

Visualization of multidimensional data is a difficult task, for which there are many tools. Among these tools, dimensionality reduction methods were shown to be particularly helpful to explore data visually. Techniques with good visual separation are very popular, such as those from the SNE-class, but those often are computationally expensive and non-parametric. An approach based on neural networks was recently proposed to address those shortcomings, but it introduces some fuzziness in the generated projection, which is not desired. In this paper we thoroughly explain the parameter space of this neural network approach and propose a new neighborhood-based learning paradigm, which further improves the quality of the projections learned by the neural networks, and we illustrate our approach on large real-world datasets.

This study was financed in part by FAPESP (2014/12236-1, 2015/22308-2 and 2017/25835-9), CNPq (303808/2018-7) and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Becker, M., Lippel, J., Stuhlsatz, A.: Regularized nonlinear discriminant analysis - an approach to robust dimensionality reduction for data visualization. In: Proceedings of the VISIGRAPP, pp. 116–127 (2017)

    Google Scholar 

  2. Cunningham, J., Ghahramani, Z.: Linear dimensionality reduction: survey, insights, and generalizations. JMLR 16, 2859–2900 (2015)

    MathSciNet  MATH  Google Scholar 

  3. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the CVPR, pp. 248–255 (2009)

    Google Scholar 

  4. Elson, J., Douceur, J.J., Howell, J., Saul, J.: Asirra: a CAPTCHA that exploits interest-aligned manual image categorization. In: Proceedings of ACM CCS, pp. 366–374 (2007)

    Google Scholar 

  5. Engel, D., Hüttenberger, L., Hamann, B.: A survey of dimension reduction methods for high-dimensional data analysis and visualization. In: Proceedings of IRTG Workshop, vol. 27, pp. 135–149. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik (2012)

    Google Scholar 

  6. Espadoto, M., Hirata, N., Telea, A.: Deep learning multidimensional projections. J. Inf. Vis. (2020). https://doi.org/10.1177/1473871620909485

  7. Espadoto, M., Hirata, N.S.T., Falcão, A.X., Telea, A.C.: Improving neural network-based multidimensional projections. In: Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: IVAPP, pp. 29–41. INSTICC, SciTePress (2020). https://doi.org/10.5220/0008877200290041

  8. Espadoto, M., Martins, R.M., Kerren, A., Hirata, N.S., Telea, A.C.: Towards a quantitative survey of dimension reduction techniques. IEEE TVCG (2019). https://doi.org/10.1109/TVCG.2019.2944182

    Article  Google Scholar 

  9. Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1

    Chapter  Google Scholar 

  10. Goldberger, J., Roweis, S., Hinton, G.E., Salakhutdinov, R.R.: Neighbourhood components analysis. In: NIPS, vol. 17, pp. 513–520 (2005)

    Google Scholar 

  11. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Hoffman, P., Grinstein, G.: A survey of visualizations for high-dimensional data mining. In: Information Visualization in Data Mining and Knowledge Discovery, pp. 47–82. Morgan Kaufmann (2002)

    Google Scholar 

  13. Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Proceedings of the AAAI (2017)

    Google Scholar 

  14. Joia, P., Coimbra, D., Cuminato, J.A., Paulovich, F.V., Nonato, L.G.: Local affine multidimensional projection. IEEE TVCG 17(12), 2563–2571 (2011)

    Google Scholar 

  15. Jolliffe, I.T.: Principal component analysis and factor analysis. In: Jolliffe, I.T. (ed.) Principal Component Analysis, pp. 115–128. Springer, Heidelberg (1986). https://doi.org/10.1007/978-1-4757-1904-8_7

    Chapter  Google Scholar 

  16. Kehrer, J., Hauser, H.: Visualization and visual analysis of multifaceted scientific data: a survey. IEEE TVCG 19(3), 495–513 (2013)

    Google Scholar 

  17. Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  18. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. CoRR abs/1312.6114 (2013)

    Google Scholar 

  19. Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: NIPS, pp. 950–957 (1992)

    Google Scholar 

  20. LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database. AT&T Labs, vol. 2 (2010). http://yann.lecun.com/exdb/mnist

  21. Liu, S., Maljovec, D., Wang, B., Bremer, P.T., Pascucci, V.: Visualizing high-dimensional data: advances in the past decade. IEEE TVCG 23(3), 1249–1268 (2015)

    Google Scholar 

  22. Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150. Association for Computational Linguistics (2011)

    Google Scholar 

  23. van der Maaten, L.: Learning a parametric embedding by preserving local structure. In: Proceedings of the AI-STATS (2009)

    Google Scholar 

  24. van der Maaten, L.: Accelerating t-SNE using tree-based algorithms. JMLR 15, 3221–3245 (2014)

    MathSciNet  MATH  Google Scholar 

  25. van der Maaten, L., Hinton, G.E.: Visualizing data using t-SNE. JMLR 9, 2579–2605 (2008)

    MATH  Google Scholar 

  26. Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs (2016)

    Google Scholar 

  27. Martins, R., Minghim, R., Telea, A.C.: Explaining neighborhood preservation for multidimensional projections. In: Proceedings of the CGVC, pp. 121–128. Eurographics (2015)

    Google Scholar 

  28. McInnes, L., Healy, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv:1802.03426 (2018)

  29. Nonato, L., Aupetit, M.: Multidimensional projection for visual analytics: linking techniques with distortions, tasks, and layout enrichment. IEEE TVCG (2018)

    Google Scholar 

  30. Park, M.Y., Hastie, T.: L1-regularization path algorithm for generalized linear models. J. R. Stat. Soc.: Ser. B 69(4), 659–677 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  31. Paulovich, F.V., Nonato, L.G., Minghim, R., Levkowitz, H.: Least square projection: a fast high-precision multidimensional projection technique and its application to document mapping. IEEE TVCG 14(3), 564–575 (2008)

    Google Scholar 

  32. Peason, K.: On lines and planes of closest fit to systems of point in space. Phil. Mag. 2(11), 559–572 (1901)

    Article  Google Scholar 

  33. Pezzotti, N., Höllt, T., Lelieveldt, B., Eisemann, E., Vilanova, A.: Hierarchical stochastic neighbor embedding. Comput. Graph. Forum 35(3), 21–30 (2016)

    Article  Google Scholar 

  34. Pezzotti, N., Lelieveldt, B., van der Maaten, L., Höllt, T., Eisemann, E., Vilanova, A.: Approximated and user steerable t-SNE for progressive visual analytics. IEEE TVCG 23, 1739–1752 (2017)

    Google Scholar 

  35. Qian, N.: On the momentum term in gradient descent learning algorithms. Neural Netw. 12(1), 145–151 (1999)

    Article  Google Scholar 

  36. Rauber, P., Falcão, A.X., Telea, A.: Visualizing time-dependent data using dynamic t-SNE. In: Proceedings of the EuroVis: Short Papers, pp. 73–77 (2016)

    Google Scholar 

  37. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)

    Article  Google Scholar 

  38. Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill, New York (1986)

    MATH  Google Scholar 

  39. Sorzano, C., Vargas, J., Pascual-Montano, A.: A survey of dimensionality reduction techniques. arXiv preprint arXiv:1403.2877 (2014)

  40. Srebro, N., Shraibman, A.: Rank, trace-norm and max-norm. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS (LNAI), vol. 3559, pp. 545–560. Springer, Heidelberg (2005). https://doi.org/10.1007/11503415_37

    Chapter  Google Scholar 

  41. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the CVPR, pp. 2818–2826 (2016)

    Google Scholar 

  42. Tenenbaum, J.B., De Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)

    Article  Google Scholar 

  43. Torgerson, W.: Theory and Methods of Scaling. Wiley, Hoboken (1958)

    Google Scholar 

  44. van der Maaten, L., Postma, E.: Dimensionality reduction: a comparative review. Technical report, Tilburg University, Netherlands (2009). tiCC 2009-005

    Google Scholar 

  45. Venna, J., Kaski, S.: Visualizing gene interaction graphs with local multidimensional scaling. In: Proceedings of the ESANN, pp. 557–562 (2006)

    Google Scholar 

  46. Vernier, E., Garcia, R., da Silva, I., Comba, J., Telea, A.: Quantitative evaluation of time-dependent multidimensional projection techniques. In: Computer Graphics Forum, vol. 39, no. 20 (2020)

    Google Scholar 

  47. Wattenberg, M.: How to use t-SNE effectively (2016). https://distill.pub/2016/misread-tsne

  48. Wilson, A., Roelofs, R., Stern, M., Srebro, N., Recht, B.: The marginal value of adaptive gradient methods in machine learning. In: NIPS, pp. 4148–4158. Curran Associates, Inc. (2017)

    Google Scholar 

  49. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

  50. Xie, H., Li, J., Xue, H.: A survey of dimensionality reduction techniques based on random projection. arXiv preprint arXiv:1706.04371 (2017)

  51. Yao, Y., Rosasco, L., Caponnetto, A.: On early stopping in gradient descent learning. Constr. Approx. 26(2), 289–315 (2007)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Utrecht University, Utrecht, The Netherlands

    Terri S. Modrakowski & Alexandru Telea

  2. University of São Paulo, São Paulo, Brazil

    Mateus Espadoto & Nina S. T. Hirata

  3. University of Groningen, Groningen, The Netherlands

    Mateus Espadoto

  4. University of Campinas, Campinas, Brazil

    Alexandre X. Falcão

Authors
  1. Terri S. Modrakowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mateus Espadoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexandre X. Falcão
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nina S. T. Hirata
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alexandru Telea
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mateus Espadoto .

Editor information

Editors and Affiliations

  1. IRISA, University of Rennes 1, Rennes, France

    Kadi Bouatouch

  2. Universidade do Porto, Porto, Portugal

    A. Augusto de Sousa

  3. University of Genova, Genova, Italy

    Manuela Chessa

  4. Mines ParisTech, Paris, France

    Alexis Paljic

  5. Linnaeus University, Växjö, Sweden

    Andreas Kerren

  6. French Civil Aviation University (ENAC), Toulouse, France

    Christophe Hurter

  7. Università di Catania, Catania, Italy

    Giovanni Maria Farinella

  8. Universitat de Barcelona, Barcelona, Spain

    Petia Radeva

  9. Escola Superior de Tecnologia de Setúbal, Setúbal, Portugal

    Jose Braz

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Modrakowski, T.S., Espadoto, M., Falcão, A.X., Hirata, N.S.T., Telea, A. (2022). Improving Deep Learning Projections by Neighborhood Analysis. In: Bouatouch, K., et al. Computer Vision, Imaging and Computer Graphics Theory and Applications. VISIGRAPP 2020. Communications in Computer and Information Science, vol 1474. Springer, Cham. https://doi.org/10.1007/978-3-030-94893-1_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-94893-1_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-94892-4

  • Online ISBN: 978-3-030-94893-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation