Skip to main content
SpringerLink
Log in

A review, framework, and R toolkit for exploring, evaluating, and comparing visualization methods

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper gives a review and synthesis of methods of evaluating dimensionality reduction techniques. Particular attention is paid to rank-order neighborhood evaluation metrics. A framework is created for exploring dimensionality reduction quality through visualization. An associated toolkit is implemented in R. The toolkit includes scatterplots, heat maps, loess smoothing, performance lift diagrams, and animation. The overall rationale is to help researchers compare dimensionality reduction techniques and use visual insights to help select and improve techniques. Examples are given for dimensionality reduction in manifolds and for dimensionality reduction applied to fashion image and consumer survey datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. This value is usually in \(\left[ 0,1\right] \). However, it is possible that if there is less agreement than random, this value will be less than 0.

  2. Data can be downloaded from https://www.kaggle.com/miroslavsabo/young-people-survey.

  3. This is a property of the Barnes–Hut approximation for t-SNE, but Rtsne also forces this for exact t-SNE.

References

  1. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Matthieu, D., Ghemawat, S., Irving, G., Isard, M., et al.: Tensorflow: a system for large-scale machine learning. In: Keeton, K., Roscoe, T. (eds.) 12th USENIX Symposium on Operating Systems Design and Implementation (\(OSDI\) 16), pp. 265–283 (2016)

  2. Akkucuk, U.: Nonlinear mapping: approaches based on optimizing an index of continuity and applying classical metric MDS on revised distances. Ph.D. thesis, Rutgers University, Newark, NJ (2004)

  3. Akkucuk, U., Carroll, J.D.: PARAMAP vs. Isomap: a comparison of two nonlinear mapping algorithms. J. Classif. 23(2), 221–254 (2006)

    MathSciNet  MATH  Google Scholar 

  4. Alvarez-Meza, A.M., Lee, J.A., Verleysen, M., Castellanos-Dominguez, G.: Kernel-based dimensionality reduction using renyi’s \(\alpha \)-entropy measures of similarity. Neurocomputing 222, 36–46 (2017)

    Google Scholar 

  5. Andrew, V.M., Purchase, H.: On the role of design in information visualization. Inf. Vis. 10(4), 356–371 (2011)

    Google Scholar 

  6. Aupetit, M.: Visualizing distortions and recovering topology in continuous projection techniques. Neurocomputing 70(7), 1304–1330 (2007)

    Google Scholar 

  7. Barbosa, A., Paulovich, F.V., Paiva, A., Goldenstein, S., Petronetto, F., Nonato, L.G.: Visualizing and interacting with kernelized data. IEEE Trans. Vis. Comput. Graph. 22(3), 1314–1325 (2015)

    Google Scholar 

  8. Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)

    MATH  Google Scholar 

  9. Berge, J.M.F.T.: Orthogonal procrustes rotation for two or more matrices. Psychometrika 42(2), 267–276 (1977)

    MathSciNet  MATH  Google Scholar 

  10. Bertini, E., Tatu, A., Keim, D.: Quality metrics in high-dimensional data visualization: an overview and systematization. IEEE Trans. Vis. Comput. Graph. 17(12), 2203–2212 (2011)

    Google Scholar 

  11. Borg, I., Groenen, P.J.F.: Modern Multidimensional Scaling: Theory and Applications, 2nd edn. Springer, New York (2005)

    MATH  Google Scholar 

  12. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, pp. 177–186. Physica-Verlag, Heidelberg (2010)

  13. Boulesteix, A.L., Dangl, R., Dean, N., Guyon, I., Hennig, C., Leisch, F., Steinley, D., Mechelen, I.V.: Benchmarking in cluster analysis: a white paper. arXiv preprint arXiv:1809.10496 (2018)

  14. Bradley, A.P.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit. 30(7), 1145–1159 (1997)

    Google Scholar 

  15. Buja, A., Swayne, D.F., Littman, M.L., Dean, N., Hofmann, H., Chen, L.: Data visualization with multidimensional scaling. J. Comput. Graph. Stat. 17(2), 444–472 (2008)

    MathSciNet  Google Scholar 

  16. Busing, F.M.T.A., Commandeur, J.J.F., Heiser, W.J.: PROXSCAL: a multidimensional scaling program for individual differences scaling with constraints. In: Softstat’97: Advances in Statistical Software, vol. 6, pp. 67–74. Lucius & Lucius, Stuttgart (1997)

  17. Campello, R.J.G.B.: Evaluation of unsupervised learning results: making the seemingly impossible possible. In: CEUR Workshop Proceedings, Aachen, Germant, pp. 4–4 (2019)

  18. Carpendale, S.: Evaluating information visualizations. In: Kerren, A., Stasko, J., Fekete, J.-D., North, C. (eds.) Information Visualization: Human-Centered Issues and Perspectives, pp. 19–45. Springer, Heidelberg (2008)

    Google Scholar 

  19. Carroll, J.D., Wish, M.: Multidimensional perceptual models and measurement methods. In: Carterette, E.C., Friedman, M.P. (eds.) Handbook of Perception, vol. 2, pp. 391–447. Academic Press, New York (1974)

    Google Scholar 

  20. Cavallo, M.: Çağatay Demiralp: a visual interaction framework for dimensionality reduction based data exploration. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, CHI EA ’18. ACM, New York (2018)

  21. Chen, L.: Local multidimensional scaling for nonlinear dimension reduction, graph layout and proximity analysis. Ph.D. dissertation, University of Pennsylvania (2006)

  22. Chen, L., Buja, A.: Local multidimensional scaling for nonlinear dimension reduction, graph drawing, and proximity analysis. J. Am. Stat. Assoc. 104(486), 209–219 (2009)

    MathSciNet  MATH  Google Scholar 

  23. Chen, L., Buja, A.: Stress functions for nonlinear dimension reduction, proximity analysis, and graph drawing. J. Mach. Learn. Res. 14, 1145–1173 (2013)

    MathSciNet  MATH  Google Scholar 

  24. Cleveland, W.S.: Robust locally weighted regression and smoothing scatterplots. J. Am. Stat. Assoc. 74(368), 829–836 (1979)

    MathSciNet  MATH  Google Scholar 

  25. Cleveland, W.S., McGill, R.: The many faces of a scatterplot. J. Am. Stat. Assoc. 79(388), 807–822 (1984)

    MathSciNet  Google Scholar 

  26. Coifman, R.R., Lafon, S.: Diffusion maps. Appl. Comput. Harm. Anal. 21(1), 5–30 (2006)

    MathSciNet  MATH  Google Scholar 

  27. Coimbra, D.B., Martins, R.M., Neves, T., Telea, A.C., Paulovich, F.V.: Explaining three-dimensional dimensionality reduction plots. Inf. Vis. 15(2), 154–172 (2016)

    Google Scholar 

  28. Cutura, R., Holzer, S., Aupetit, M., Sedlmair, M.: VisCoDeR: a tool for visually comparing dimensionality reduction algorithms. In: Proceedings of European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) (2018)

  29. Dang, T.N., Wilkinson, L.: ScagExplorer: exploring scatterplots by their scagnostics. pp. 73–80. IEEE, Piscataway (2014)

  30. de Leeuw, J., Mair, P.: Multidimensional scaling using majorization: SMACOF in R. J. Stat. Softw. 31(3), 1–30 (2009)

    Google Scholar 

  31. Demartines, P., Herault, J.: Curvilinear component analysis: a self-organizing neural network for nonlinear mapping of data sets. IEEE Trans. Neural Netw. 8(1), 148–154 (1997)

    Google Scholar 

  32. Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1(1), 269–271 (1959)

    MathSciNet  MATH  Google Scholar 

  33. Etemadpour, R., Motta, R., de Souza Paiva, J.G., Minghim, R., Oliveira, M.C.F., Linsen, L.: Perception-based evaluation of projection methods for multidimensional data visualization. IEEE Trans. Vis. Comput. Graph. 21(1), 81–94 (2015)

    Google Scholar 

  34. France, S.L.: Properties of a General Measure of Configuration Agreement. Algorithms from and for Nature and Life. Springer, Heidelberg (2013)

    Google Scholar 

  35. France, S.L.: The Shape and the Space: Marketing Mapping in the Big Data Era, pp. 1–7. Academy of Marketing, Helensburgh (2019)

    Google Scholar 

  36. France, S.L., Carroll, J.D.: Development of an agreement metric based upon the Rand index for the evaluation of dimensionality reduction techniques, with applications to mapping customer data. In: Perner, P. (ed.) Lecture Notes in Artificial Intelligence, Proceedings Conference MLDM 2007, pp. 499–517. Springer, Heidelberg (2007)

    Google Scholar 

  37. France, S.L., Carroll, J.D.: Two-way multidimensional scaling: a review. IEEE Transactions on Systems, Man, and Cybernetics. Part C (Applications and Reviews) 41(5), 644–661 (2011)

    Google Scholar 

  38. France, S.L., Ghose, S.: Marketing analytics: methods, practice, implementation, and links to other fields. Expert Syst. Appl. 119, 456–475 (2019)

    Google Scholar 

  39. Groenen, P.J.K., Mathar, R., Heiser, W.J.: The majorization approach to multidimensional scaling for Minkowski distances. J. Classif. 12(1), 3–19 (1995)

    MathSciNet  MATH  Google Scholar 

  40. Hahsler, M., Hornik, K., Buchta, C.: Getting things in order: an introduction to the R package seriation. J. Stat. Softw. 25(3), 1–34 (2008)

    Google Scholar 

  41. Hand, D.J., Till, R.J.: A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach. Learn. 45(2), 171–186 (2001)

    MATH  Google Scholar 

  42. Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1), 29–36 (1982)

    Google Scholar 

  43. Hinton, G., Roweis, S.: Stochastic neighbor embedding. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Proceedings of the 15th International Conference on Neural Information Processing Systems, pp. 857–864. MIT Press, Cambridge (2002)

    Google Scholar 

  44. Hubert, L.J., Arabie, P.: Comparing partitions. J. Classif. 2(1), 193–218 (1985)

    MATH  Google Scholar 

  45. Iacobucci, D., Grisaffe, D., DeSarbo, W.: Statistical perceptual maps: using confidence region ellipses to enhance the interpretations of brand positions in multidimensional scaling. J. Mark. Anal. 5(3), 81–98 (2017)

    Google Scholar 

  46. Ingram, S., Munzner, T., Irvine, V., Tory, M., Bergner, S., Möller, T.: DimStiller: workflows for dimensional analysis and reduction. In: MacEachren, A., Miksch, S. (eds.) 2010 IEEE Symposium on Visual Analytics Science and Technology, pp. 3–10. IEEE, Piscataway (2010)

    Google Scholar 

  47. Jankun-Kelly, T.J., Ma, K.L., Gertz, M.: A model and framework for visualization exploration. IEEE Trans. Vis. Comput. Graph. 13(2), 357–369 (2007)

    Google Scholar 

  48. Jefferson, L., Harvey, R.: Accommodating color blind computer users. In: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility, Assets ’06, pp. 40–47. ACM, New York (2006)

  49. Joyce, J.M.: Kullback–Leibler divergence. In: Lovric, M. (ed.) International Encyclopedia of Statistical Science, pp. 720–722. Springer, Berlin (2011)

    Google Scholar 

  50. Karatzoglou, A., Smola, A., Hornik, K., Zeileis, A.: kernlab—an S4 package for kernel methods in R. J. Stat. Softw. 11(9), 1–20 (2004)

    Google Scholar 

  51. Kaski, S., Nikkila, J., Oja, M., Venna, J., Toronen, P., Castren, E.: Trustworthiness and metrics in visualizing similarity of gene expression. BMC Bioinform. 4(1), 48 (2003)

    Google Scholar 

  52. Kraemer, G.: Package ‘dimRed’ (2017). https://cran.r-project.org/web/packages/dimRed/

  53. Krijthe, J., van der Maaten, L., Krijthe, M.J.: Package ‘rtsne’ (2018). https://cran.r-project.org/web/packages/dimRed/

  54. Kruskal, J.B.: Multidimensional scaling for optimizing a goodness of fit metric to a nonmetric hypothesis. Psychometrika 29(1), 1–27 (1964)

    MathSciNet  MATH  Google Scholar 

  55. Kruskal, J.B.: Nonmetric multidimensional scaling: a numerical method. Psychometrika 29(2), 115–129 (1964)

    MathSciNet  MATH  Google Scholar 

  56. Laskowski, P.H.: The traditional and modern look at Tissot’s indicatrix. Am. Cartogr. 16(2), 123–133 (1989)

    Google Scholar 

  57. Lee, J., Verleysen, M.: Quality assessment of nonlinear dimensionality reduction based on k-ary neighborhoods. In: Saeys, Y., Liu, H., Inza, I., Wehenkel, L., de Peer, Y.V. (eds.) New Challenges for Feature Selection in Data Mining and Knowledge Discovery, vol. 4, pp. 21–35. JMLR: Workshop and Conference Proceedings (2008)

  58. Lee, J.A., Peluffo-Ordónez, D.H., Verleysen, M.: Multi-scale similarities in stochastic neighbour embedding: reducing dimensionality while preserving both local and global structure. Neurocomputing 169, 246–261 (2015)

    Google Scholar 

  59. Lee, J.A., Verleysen, M.: Quality assessment of dimensionality reduction: rank-based criteria. Neurocomputing 72(7–9), 1431–1443 (2009)

    Google Scholar 

  60. Lee, J.A., Verleysen, M.: Scale-independent quality criteria for dimensionality reduction. Pattern Recognit. Lett. 31(14), 2248–2257 (2010)

    Google Scholar 

  61. Lespinats, S., Aupetit, M.: CheckViz: sanity check and topological clues for linear and non-linear mappings. Comput. Graph. Forum 30(1), 113–125 (2011)

    Google Scholar 

  62. Liiv, I.: Seriation and matrix reordering methods: an historical overview. Stat. Anal. Data Mining ASA Data Sci. J. 3(2), 70–91 (2010)

    MathSciNet  MATH  Google Scholar 

  63. Lu, Y., Zhou, K., Wu, X., Gong, P.: A novel multi-graph framework for salient object detection. Vis. Comput. 35(11), 1683–1699 (2019)

    Google Scholar 

  64. Lueks, W., Mokbel, B., Biehl, M., Hammer, B.: How to evaluate dimensionality reduction? In: Hammer, B., Villmann, T. (eds.) Proceedings of the Workshop—New Challenges in Neural Computation 2011, vol. 5, pp. 29–37 (2011)

  65. Ma, Y., Fu, Y.: Manifold Learning Theory and Applications, 1st edn. CRC Press, Boca Raton (2011)

    Google Scholar 

  66. Martins, R.M., Coimbra, D.B., Minghim, R., Telea, A.C.: Visual analysis of dimensionality reduction quality for parameterized projections. Comput. Graph. 41, 26–42 (2014)

    Google Scholar 

  67. Martins, R.M., Minghim, R., Telea, A.C.: Explaining neighborhood preservation for multidimensional projections. In: Borgo, R., Turkay, C. (eds.) Computer Graphics and Visual Computing (CGVC), pp. 1–8. The Eurographics Association, Goslar (2015)

    Google Scholar 

  68. Matute, J., Telea, A.C., Linsen, L.: Skeleton-based scagnostics. IEEE Trans. Vis. Comput. Graph. 24(1), 542–552 (2018)

    Google Scholar 

  69. McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction, pp. 1–51. arXiv preprint arXiv:1802.03426 (2018)

  70. McKenna, S., Mazur, D., Agutter, J., Meyer, M.: Design activity framework for visualization design. IEEE Trans. Vis. Comput. Graph. 20(12), 2191–2200 (2014)

    Google Scholar 

  71. Mokbel, B., Lueks, W., Gisbrecht, A., Hammer, B.: Visualizing the quality of dimensionality reduction. Neurocomputing 112, 109–123 (2013)

    Google Scholar 

  72. Motta, R., Minghim, R., de Andrade Lopes, A., Oliveira, M.C.F.: Graph-based measures to assist user assessment of multidimensional projections. Neurocomputing 150((B)), 583–598 (2015)

    Google Scholar 

  73. North, C.: Toward measuring visualization insight. IEEE Comput. Graph. Appl. 26(3), 6–9 (2006)

    Google Scholar 

  74. Ntoutsi, E., Schubert, E., Zimek, A., Zimmermann, A.: 1st workshop on evaluation and experimental design in data mining and machine learning (EDML 2019). CEUR Workshop Proceedings, Aachen, Germany, pp. 1–3 (2019)

  75. Pagliosa, P., Paulovich, F.V., Minghim, R., Levkowitz, H., Nonato, L.G.: Projection inspector: assessment and synthesis of multidimensional projections. Neurocomputing 150((B)), 599–610 (2015)

    Google Scholar 

  76. Pandey, A.V., Krause, J., Felix, C., Boy, J., Bertini, E.: Towards understanding human similarity perception in the analysis of large sets of scatter plots. In: Kaye, J., Druin, A., Lampe, C., Morris, D., Hourcade, J.P. (eds.) Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 3659–3669. ACM, New York (2016)

    Google Scholar 

  77. Paulovich, F.V., Silva, C.T., Nonato, L.G.: Two-phase mapping for projecting massive data sets. IEEE Trans. Vis. Comput. Graph. 16(6), 1281–1290 (2010)

    Google Scholar 

  78. Peay, E.R.: Multidimensional rotation and scaling of configurations to optimal agreement. Psychometrika 53(2), 199–208 (1988)

    MathSciNet  MATH  Google Scholar 

  79. Plaisant, C.: The challenge of information visualization evaluation. In: Costabile, M.F. (ed.) Proceedings of the Working Conference on Advanced Visual Interfaces, AVI ’04, pp. 109–116. ACM, New York (2004)

    Google Scholar 

  80. Pryke, A., Mostaghim, S., Nazemi, A.: Heatmap visualization of population based multi objective algorithms. In: Obayashi, S., Deb, K., Poloni, C., Hiroyasu, T., Murata, T. (eds.) Evolutionary Multi-Criterion Optimization, pp. 361–375. Springer, Heidelberg (2007)

    Google Scholar 

  81. Qu, Z., Hullman, J.: Evaluating visualization sets: trade-offs between local effectiveness and global consistency. In: Proceedings of the Sixth Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization, BELIV ’16, pp. 44–52. ACM, New York (2016)

  82. Rand, W.M.: Objective criteria for the evaluation of clustering methods. J. Am. Stat. Assoc. 66(336), 846–850 (1971)

    Google Scholar 

  83. Rasul, K., Xiao, H.: Fashion-MNIST (2017). https://github.com/zalandoresearch/fashion-mnist

  84. Ringel, D.M., Skiera, B.: Understanding competition using big consumer search data. In: R.H.S. Jr. (ed.) 47th Hawaii International Conference on System Sciences, pp. 3129–3138 (2014)

  85. Ringel, D.M., Skiera, B.: Visualizing asymmetric competition among more than 1,000 products using big search data. Mark. Sci. 35(3), 511–534 (2016)

    Google Scholar 

  86. Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)

    Google Scholar 

  87. Sacha, D., Zhang, L., Sedlmair, M., Lee, J.A., Peltonen, J., Weiskopf, D., North, S.C., Keim, D.A.: Visual interaction with dimensionality reduction: a structured literature analysis. IEEE Trans. Vis. Comput. Graph. 23(1), 241–250 (2017)

    Google Scholar 

  88. Schönemann, P.H., Carroll, R.M.: Fitting one matrix to another under choice of a central dilation and a rigid motion. Psychometrika 35(2), 245–255 (1970)

    Google Scholar 

  89. Sedlmair, M., Munzner, T., Tory, M.: Empirical guidance on scatterplot and dimension reduction technique choices. IEEE Trans. Vis. Comput. Graph. 19(12), 2634–2643 (2013)

    Google Scholar 

  90. Seifert, C., Sabol, V., Kienreich, W.: Stress maps: Analysing local phenomena in dimensionality reduction based visualisations. In: Proceedings of the 1st European Symposium on Visual Analytics Science and Technology (EuroVAST’10), vol. 1, pp. 1–6 (2010)

  91. Shlens, J.: A tutorial on principal components analysis (2005). arXiv:1404.1100

  92. Shyu, W.M., Grosse, E., Cleveland, W.S.: Local regression models. In: Hastie, T. (ed.) Statistical Models in S, pp. 309–376. Chapman and Hall, Boca Raton (1991)

    Google Scholar 

  93. Sibson, R.: Studies in the robustness of multidimensional scaling: procrustes statistics. J. R. Stat. Soc. B 40(2), 234–238 (1978)

    MathSciNet  MATH  Google Scholar 

  94. Silva, V.D., Tenenbaum, J.B.: Global versus local methods in nonlinear dimensionality reduction. In: Becker, S., Thrun, S., Obermayer, K. (eds.) Advances in Neural Information Processing Systems 15, pp. 705–712. MIT Press, Cambridge (2003)

    Google Scholar 

  95. Snyder, J.P.: Flattening the Earth: Two Thousand Years of Map Projections, 1st edn. University of Chicago Press, Chicago (1997)

    Google Scholar 

  96. Sobczyk, A.: Projections in Minkowski and Banach spaces. Duke Math. J. 8(1), 78–106 (1941)

    MathSciNet  MATH  Google Scholar 

  97. Spathis, D., Passalis, N., Tefas, A.: Fast, visual and interactive semi-supervised dimensionality reduction. In: Leal-Taixé, L., Roth, S. (eds.) Computer Vision—ECCV 2018 Workshops, pp. 550–563. Springer, Cham (2019)

    Google Scholar 

  98. Stahnke, J., Dörk, M., Müller, B., Thom, A.: Probing projections: interaction techniques for interpreting arrangements and errors of dimensionality reductions. IEEE Trans. Vis. Comput. Graph. 22(1), 629–638 (2016)

    Google Scholar 

  99. Stasko, J.: Value-driven evaluation of visualizations. In: Lam, H., Isenberg, P., Isenberg, T., Sedlmair, M. (eds.) Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization, BELIV ’14, pp. 46–53. ACM, New York (2014)

    Google Scholar 

  100. Steinley, D.: Properties of the Hubert–Arabie adjusted Rand index. Psychol. Methods 9(3), 386–396 (2004)

    Google Scholar 

  101. Tatu, A., Bak, P., Bertini, E., Keim, D., Schneidewind, J.: Visual quality metrics and human perception: an initial study on 2D projections of large multidimensional data. In: Santucci, G. (ed.) Proceedings of the International Conference on Advanced Visual Interfaces, AVI ’10, pp. 49–56. ACM, New York (2010)

    Google Scholar 

  102. Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290(5500), 2319–2323 (2000)

    Google Scholar 

  103. Torgerson, W.S.: Multidimensional scaling, I: theory and method. Psychometrika 17(4), 401–419 (1952)

    MathSciNet  MATH  Google Scholar 

  104. Torgerson, W.S.: Theory and Methods of Scaling, 1st edn. Wiley, New York (1958)

    Google Scholar 

  105. Tory, M., Möller, T.: Rethinking visualization: a high-level taxonomy. In: Ward, M., Munzner, T. (eds.) IEEE Symposium on Information Visualization, pp. 151–158 (2004)

  106. Trosset, M.W.: A new formulation of the nonmetric strain problem in multidimensional scaling. J. Classif. 15(1), 15–35 (1998)

    MATH  Google Scholar 

  107. Tuan, N.D., Wilkinson, L.: Transforming scagnostics to reveal hidden features. IEEE Trans. Vis. Comput. Graph. 20(12), 1624–1632 (2014)

    Google Scholar 

  108. Tukey, J.W.: Exploratory Data Analysis, 1st edn. Addison-Wesley, Reading (1977)

    MATH  Google Scholar 

  109. Tukey, J.W., Tukey, P.A.: Computer graphics and exploratory data analysis: An introduction. In: Proceedings of the Sixth Annual Conference and Exposition: Computer Graphics 85, pp. 773–785. National Computer Graphics Association, Fairfax (1985)

  110. Tversky, A., Hutchinson, J.W.: Nearest neighbor analysis of psychological spaces. Psychol. Rev. 93(1), 3–22 (1986)

    Google Scholar 

  111. Tversky, A., Rinott, Y., Newman, C.M.: Nearest neighbor analysis of point processes: applications to multidimensional scaling. J. Math. Psychol. 27(3), 235–250 (1983)

    MathSciNet  MATH  Google Scholar 

  112. Upson Jr., C., Faulhaber, T., Kamins, D., Laidlaw, D., Schlegel, D., Vroom, J., Gurwitz, R., van Dam, A.: The application visualization system: a computational environment for scientific visualization. IEEE Comput. Graph. Appl. 9(4), 30–42 (1989)

    Google Scholar 

  113. van der Maaten, L.: Accelerating t-SNE using tree-based algorithms. J. Mach. Learn. Res. 15(1), 3221–3245 (2014)

    MathSciNet  MATH  Google Scholar 

  114. van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  115. van der Maaten, L., Postma, E., den Herik, J.V.: Dimensionality reduction: a comparative review: Working paper, Maastricht University (2009)

  116. Venna, J., Kaski, S.: Neighborhood preservation in nonlinear projection methods: an experimental study. In: Dorffner, G., Bischof, H., Hornik, K. (eds.) Artificial Neural Networks—ICANN 2001, pp. 485–491. Springer, Berlin (2001)

    Google Scholar 

  117. Venna, J., Kaski, S.: Local multidimensional scaling. Neural Netw. 19(6–7), 889–899 (2006)

    MATH  Google Scholar 

  118. Vidal, R., Ma, Y., Sastry, S.: Generalized Principal Component Analysis, 1st edn. Springer, New York (2016)

    MATH  Google Scholar 

  119. Viégas, F.B., Wattenberg, M.: Artistic data visualization: beyond visual analytics. In: Schuler, D. (ed.) Online Communities and Social Computing, pp. 182–191. Springer, Heidelberg (2007)

    Google Scholar 

  120. Wedel, M., Kamakura, W.A.: Market Segmentation: Conceptual and Methodological Foundations, 2nd edn. Kluwer Academic Publishers, Boston (2000)

    Google Scholar 

  121. Wilkinson, L., Anand, A., Grossman, R.: Graph-theoretic scagnostics. In: Stasko, J.T., Ward, M. (eds.) IEEE Symposium on Information Visualization, 2005, INFOVIS 2005, pp. 157–164. Piscataway, IEEE (2005)

    Google Scholar 

  122. Yang, L., Jin, R.: Distance metric learning: a comprehensive survey. Working paper, Michigan State University. (2006). http://www.cse.msu.edu/~rongjin/semisupervised/dist-metric-survey.pdf

  123. Young, F.W., Takane, Y., Lewyckyj, R.: ALSCAL: a nonmetric multidimensional scaling program with several individual-differences options. Behav. Res. Methods Instrum. 10(3), 451–453 (1978)

    Google Scholar 

  124. Zeileis, A., Hornik, K., Murrell, P.: Escaping RGBland: selecting colors for statistical graphics. Comput. Stat. Data Anal. 53(9), 3259–3270 (2009)

    MathSciNet  MATH  Google Scholar 

  125. Zhang, P., Ren, Y., Zhang, B.: A new embedding quality assessment method for manifold learning. Neurocomputing 97, 251–266 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mississippi State University, Mississippi State, MS, 39762, USA

    Stephen L. France

  2. Department of Managment, Bogazici University, 34342, Bebek, Istanbul, Turkey

    Ulas Akkucuk

Authors
  1. Stephen L. France
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ulas Akkucuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen L. France.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

France, S.L., Akkucuk, U. A review, framework, and R toolkit for exploring, evaluating, and comparing visualization methods. Vis Comput 37, 457–475 (2021). https://doi.org/10.1007/s00371-020-01817-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-020-01817-5

Keywords

Search

Navigation