A survey of visual analytics techniques for machine learning

Abstract

Visual analytics for machine learning has recently evolved as one of the most exciting areas in the field of visualization. To better identify which research topics are promising and to learn how to apply relevant techniques in visual analytics, we systematically review 259 papers published in the last ten years together with representative works before 2010. We build a taxonomy, which includes three first-level categories: techniques before model building, techniques during modeling building, and techniques after model building. Each category is further characterized by representative analysis tasks, and each task is exemplified by a set of recent influential works. We also discuss and highlight research challenges and promising potential future research opportunities useful for visual analytics researchers.

References

  1. [1]

    Liu, S. X.; Wang, X. T.; Liu, M. C.; Zhu, J. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics Vol. 1, No. 1, 48–56, 2017.

    Article  Google Scholar 

  2. [2]

    Choo, J.; Liu, S. X. Visual analytics for explainable deep learning. IEEE Computer Graphics and Applications Vol. 38, No. 4, 84–92, 2018.

    Article  Google Scholar 

  3. [3]

    Hohman, F.; Kahng, M.; Pienta, R.; Chau, D. H. Visual analytics in deep learning: An interrogative survey for the next frontiers. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 8, 2674–2693, 2019.

    Article  Google Scholar 

  4. [4]

    Zeiler, M. D.; Fergus, R. Visualizing and understanding convolutional networks. In: Computer Vision-ECCV 2014. Lecture Notes in Computer Science, Vol. 8689. Fleet, D.; Pajdla, T.; Schiele, B.; Tuytelaars, T. Eds. Springer Cham, 818–833, 2014.

    Google Scholar 

  5. [5]

    Liu, S. X.; Wang, X. T.; Collins, C.; Dou, W. W.; Ouyang, F.; El-Assady, M.; Jiang, L.; Keim, D. A. Bridging text visualization and mining: A task-driven survey. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 7, 2482–2504, 2019.

    Article  Google Scholar 

  6. [6]

    Lu, Y. F.; Garcia, R.; Hansen, B.; Gleicher, M.; Maciejewski, R. The state-of-the-art in predictive visual analytics. Computer Graphics Forum Vol. 36, No. 3, 539–562, 2017.

    Article  Google Scholar 

  7. [7]

    Sacha, D.; Kraus, M.; Keim, D. A.; Chen, M. VIS4ML: An ontology for visual analytics assisted machine learning. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 385–395, 2019.

    Article  Google Scholar 

  8. [8]

    Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; Batra, D. Grad-CAM: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision Vol. 128, 336–359, 2020.

    Article  Google Scholar 

  9. [9]

    Zhang, Q. S.; Zhu, S. C. Visual interpretability for deep learning: A survey. Frontiers of Information Technology & Electronic Engineering Vol. 19, No. 1, 27–39, 2018.

    Article  Google Scholar 

  10. [10]

    Kandel, S.; Parikh, R.; Paepcke, A.; Hellerstein, J. M.; Heer, J. Profiler: Integrated statistical analysis and visualization for data quality assessment. In: Proceedings of the International Working Conference on Advanced Visual Interfaces, 547–554, 2012.

  11. [11]

    Marsland, S. Machine Learning: an Algorithmic Perspective. Chapman and Hall/CRC, 2015.

  12. [12]

    Hung, N. Q. V.; Thang, D. C.; Weidlich, M.; Aberer, K. Minimizing efforts in validating crowd answers. In: Proceedings of the ACM SIGMOD International Conference on Management of Data, 999–1014, 2015.

  13. [13]

    Choo, J.; Lee, C.; Reddy, C. K.; Park, H. UTOPIAN: User-driven topic modeling based on interactive nonnegative matrix factorization. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 1992–2001, 2013.

    Article  Google Scholar 

  14. [14]

    Alemzadeh, S.; Niemann, U.; Ittermann, T.; Völzke, H.; Schneider, D.; Spiliopoulou, M.; Bühler, K.; Preim, B. Visual analysis of missing values in longitudinal cohort study data. Computer Graphics Forum Vol. 39, No. 1, 63–75, 2020.

    Article  Google Scholar 

  15. [15]

    Arbesser, C.; Spechtenhauser, F.; Muhlbacher, T.; Piringer, H. Visplause: Visual data quality assessment of many time series using plausibility checks. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 641–650, 2017.

    Article  Google Scholar 

  16. [16]

    Bäuerle, A.; Neumann, H.; Ropinski, T. Classifier-guided visual correction of noisy labels for image classification tasks. Computer Graphics Forum Vol. 39, No. 3, 195–205, 2020.

    Article  Google Scholar 

  17. [17]

    Bernard, J.; Hutter, M.; Reinemuth, H.; Pfeifer, H.; Bors, C.; Kohlhammer, J. Visual-interactive preprocessing of multivariate time series data. Computer Graphics Forum Vol. 38, No. 3, 401–412, 2019.

    Article  Google Scholar 

  18. [18]

    Bernard, J.; Hutter, M.; Zeppelzauer, M.; Fellner, D.; Sedlmair, M. Comparing visual-interactive labeling with active learning: An experimental study. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 298–308, 2018.

    Article  Google Scholar 

  19. [19]

    Bernard, J.; Zeppelzauer, M.; Lehmann, M.; Müller, M.; Sedlmair, M. Towards user-centered active learning algorithms. Computer Graphics Forum Vol. 37, No. 3, 121–132, 2018.

    Article  Google Scholar 

  20. [20]

    Bors, C.; Gschwandtner, T.; Miksch, S. Capturing and visualizing provenance from data wrangling. IEEE Computer Graphics and Applications Vol. 39, No. 6, 61–75, 2019.

    Article  Google Scholar 

  21. [21]

    Chen, C. J.; Yuan, J.; Lu, Y. F.; Liu, Y.; Su, H.; Yuan, S. T.; Liu, S. X. OoDAnalyzer: Interactive analysis of out-of-distribution samples. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2020.2973258, 2020.

  22. [22]

    Dextras-Romagnino, K.; Munzner, T. Segmen++ tifier: Interactive refinement of clickstream data. Computer Graphics Forum Vol. 38, No. 3, 623–634, 2019.

    Article  Google Scholar 

  23. [23]

    Gschwandtner, T.; Erhart, O. Know your enemy: Identifying quality problems of time series data. In: Proceedings of the IEEE Pacific Visualization Symposium, 205–214, 2018.

  24. [24]

    Halter, G.; Ballester-Ripoll, R.; Flueckiger, B.; Pajarola, R. VIAN: A visual annotation tool for film analysis. Computer Graphics Forum Vol. 38, No. 3, 119–129, 2019.

    Article  Google Scholar 

  25. [25]

    Heimerl, F.; Koch, S.; Bosch, H.; Ertl, T. Visual classifier training for text document retrieval. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 12, 2839–2848, 2012.

    Article  Google Scholar 

  26. [26]

    Höferlin, B.; Netzel, R.; Höferlin, M.; Weiskopf, D.; Heidemann, G. Inter-active learning of ad-hoc classifiers for video visual analytics. In: Proceedings of the Conference on Visual Analytics Science and Technology, 23–32, 2012.

  27. [27]

    Soares Junior, A.; Renso, C.; Matwin, S. ANALYTiC: An active learning system for trajectory classification. IEEE Computer Graphics and Applications Vol. 37, No. 5, 28–39, 2017.

    Article  Google Scholar 

  28. [28]

    Khayat, M.; Karimzadeh, M.; Zhao, J. Q.; Ebert, D. S. VASSL: A visual analytics toolkit for social spambot labeling. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 874–883, 2020.

    Article  Google Scholar 

  29. [29]

    Kurzhals, K.; Hlawatsch, M.; Seeger, C.; Weiskopf, D. Visual analytics for mobile eye tracking. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 301–310, 2017.

    Article  Google Scholar 

  30. [30]

    Lekschas, F.; Peterson, B.; Haehn, D.; Ma, E.; Gehlenborg, N.; Pfister, H. 2019. PEAX: interactive visual pattern search in sequential data using unsupervised deep representation learning. bioRxiv 597518, https://doi.org/10.1101/597518, 2020.

  31. [31]

    Liu, S. X.; Chen, C. J.; Lu, Y. F.; Ouyang, F. X.; Wang, B. An interactive method to improve crowdsourced annotations. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 235–245, 2019.

    Article  Google Scholar 

  32. [32]

    Moehrmann, J.; Bernstein, S.; Schlegel, T.; Werner, G.; Heidemann, G. Improving the usability of hierarchical representations for interactively labeling large image data sets. In: Human-Computer Interaction. Design and Development Approaches. Lecture Notes in Computer Science, Vol. 6761. Jacko, J. A. Ed. Springer Berlin, 618–627, 2011.

    Google Scholar 

  33. [33]

    Paiva, J. G. S.; Schwartz, W. R.; Pedrini, H.; Minghim, R. An approach to supporting incremental visual data classification. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 1, 4–17, 2015.

    Article  Google Scholar 

  34. [34]

    Park, J. H.; Nadeem, S.; Boorboor, S.; Marino, J.; Kaufman, A. E. CMed: Crowd analytics for medical imaging data. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2019.2953026, 2019.

  35. [35]

    Park, J. H.; Nadeem, S.; Mirhosseini, S.; Kaufman, A. C2A: Crowd consensus analytics for virtual colonoscopy. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 21–30, 2016.

  36. [36]

    De Rooij, O.; van Wijk, J. J.; Worring, M. MediaTable: Interactive categorization of multimedia collections. IEEE Computer Graphics and Applications Vol. 30, No. 5, 42–51, 2010.

    Article  Google Scholar 

  37. [37]

    Snyder, L. S.; Lin, Y. S.; Karimzadeh, M.; Goldwasser, D.; Ebert, D. S. Interactive learning for identifying relevant tweets to support real-time situational awareness. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 558–568, 2020.

    Google Scholar 

  38. [38]

    Sperrle, F.; Sevastjanova, R.; Kehlbeck, R.; El-Assady, M. VIANA: Visual interactive annotation of argumentation. In: Proceedings of the Conference on Visual Analytics Science and Technology, 11–22, 2019.

  39. [39]

    Stein, M.; Janetzko, H.; Breitkreutz, T.; Seebacher, D.; Schreck, T.; Grossniklaus, M.; Couzin, I. D.; Keim, D. A. Director’s cut: Analysis and annotation of soccer matches. IEEE Computer Graphics and Applications Vol. 36, No. 5, 50–60, 2016.

    Article  Google Scholar 

  40. [40]

    Wang, X. M.; Chen, W.; Chou, J. K.; Bryan, C.; Guan, H. H.; Chen, W. L.; Pan, R.; Ma, K.-L. GraphProtector: A visual interface for employing and assessing multiple privacy preserving graph algorithms. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 193–203, 2019.

    Article  Google Scholar 

  41. [41]

    Wang, X. M.; Chou, J. K.; Chen, W.; Guan, H. H.; Chen, W. L.; Lao, T. Y.; Ma, K.-L. A utility-aware visual approach for anonymizing multi-attribute tabular data. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 351–360, 2018.

    Article  Google Scholar 

  42. [42]

    Willett, W.; Ginosar, S.; Steinitz, A.; Hartmann, B.; Agrawala, M. Identifying redundancy and exposing provenance in crowdsourced data analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2198–2206, 2013.

    Article  Google Scholar 

  43. [43]

    Xiang, S.; Ye, X.; Xia, J.; Wu, J.; Chen, Y.; Liu, S. Interactive correction of mislabeled training data. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 57–68, 2019.

  44. [44]

    Ingram, S.; Munzner, T.; Irvine, V.; Tory, M.; Bergner, S.; Möller, T. DimStiller: Workflows for dimensional analysis and reduction. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 3–10, 2010.

  45. [45]

    Krause, J.; Perer, A.; Bertini, E. INFUSE: Interactive feature selection for predictive modeling of high dimensional data. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1614–1623, 2014.

    Article  Google Scholar 

  46. [46]

    May, T.; Bannach, A.; Davey, J.; Ruppert, T.; Kohlhammer, J. Guiding feature subset selection with an interactive visualization. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 111–120, 2011.

  47. [47]

    Muhlbacher, T.; Piringer, H. A partition-based framework for building and validating regression models. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 1962–1971, 2013.

    Article  Google Scholar 

  48. [48]

    Seo, J.; Shneiderman, B. A rank-by-feature framework for interactive exploration of multidimensional data. Information Visualization Vol. 4, No. 2, 96–113, 2005.

    Article  Google Scholar 

  49. [49]

    Tam, G. K. L.; Fang, H.; Aubrey, A. J.; Grant, P. W.; Rosin, P. L.; Marshall, D.; Chen, M. Visualization of time-series data in parameter space for understanding facial dynamics. Computer Graphics Forum Vol. 30, No. 3, 901–910, 2011.

    Article  Google Scholar 

  50. [50]

    Broeksema, B.; Baudel, T.; Telea, A.; Crisafulli, P. Decision exploration lab: A visual analytics solution for decision management. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 1972–1981, 2013.

    Article  Google Scholar 

  51. [51]

    Cashman, D.; Patterson, G.; Mosca, A.; Watts, N.; Robinson, S.; Chang, R. RNNbow: Visualizing learning via backpropagation gradients in RNNs. IEEE Computer Graphics and Applications Vol. 38, No. 6, 39–50, 2018.

    Article  Google Scholar 

  52. [52]

    Collaris, D.; van Wijk, J. J. ExplainExplore: Visual exploration of machine learning explanations. In: Proceedings of the IEEE Pacific Visualization Symposium, 26–35, 2020.

  53. [53]

    Eichner, C.; Schumann, H.; Tominski, C. Making parameter dependencies of time-series segmentation visually understandable. Computer Graphics Forum Vol. 39, No. 1, 607–622, 2020.

    Article  Google Scholar 

  54. [54]

    Ferreira, N.; Lins, L.; Fink, D.; Kelling, S.; Wood, C.; Freire, J.; Silva, C. BirdVis: Visualizing and understanding bird populations. IEEE Transactions on Visualization and Computer Graphics Vol. 17, No. 12, 2374–2383, 2011.

    Article  Google Scholar 

  55. [55]

    Fröhler, B.; Moller, T.; Heinzl, C. GEMSe: Visualization-guided exploration of multi-channel segmentation algorithms. Computer Graphics Forum Vol. 35, No. 3, 191–200, 2016.

    Article  Google Scholar 

  56. [56]

    Hohman, F.; Park, H.; Robinson, C.; Polo Chau, D. H. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1096–1106, 2020.

    Article  Google Scholar 

  57. [57]

    Jaunet, T.; Vuillemot, R.; Wolf, C. DRLViz: Understanding decisions and memory in deep reinforcement learning. Computer Graphics Forum Vol. 39, No. 3, 49–61, 2020.

    Article  Google Scholar 

  58. [58]

    Jean, C. S.; Ware, C.; Gamble, R. Dynamic change arcs to explore model forecasts. Computer Graphics Forum Vol. 35, No. 3, 311–320, 2016.

    Article  Google Scholar 

  59. [59]

    Kahng, M.; Andrews, P. Y.; Kalro, A.; Chau, D. H. ActiVis: Visual exploration of industry-scale deep neural network models. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 88–97, 2018.

    Article  Google Scholar 

  60. [60]

    Kahng, M.; Thorat, N.; Chau, D. H. P.; Viegas, F. B.; Wattenberg, M. GAN lab: Understanding complex deep generative models using interactive visual experimentation. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 310–320, 2019.

    Article  Google Scholar 

  61. [61]

    Kwon, B. C.; Anand, V.; Severson, K. A.; Ghosh, S.; Sun, Z. N.; Frohnert, B. I.; Lundgren, M.; Ng, K. DPVis: Visual analytics with hidden Markov models for disease progression pathways. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2020.2985689, 2020.

  62. [62]

    Liu, M. C.; Shi, J. X.; Li, Z.; Li, C. X.; Zhu, J.; Liu, S. X. Towards better analysis of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 91–100, 2017.

    Article  Google Scholar 

  63. [63]

    Liu, S. S.; Li, Z. M.; Li, T.; Srikumar, V.; Pascucci, V.; Bremer, P. T. NLIZE: A perturbation-driven visual interrogation tool for analyzing and interpreting natural language inference models. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 651–660, 2019.

    Article  Google Scholar 

  64. [64]

    Migut, M.; van Gemert, J.; Worring, M. Interactive decision making using dissimilarity to visually represented prototypes. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 141–149, 2011.

  65. [65]

    Ming, Y.; Cao, S.; Zhang, R.; Li, Z.; Chen, Y.; Song, Y.; Qu, H. Understanding hidden memories of recurrent neural networks. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 13–24, 2017.

  66. [66]

    Ming, Y.; Qu, H. M.; Bertini, E. RuleMatrix: Visualizing and understanding classifiers with rules. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 342–352, 2019.

    Article  Google Scholar 

  67. [67]

    Murugesan, S.; Malik, S.; Du, F.; Koh, E.; Lai, T. M. DeepCompare: Visual and interactive comparison of deep learning model performance. IEEE Computer Graphics and Applications Vol. 39, No. 5, 47–59, 2019.

    Article  Google Scholar 

  68. [68]

    Nie, S.; Healey, C.; Padia, K.; Leeman-Munk, S.; Benson, J.; Caira, D.; Sethi, S.; Devarajan, R. Visualizing deep neural networks for text analytics. In: Proceedings of the IEEE Pacific Visualization Symposium, 180–189, 2018.

  69. [69]

    Rauber, P. E.; Fadel, S. G.; Falcao, A. X.; Telea, A. C. Visualizing the hidden activity of artificial neural networks. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 101–110, 2017.

    Article  Google Scholar 

  70. [70]

    Rohlig, M.; Luboschik, M.; Kruger, F.; Kirste, T.; Schumann, H.; Bogl, M.; Alsallakh, B.; Miksch. S. Supporting activity recognition by visual analytics. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 41–48, 2015.

  71. [71]

    Scheepens, R.; Michels, S.; van de Wetering, H.; van Wijk, J. J. Rationale visualization for safety and security. Computer Graphics Forum Vol. 34, No. 3, 191–200, 2015.

    Article  Google Scholar 

  72. [72]

    Shen, Q.; Wu, Y.; Jiang, Y.; Zeng, W.; LAU, A. K. H.; Vianova, A.; Qu, H. Visual interpretation of recurrent neural network on multi-dimensional time-series forecast. In: Proceedings of the IEEE Pacific Visualization Symposium, 61–70, 2020.

  73. [73]

    Strobelt, H.; Gehrmann, S.; Pfister, H.; Rush, A. M. LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent neural networks. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 667–676, 2018.

    Article  Google Scholar 

  74. [74]

    Wang, J. P.; Gou, L.; Yang, H.; Shen, H. W. GANViz: A visual analytics approach to understand the adversarial game. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 6, 1905–1917, 2018.

    Article  Google Scholar 

  75. [75]

    Wang, J. P.; Gou, L.; Zhang, W.; Yang, H.; Shen, H. W. DeepVID: Deep visual interpretation and diagnosis for image classifiers via knowledge distillation. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 6, 2168–2180, 2019.

    Article  Google Scholar 

  76. [76]

    Wang, J.; Zhang, W.; Yang, H. SCANViz: Interpreting the symbol-concept association captured by deep neural networks through visual analytics. In: Proceedings of the IEEE Pacific Visualization Symposium, 51–60, 2020.

  77. [77]

    Wongsuphasawat, K.; Smilkov, D.; Wexler, J.; Wilson, J.; Mane, D.; Fritz, D.; Krishnan, D.; Viegas, F. B.; Wattenberg, M. Visualizing dataflow graphs of deep learning models in TensorFlow. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 1–12, 2018.

    Article  Google Scholar 

  78. [78]

    Zhang, C.; Yang, J.; Zhan, F. B.; Gong, X.; Brender, J. D.; Langlois, P. H.; Barlowe, S.; Zhao, Y. A visual analytics approach to high-dimensional logistic regression modeling and its application to an environmental health study. In: Proceedings of the IEEE Pacific Visualization Symposium, 136–143, 2016.

  79. [79]

    Zhao, X.; Wu, Y. H.; Lee, D. L.; Cui, W. W. iForest: Interpreting random forests via visual analytics. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 407–416, 2019.

    Article  Google Scholar 

  80. [80]

    Ahn, Y.; Lin, Y. R. FairSight: Visual analytics for fairness in decision making. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1086–1095, 2019.

    Google Scholar 

  81. [81]

    Alsallakh, B.; Hanbury, A.; Hauser, H.; Miksch, S.; Rauber, A. Visual methods for analyzing probabilistic classification data. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1703–1712, 2014.

    Article  Google Scholar 

  82. [82]

    Bilal, A.; Jourabloo, A.; Ye, M.; Liu, X. M.; Ren, L. 2018. Do convolutional neural networks learn class hierarchy? IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 152–162, 2018.

    Article  Google Scholar 

  83. [83]

    Cabrera, A. A.; Epperson, W.; Hohman, F.; Kahng, M.; Morgenstern, J.; Chau, D. H.; FAIRVIS: Visual analytics for discovering intersectional bias in machine learning. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 46–56, 2019.

  84. [84]

    Cao, K. L.; Liu, M. C.; Su, H.; Wu, J.; Zhu, J.; Liu, S. X. Analyzing the noise robustness of deep neural networks. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2020.2969185, 2020.

  85. [85]

    Diehl, A.; Pelorosso, L.; Delrieux, C.; Matković, K.; Ruiz, J.; Gröller, M. E.; Bruckner, S. Albero: A visual analytics approach for probabilistic weather forecasting. Computer Graphics Forum Vol. 36, No. 7, 135–144, 2017.

    Article  Google Scholar 

  86. [86]

    Gleicher, M.; Barve, A.; Yu, X. Y.; Heimerl, F. Boxer: Interactive comparison of classifier results. Computer Graphics Forum Vol. 39, No. 3, 181–193, 2020.

    Article  Google Scholar 

  87. [87]

    He, W.; Lee, T.-Y.; van Baar, J.; Wittenburg, K.; Shen, H.-W. DynamicsExplorer: Visual analytics for robot control tasks involving dynamics and LSTM-based control policies. In: Proceedings of the IEEE Pacific Visualization Symposium, 36–45, 2020.

  88. [88]

    Krause, J.; Dasgupta, A.; Swartz, J.; Aphinyanaphongs, Y.; Bertini, E. A workow for visual diagnostics of binary classifiers using instance-level explanations. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 162–172, 2017.

  89. [89]

    Liu, M. C.; Shi, J. X.; Cao, K. L.; Zhu, J.; Liu, S. X. Analyzing the training processes of deep generative models. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 77–87, 2018.

    Article  Google Scholar 

  90. [90]

    Liu, S. X.; Xiao, J. N.; Liu, J. L.; Wang, X. T.; Wu, J.; Zhu, J. Visual diagnosis of tree boosting methods. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 163–173, 2018.

    Article  Google Scholar 

  91. [91]

    Ma, Y. X.; Xie, T. K.; Li, J. D.; Maciejewski, R. Explaining vulnerabilities to adversarial machine learning through visual analytics. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1075–1085, 2020.

    Article  Google Scholar 

  92. [92]

    Pezzotti, N.; Hollt, T.; van Gemert, J.; Lelieveldt, B. P. F.; Eisemann, E.; Vilanova, A. DeepEyes: Progressive visual analytics for designing deep neural networks. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 98–108, 2018.

    Article  Google Scholar 

  93. [93]

    Ren, D. H.; Amershi, S.; Lee, B.; Suh, J.; Williams, J. D. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 61–70, 2017.

    Article  Google Scholar 

  94. [94]

    Spinner, T.; Schlegel, U.; Schafer, H.; El-Assady, M. explAIner: A visual analytics framework for interactive and explainable machine learning. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1064–1074, 2020.

    Google Scholar 

  95. [95]

    Strobelt, H.; Gehrmann, S.; Behrisch, M.; Perer, A.; Pfister, H.; Rush, A. M. Seq2seq-Vis: A visual debugging tool for sequence-to-sequence models. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 353–363, 2019.

    Article  Google Scholar 

  96. [96]

    Wang, J. P.; Gou, L.; Shen, H. W.; Yang, H. DQNViz: A visual analytics approach to understand deep Q-networks. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 288–298, 2019.

    Article  Google Scholar 

  97. [97]

    Wexler, J.; Pushkarna, M.; Bolukbasi, T.; Wattenberg, M.; Viegas, F.; Wilson, J. The what-if tool: Interactive probing of machine learning models. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 56–65, 2019.

    Google Scholar 

  98. [98]

    Zhang, J. W.; Wang, Y.; Molino, P.; Li, L. Z.; Ebert, D. S. Manifold: A model-agnostic framework for interpretation and diagnosis of machine learning models. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 364–373, 2019.

    Article  Google Scholar 

  99. [99]

    Bogl, M.; Aigner, W.; Filzmoser, P.; Lammarsch, T.; Miksch, S.; Rind, A. Visual analytics for model selection in time series analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2237–2246, 2013.

    Article  Google Scholar 

  100. [100]

    Cashman, D.; Perer, A.; Chang, R.; Strobelt, H. Ablate, variate, and contemplate: Visual analytics for discovering neural architectures. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 863–873, 2020.

    Article  Google Scholar 

  101. [101]

    Cavallo, M.; Demiralp, C. Track xplorer: A system for visual analysis of sensor-based motor activity predictions. Computer Graphics Forum Vol. 37, No. 3, 339–349, 2018.

    Article  Google Scholar 

  102. [102]

    Cavallo, M.; Demiralp, C. Clustrophile 2: Guided visual clustering analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 267–276, 2019.

    Article  Google Scholar 

  103. [103]

    Das, S.; Cashman, D.; Chang, R.; Endert, A. BEAMES: Interactive multimodel steering, selection, and inspection for regression tasks. IEEE Computer Graphics and Applications Vol. 39, No. 5, 20–32, 2019.

    Article  Google Scholar 

  104. [104]

    Dingen, D.; van’t Veer, M.; Houthuizen, P.; Mestrom, E. H. J.; Korsten, E. H. H. M.; Bouwman, A. R. A.; van Wijk. J. J. RegressionExplorer: Interactive exploration of logistic regression models with subgroup analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 246–255, 2019.

    Article  Google Scholar 

  105. [105]

    Dou, W. W.; Yu, L.; Wang, X. Y.; Ma, Z. Q.; Ribarsky, W. HierarchicalTopics: Visually exploring large text collections using topic hierarchies. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2002–2011, 2013.

    Article  Google Scholar 

  106. [106]

    El-Assady, M.; Kehlbeck, R.; Collins, C.; Keim, D.; Deussen, O. Semantic concept spaces: Guided topic model refinement using word-embedding projections. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1001–1011, 2020.

    Google Scholar 

  107. [107]

    El-Assady, M.; Sevastjanova, R.; Sperrle, F.; Keim, D.; Collins, C. Progressive learning of topic modeling parameters: A visual analytics framework. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 382–391, 2018.

    Article  Google Scholar 

  108. [108]

    El-Assady, M.; Sperrle, F.; Deussen, O.; Keim, D.; Collins, C. Visual analytics for topic model optimization based on user-steerable speculative execution. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 374–384, 2019.

    Article  Google Scholar 

  109. [109]

    Kim, H.; Drake, B.; Endert, A.; Park, H. ArchiText: Interactive hierarchical topic modeling. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2020.2981456, 2020.

  110. [110]

    Kwon, B. C.; Choi, M. J.; Kim, J. T.; Choi, E.; Kim, Y. B.; Kwon, S.; Sun, J.; Choo, J. RetainVis: Visual analytics with interpretable and interactive recurrent neural networks on electronic medical records. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 299–309, 2019.

    Article  Google Scholar 

  111. [111]

    Lee, H.; Kihm, J.; Choo, J.; Stasko, J.; Park, H. iVisClustering: An interactive visual document clustering via topic modeling. Computer Graphics Forum Vol. 31, No. 3, 1155–1164, 2012.

    Article  Google Scholar 

  112. [112]

    Liu, M. C.; Liu, S. X.; Zhu, X. Z.; Liao, Q. Y.; Wei, F. R.; Pan, S. M. An uncertainty-aware approach for exploratory microblog retrieval. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 250–259, 2016.

    Article  Google Scholar 

  113. [113]

    Lowe, T.; Forster, E. C.; Albuquerque, G.; Kreiss, J. P.; Magnor, M. Visual analytics for development and evaluation of order selection criteria for autoregressive processes. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 151–159, 2016.

    Article  Google Scholar 

  114. [114]

    MacInnes, J.; Santosa, S.; Wright, W. Visual classification: Expert knowledge guides machine learning. IEEE Computer Graphics and Applications Vol. 30, No. 1, 8–14, 2010.

    Article  Google Scholar 

  115. [115]

    Migut, M.; Worring, M. Visual exploration of classification models for risk assessment. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 11–18, 2010.

  116. [116]

    Ming, Y.; Xu, P. P.; Cheng, F. R.; Qu, H. M.; Ren, L. ProtoSteer: Steering deep sequence model with prototypes. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 238–248, 2020.

    Article  Google Scholar 

  117. [117]

    Muhlbacher, T.; Linhardt, L.; Moller, T.; Piringer, H. TreePOD: Sensitivity-aware selection of Pareto-optimal decision trees. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 174–183, 2018.

    Article  Google Scholar 

  118. [118]

    Packer, E.; Bak, P.; Nikkila, M.; Polishchuk, V.; Ship, H. J. Visual analytics for spatial clustering: Using a heuristic approach for guided exploration. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2179–2188, 2013.

    Article  Google Scholar 

  119. [119]

    Piringer, H.; Berger, W.; Krasser, J. HyperMoVal: Interactive visual validation of regression models for real-time simulation. Computer Graphics Forum Vol. 29, No. 3, 983–992, 2010.

    Article  Google Scholar 

  120. [120]

    Sacha, D.; Kraus, M.; Bernard, J.; Behrisch, M.; Schreck, T.; Asano, Y.; Keim, D. A. SOMFlow: Guided exploratory cluster analysis with self-organizing maps and analytic provenance. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 120–130, 2018.

    Article  Google Scholar 

  121. [121]

    Schultz, T.; Kindlmann, G. L. Open-box spectral clustering: Applications to medical image analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2100–2108, 2013.

    Article  Google Scholar 

  122. [122]

    Van den Elzen, S.; van Wijk, J. J. BaobabView: Interactive construction and analysis of decision trees. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 151–160, 2011.

  123. [123]

    Vrotsou, K.; Nordman, A. Exploratory visual sequence mining based on pattern-growth. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 8, 2597–2610, 2019.

    Article  Google Scholar 

  124. [124]

    Wang, X. T.; Liu, S. X.; Liu, J. L.; Chen, J. F.; Zhu, J.; Guo, B. N. TopicPanorama: A full picture of relevant topics. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 12, 2508–2521, 2016.

    Article  Google Scholar 

  125. [125]

    Yang, W. K.; Wang, X. T.; Lu, J.; Dou, W. W.; Liu, S. X. Interactive steering of hierarchical clustering. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2020.2995100, 2020.

  126. [126]

    Zhao, K. Y.; Ward, M. O.; Rundensteiner, E. A.; Higgins, H. N. LoVis: Local pattern visualization for model refinement. Computer Graphics Forum Vol. 33, No. 3, 331–340, 2014.

    Article  Google Scholar 

  127. [127]

    Alexander, E.; Kohlmann, J.; Valenza, R.; Witmore, M.; Gleicher, M. Serendip: Topic model-driven visual exploration of text corpora. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 173–182, 2014.

  128. [128]

    Berger, M.; McDonough, K.; Seversky, L. M. Cite2vec: Citation-driven document exploration via word embeddings. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 691–700, 2017.

    Article  Google Scholar 

  129. [129]

    Blumenschein, M.; Behrisch, M.; Schmid, S.; Butscher, S.; Wahl, D. R.; Villinger, K.; Renner, B.; Reiterer, H.; Keim, D. A. SMARTexplore: Simplifying high-dimensional data analysis through a table-based visual analytics approach. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 36–47, 2018.

  130. [130]

    Bradel, L.; North, C.; House, L. Multi-model semantic interaction for text analytics. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 163–172, 2014.

  131. [131]

    Broeksema, B.; Telea, A. C.; Baudel, T. Visual analysis of multi-dimensional categorical data sets. Computer Graphics Forum Vol. 32, No. 8, 158–169, 2013.

    Article  Google Scholar 

  132. [132]

    Cao, N.; Sun, J. M.; Lin, Y. R.; Gotz, D.; Liu, S. X.; Qu, H. M. FacetAtlas: Multifaceted visualization for rich text corpora. IEEE Transactions on Visualization and Computer Graphics Vol. 16, No. 6, 1172–1181, 2010.

    Article  Google Scholar 

  133. [133]

    Chandrasegaran, S.; Badam, S. K.; Kisselburgh, L.; Ramani, K.; Elmqvist, N. Integrating visual analytics support for grounded theory practice in qualitative text analysis. Computer Graphics Forum Vol. 36, No. 3, 201–212, 2017.

    Article  Google Scholar 

  134. [134]

    Chen, S. M.; Andrienko, N.; Andrienko, G.; Adilova, L.; Barlet, J.; Kindermann, J.; Nguyen, P. H.; Thonnard, O.; Turkay, C. LDA ensembles for interactive exploration and categorization of behaviors. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 9, 2775–2792, 2020.

    Article  Google Scholar 

  135. [135]

    Correll, M.; Witmore, M.; Gleicher, M. Exploring collections of tagged text for literary scholarship. Computer Graphics Forum Vol. 30, No. 3, 731–740, 2011.

    Article  Google Scholar 

  136. [136]

    Dou, W.; Cho, I.; ElTayeby, O.; Choo, J.; Wang, X.; Ribarsky, W.; DemographicVis: Analyzing demographic information based on user generated content. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 57–64, 2015.

  137. [137]

    El-Assady, M.; Gold, V.; Acevedo, C.; Collins, C.; Keim, D. ConToVi: Multi-party conversation exploration using topic-space views. Computer Graphics Forum Vol. 35, No. 3, 431–440, 2016.

    Article  Google Scholar 

  138. [138]

    El-Assady, M.; Sevastjanova, R.; Keim, D.; Collins, C. ThreadReconstructor: Modeling reply-chains to untangle conversational text through visual analytics. Computer Graphics Forum Vol. 37, No. 3, 351–365, 2018.

    Article  Google Scholar 

  139. [139]

    Filipov, V.; Arleo, A.; Federico, P.; Miksch, S. CV3: Visual exploration, assessment, and comparison of CVs. Computer Graphics Forum Vol. 38, No. 3, 107–118, 2019.

    Article  Google Scholar 

  140. [140]

    Fried, D.; Kobourov, S. G. Maps of computer science. In: Proceedings of the IEEE Pacific Visualization Symposium, 113–120, 2014.

  141. [141]

    Fulda, J.; Brehmer, M.; Munzner, T. TimeLineCurator: Interactive authoring of visual timelines from unstructured text. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 300–309, 2016.

    Article  Google Scholar 

  142. [142]

    Glueck, M.; Naeini, M. P.; Doshi-Velez, F.; Chevalier, F.; Khan, A.; Wigdor, D.; Brudno, M. PhenoLines: Phenotype comparison visualizations for disease subtyping via topic models. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 371–381, 2018.

    Article  Google Scholar 

  143. [143]

    Gorg, C.; Liu, Z. C.; Kihm, J.; Choo, J.; Park, H.; Stasko, J. Combining computational analyses and interactive visualization for document exploration and sensemaking in jigsaw. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 10, 1646–1663, 2013.

    Article  Google Scholar 

  144. [144]

    Guo, H.; Laidlaw, D. H. Topic-based exploration and embedded visualizations for research idea generation. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 3, 1592–1607, 2020.

    Article  Google Scholar 

  145. [145]

    Heimerl, F.; John, M.; Han, Q.; Koch, S.; Ertl. T. DocuCompass: Effective exploration of document landscapes. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 11–20, 2016.

  146. [146]

    Hong, F.; Lai, C.; Guo, H.; Shen, E.; Yuan, X.; Li. S. FLDA: Latent Dirichlet allocation based unsteady flow analysis. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 2545–2554, 2014.

    Article  Google Scholar 

  147. [147]

    Hoque, E.; Carenini, G. ConVis: A visual text analytic system for exploring blog conversations. Computer Graphics Forum Vol. 33, No. 3, 221–230, 2014.

    Article  Google Scholar 

  148. [148]

    Hu, M. D.; Wongsuphasawat, K.; Stasko, J. Visualizing social media content with SentenTree. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 621–630, 2017.

    Article  Google Scholar 

  149. [149]

    Jänicke, H.; Borgo, R.; Mason, J. S. D.; Chen, M. SoundRiver: Semantically-rich sound illustration. Computer Graphics Forum Vol. 29, No. 2, 357–366, 2010.

    Article  Google Scholar 

  150. [150]

    Jänicke, S.; Wrisley, D. J. Interactive visual alignment of medieval text versions. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 127–138, 2017.

  151. [151]

    Jankowska, M.; Kefiselj, V.; Milios, E. Relative N-gram signatures: Document visualization at the level of character n-grams. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 103–112, 2012.

  152. [152]

    Ji, X. N.; Shen, H. W.; Ritter, A.; Machiraju, R.; Yen, P. Y. Visual exploration of neural document embedding in information retrieval: Semantics and feature selection. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 6, 2181–2192, 2019.

    Article  Google Scholar 

  153. [153]

    Kakar, T.; Qin, X.; Rundensteiner, E. A.; Harrison, L.; Sahoo, S. K.; De, S. DIVA: Exploration and validation of hypothesized drug-drug interactions. Computer Graphics Forum Vol. 38, No. 3, 95–106, 2019.

    Article  Google Scholar 

  154. [154]

    Kim, H.; Choi, D.; Drake, B.; Endert, A.; Park, H. TopicSifter: Interactive search space reduction through targeted topic modeling. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 35–45, 2019.

  155. [155]

    Kim, M.; Kang, K.; Park, D.; Choo, J.; Elmqvist, N. TopicLens: Efficient multi-level visual topic exploration of large-scale document collections. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 151–160, 2017.

    Article  Google Scholar 

  156. [156]

    Kochtchi, A.; von Landesberger, T.; Biemann, C. Networks of names: Visual exploration and semiautomatic tagging of social networks from newspaper articles. Computer Graphics Forum Vol. 33, No. 3, 211–220, 2014.

    Article  Google Scholar 

  157. [157]

    Li, M. Z.; Choudhury, F.; Bao, Z. F.; Samet, H.; Sellis, T. ConcaveCubes: Supporting cluster-based geographical visualization in large data scale. Computer Graphics Forum Vol. 37, No. 3, 217–228, 2018.

    Article  Google Scholar 

  158. [158]

    Liu, S.; Wang, B.; Thiagarajan, J. J.; Bremer, P. T.; Pascucci, V. Visual exploration of high-dimensional data through subspace analysis and dynamic projections. Computer Graphics Forum Vol. 34, No. 3, 271–280, 2015.

    Article  Google Scholar 

  159. [159]

    Liu, S.; Wang, X.; Chen, J.; Zhu, J.; Guo, B. TopicPanorama: A full picture of relevant topics. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 183–192, 2014.

  160. [160]

    Liu, X.; Xu, A.; Gou, L.; Liu, H.; Akkiraju, R.; Shen, H. W. SocialBrands: Visual analysis of public perceptions of brands on social media. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 71–80, 2016.

  161. [161]

    Oelke, D.; Strobelt, H.; Rohrdantz, C.; Gurevych, I.; Deussen, O. Comparative exploration of document collections: A visual analytics approach. Computer Graphics Forum Vol. 33, No. 3, 201–210, 2014.

    Article  Google Scholar 

  162. [162]

    Park, D.; Kim, S.; Lee, J.; Choo, J.; Diakopoulos, N.; Elmqvist, N. ConceptVector: text visual analytics via interactive lexicon building using word embedding. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 361–370, 2018.

    Article  Google Scholar 

  163. [163]

    Paulovich, F. V.; Toledo, F. M. B.; Telles, G. P.; Minghim, R.; Nonato, L. G. Semantic wordification of document collections. Computer Graphics Forum Vol. 31, No. 3pt3, 1145–1153, 2012.

    Article  Google Scholar 

  164. [164]

    Shen, Q. M.; Zeng, W.; Ye, Y.; Arisona, S. M.; Schubiger, S.; Burkhard, R.; Qu, H. StreetVizor: Visual exploration of human-scale urban forms based on street views. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 1004–1013, 2018.

    Article  Google Scholar 

  165. [165]

    Von Landesberger, T.; Basgier, D.; Becker, M. Comparative local quality assessment of 3D medical image segmentations with focus on statistical shape model-based algorithms. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 12, 2537–2549, 2016.

    Article  Google Scholar 

  166. [166]

    Wall, E.; Das, S.; Chawla, R.; Kalidindi, B.; Brown, E. T.; Endert, A. Podium: Ranking data using mixed-initiative visual analytics. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 288–297, 2018.

    Article  Google Scholar 

  167. [167]

    Xie, X.; Cai, X. W.; Zhou, J. P.; Cao, N.; Wu, Y. C. A semantic-based method for visualizing large image collections. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 7, 2362–2377, 2019.

    Article  Google Scholar 

  168. [168]

    Zhang, L.; Huang, H. Hierarchical narrative collage for digital photo album. Computer Graphics Forum Vol. 31, No. 7, 2173–2181, 2012.

    Article  Google Scholar 

  169. [169]

    Zhao, J.; Chevalier, F.; Collins, C.; Balakrishnan, R. Facilitating discourse analysis with interactive visualization. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 12, 2639–2648, 2012.

    Article  Google Scholar 

  170. [170]

    Alsakran, J.; Chen, Y.; Luo, D. N.; Zhao, Y.; Yang, J.; Dou, W. W.; Liu, S. Real-time visualization of streaming text with a force-based dynamic system. IEEE Computer Graphics and Applications Vol. 32, No. 1, 34–45, 2012.

    Article  Google Scholar 

  171. [171]

    Alsakran, J.; Chen, Y.; Zhao, Y.; Yang, J.; Luo, D. STREAMIT: Dynamic visualization and interactive exploration of text streams. In: Proceedings of the IEEE Pacific Visualization Symposium, 131–138, 2011.

  172. [172]

    Andrienko, G.; Andrienko, N.; Anzer, G.; Bauer, P.; Budziak, G.; Fuchs, G.; Hecker, D.; Weber, H.; Wrobel, S. Constructing spaces and times for tactical analysis in football. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2019.2952129, 2019.

  173. [173]

    Andrienko, G.; Andrienko, N.; Bremm, S.; Schreck, T.; von Landesberger, T.; Bak, P.; Keim, D. Space-in-time and time-in-space self-organizing maps for exploring spatiotemporal patterns. Computer Graphics Forum Vol. 29, No. 3, 913–922, 2010.

    Article  Google Scholar 

  174. [174]

    Andrienko, G.; Andrienko, N.; Hurter, C.; Rinzivillo, S.; Wrobel, S. Scalable analysis of movement data for extracting and exploring significant places. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 7, 1078–1094, 2013.

    Article  Google Scholar 

  175. [175]

    Blascheck, T.; Beck, F.; Baltes, S.; Ertl, T.; Weiskopf, D. Visual analysis and coding of data-rich user behavior. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 141–150, 2016.

  176. [176]

    Bogl, M.; Filzmoser, P.; Gschwandtner, T.; Lammarsch, T.; Leite, R. A.; Miksch, S.; Rind, A. Cycle plot revisited: Multivariate outlier detection using a distance-based abstraction. Computer Graphics Forum Vol. 36, No. 3, 227–238, 2017.

    Article  Google Scholar 

  177. [177]

    Bosch, H.; Thom, D.; Heimerl, F.; Puttmann, E.; Koch, S.; Kruger, R.; Worner, M.; Ertl, T. ScatterBlogs2: real-time monitoring of microblog messages through user-guided filtering. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2022–2031, 2013.

    Article  Google Scholar 

  178. [178]

    Buchmüller, J.; Janetzko, H.; Andrienko, G.; Andrienko, N.; Fuchs, G.; Keim, D. A. Visual analytics for exploring local impact of air traffic. Computer Graphics Forum Vol. 34, No. 3, 181–190, 2015.

    Article  Google Scholar 

  179. [179]

    Cao, N.; Lin, C. G.; Zhu, Q. H.; Lin, Y. R.; Teng, X.; Wen, X. D. Voila: Visual anomaly detection and monitoring with streaming spatiotemporal data. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 23–33, 2018.

    Article  Google Scholar 

  180. [180]

    Cao, N.; Lin, Y. R.; Sun, X. H.; Lazer, D.; Liu, S. X.; Qu, H. M. Whisper: Tracing the spatiotemporal process of information diffusion in real time. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 12, 2649–2658, 2012.

    Article  Google Scholar 

  181. [181]

    Cao, N.; Shi, C. L.; Lin, S.; Lu, J.; Lin, Y. R.; Lin, C. Y. TargetVue: Visual analysis of anomalous user behaviors in online communication systems. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 280–289, 2016.

    Article  Google Scholar 

  182. [182]

    Chae, J.; Thom, D.; Bosch, H.; Jang, Y.; Maciejewski, R.; Ebert, D. S.; Ertl, T. Spatiotemporal social media analytics for abnormal event detection and examination using seasonal-trend decomposition. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 143–152, 2012.

  183. [183]

    Chen, Q.; Yue, X. W.; Plantaz, X.; Chen, Y. Z.; Shi, C. L.; Pong, T. C.; Qu, H. ViSeq: Visual analytics of learning sequence in massive open online courses. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 3, 1622–1636, 2020.

    Article  Google Scholar 

  184. [184]

    Chen, S.; Chen, S.; Lin, L.; Yuan, X.; Liang, J.; Zhang, X. E-map: A visual analytics approach for exploring significant event evolutions in social media. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 36–47, 2017.

  185. [185]

    Chen, S.; Chen, S.; Wang, Z.; Liang, J.; Yuan, X.; Cao, N.; Wu, Y. D-Map: Visual analysis of egocentric information difiusion patterns in social media. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 41–50, 2016.

  186. [186]

    Chen, S. M.; Yuan, X. R.; Wang, Z. H.; Guo, C.; Liang, J.; Wang, Z. C.; Zhang, X.; Zhang, J. Interactive visual discovering of movement patterns from sparsely sampled geo-tagged social media data. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 270–279, 2016.

    Article  Google Scholar 

  187. [187]

    Chen, Y.; Chen, Q.; Zhao, M.; Boyer, S.; Veeramachaneni, K.; Qu, H. DropoutSeer: Visualizing learning patterns in massive open online courses for dropout reasoning and prediction. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 111–120, 2016.

  188. [188]

    Chen, Y. Z.; Xu, P. P.; Ren, L. Sequence synopsis: Optimize visual summary of temporal event data. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 45–55, 2018.

    Article  Google Scholar 

  189. [189]

    Chu, D.; Sheets, D. A.; Zhao, Y.; Wu, Y.; Yang, J.; Zheng, M.; Chen, G. Visualizing hidden themes of taxi movement with semantic transformation. In: Proceedings of the IEEE Pacific Visualization Symposium, 137–144, 2014.

  190. [190]

    Cui, W. W.; Liu, S. X.; Tan, L.; Shi, C. L.; Song, Y. Q.; Gao, Z. K.; Qu, H. M.; Tong, X. TextFlow: Towards better understanding of evolving topics in text. IEEE Transactions on Visualization and Computer Graphics Vol. 17, No. 12, 2412–2421, 2011.

    Article  Google Scholar 

  191. [191]

    Cui, W. W.; Liu, S. X.; Wu, Z. F.; Wei, H. How hierarchical topics evolve in large text corpora. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 2281–2290, 2014.

    Article  Google Scholar 

  192. [192]

    Di Lorenzo, G.; Sbodio, M.; Calabrese, F.; Berlingerio, M.; Pinelli, F.; Nair, R. AllAboard: Visual exploration of cellphone mobility data to optimise public transport. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 2, 1036–1050, 2016.

    Article  Google Scholar 

  193. [193]

    Dou, W.; Wang, X.; Chang, R.; Ribarsky, W. ParallelTopics: A probabilistic approach to exploring document collections. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 231–240, 2011.

  194. [194]

    Dou, W.; Wang, X.; Skau, D.; Ribarsky, W.; Zhou, M. X. Leadline: Interactive visual analysis of text data through event identification and exploration. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 93–102, 2012.

  195. [195]

    Du, F.; Plaisant, C.; Spring, N.; Shneiderman, B. EventAction: Visual analytics for temporal event sequence recommendation. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 61–70, 2016.

  196. [196]

    El-Assady, M.; Sevastjanova, R.; Gipp, B.; Keim, D.; Collins, C. NEREx: Named-entity relationship exploration in multi-party conversations. Computer Graphics Forum Vol. 36, No. 3, 213–225, 2017.

    Article  Google Scholar 

  197. [197]

    Fan, M. M.; Wu, K.; Zhao, J.; Li, Y.; Wei, W.; Truong, K. N. VisTA: Integrating machine intelligence with visualization to support the investigation of think-aloud sessions. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 343–352, 2020.

    Google Scholar 

  198. [198]

    Ferreira, N.; Poco, J.; Vo, H. T.; Freire, J.; Silva, C. T. Visual exploration of big spatio-temporal urban data: A study of New York City taxi trips. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2149–2158, 2013.

    Article  Google Scholar 

  199. [199]

    Gobbo, B.; Balsamo, D.; Mauri, M.; Bajardi, P.; Panisson, A.; Ciuccarelli, P. Topic Tomographies (TopTom): A visual approach to distill information from media streams. Computer Graphics Forum Vol. 38, No. 3, 609–621, 2019.

    Article  Google Scholar 

  200. [200]

    Gotz, D.; Stavropoulos, H. DecisionFlow: Visual analytics for high-dimensional temporal event sequence data. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1783–1792, 2014.

    Article  Google Scholar 

  201. [201]

    Guo, S. N.; Jin, Z. C.; Gotz, D.; Du, F.; Zha, H. Y.; Cao, N. Visual progression analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 417–426, 2019.

    Article  Google Scholar 

  202. [202]

    Guo, S. N.; Xu, K.; Zhao, R. W.; Gotz, D.; Zha, H. Y.; Cao, N. EventThread: Visual summarization and stage analysis of event sequence data. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 56–65, 2018.

    Article  Google Scholar 

  203. [203]

    Gutenko, I.; Dmitriev, K.; Kaufman, A. E.; Barish, M. A. AnaFe: Visual analytics of image-derived temporal features: Focusing on the spleen. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 171–180, 2017.

    Article  Google Scholar 

  204. [204]

    Havre, S.; Hetzler, E.; Whitney, P.; Nowell, L. ThemeRiver: Visualizing thematic changes in large document collections. IEEE Transactions on Visualization and Computer Graphics Vol. 8, No. 1, 9–20, 2002.

    Article  Google Scholar 

  205. [205]

    Heimerl, F.; Han, Q.; Koch, S.; Ertl, T. CiteRivers: Visual analytics of citation patterns. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 190–199, 2016.

    Article  Google Scholar 

  206. [206]

    Itoh, M.; Toyoda, M.; Zhu, C. Z.; Satoh, S.; Kitsuregawa, M. Image flows visualization for intermedia comparison. In: Proceedings of the IEEE Pacific Visualization Symposium, 129–136, 2014.

  207. [207]

    Itoh, M.; Yoshinaga, N.; Toyoda, M.; Kitsuregawa, M. Analysis and visualization of temporal changes in bloggers’ activities and interests. In: Proceedings of the IEEE Pacific Visualization Symposium, 57–64, 2012.

  208. [208]

    Kamaleswaran, R.; Collins, C.; James, A.; McGregor, C. PhysioEx: Visual analysis of physiological event streams. Computer Graphics Forum Vol. 35, No. 3, 331–340, 2016.

    Article  Google Scholar 

  209. [209]

    Karduni, A.; Cho, I.; Wessel, G.; Ribarsky, W.; Sauda, E.; Dou, W. W. Urban space explorer: A visual analytics system for urban planning. IEEE Computer Graphics and Applications Vol. 37, No. 5, 50–60, 2017.

    Article  Google Scholar 

  210. [210]

    Krueger, R.; Han, Q.; Ivanov, N.; Mahtal, S.; Thom, D.; Pfister, H.; Ertl, T. Bird’s-eye-large-scale visual analytics of city dynamics using social location data. Computer Graphics Forum Vol. 38, No. 3, 595–607, 2019.

    Article  Google Scholar 

  211. [211]

    Krueger, R.; Thom, D.; Ertl, T. Visual analysis of movement behavior using web data for context enrichment. In: Proceedings of the IEEE Pacific Visualization Symposium, 193–200, 2014.

  212. [212]

    Krueger, R.; Thom, D.; Ertl, T. Semantic enrichment of movement behavior with foursquare—A visual analytics approach. IEEE Transactions on Visualization and Computer Graphics Vol. 21, No. 8, 903–915, 2015.

    Article  Google Scholar 

  213. [213]

    Lee, C.; Kim, Y.; Jin, S.; Kim, D.; Maciejewski, R.; Ebert, D.; Ko, S. A visual analytics system for exploring, monitoring, and forecasting road traffic congestion. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 11, 3133–3146, 2020.

    Article  Google Scholar 

  214. [214]

    Leite, R. A.; Gschwandtner, T.; Miksch, S.; Kriglstein, S.; Pohl, M.; Gstrein, E.; Kuntner, J. EVA: Visual analytics to identify fraudulent events. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 1, 330–339, 2018.

    Article  Google Scholar 

  215. [215]

    Li, J.; Chen, S. M.; Chen, W.; Andrienko, G.; Andrienko, N. Semantics-space-time cube. A conceptual framework for systematic analysis of texts in space and time. IEEE Transactions on Visualization and Computer Graphics, Vol. 26, No. 4, 1789–1806, 2019.

    Google Scholar 

  216. [216]

    Li, Q.; Wu, Z. M.; Yi, L. L.; Kristanto, S. N.; Qu, H. M.; Ma, X. J. WeSeer: Visual analysis for better information cascade prediction of WeChat articles. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 2, 1399–1412, 2020.

    Article  Google Scholar 

  217. [217]

    Li, Z. Y.; Zhang, C. H.; Jia, S. C.; Zhang, J. W. Galex: Exploring the evolution and intersection of disciplines. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 1182–1192, 2019.

    Google Scholar 

  218. [218]

    Liu, H.; Jin, S. C.; Yan, Y. Y.; Tao, Y. B.; Lin, H. Visual analytics of taxi trajectory data via topical sub-trajectories. Visual Informatics Vol. 3, No. 3, 140–149, 2019.

    Article  Google Scholar 

  219. [219]

    Liu, S. X.; Yin, J. L.; Wang, X. T.; Cui, W. W.; Cao, K. L.; Pei, J. Online visual analytics of text streams. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 11, 2451–2466, 2016.

    Article  Google Scholar 

  220. [220]

    Liu, S.; Zhou, M. X.; Pan, S.; Song, Y.; Qian, W.; Cai, W.; Lian, X. TIARA: Interactive, topic-based visual text summarization and analysis. ACM Transactions on Intelligent Systems and Technology Vol. 3, No. 2, Article No. 25, 2012.

    Google Scholar 

  221. [221]

    Liu, Z. C.; Kerr, B.; Dontcheva, M.; Grover, J.; Hoffman, M.; Wilson, A. CoreFlow: Extracting and visualizing branching patterns from event sequences. Computer Graphics Forum Vol. 36, No. 3, 527–538, 2017.

    Article  Google Scholar 

  222. [222]

    Liu, Z.; Wang, Y.; Dontcheva, M.; Hofiman, M.; Walker, S.; Wilson, A. Patterns and sequences: Interactive exploration of clickstreams to understand common visitor paths. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 321–330, 2017.

    Article  Google Scholar 

  223. [223]

    Lu, Y. F.; Steptoe, M.; Burke, S.; Wang, H.; Tsai, J. Y.; Davulcu, H.; Montgomery, D.; Corman, S. R.; Maciejewski, R. Exploring evolving media discourse through event cueing. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 220–229, 2016.

    Article  Google Scholar 

  224. [224]

    Lu, Y. F.; Wang, F.; Maciejewski, R. Business intelligence from social media: A study from the VAST box office challenge. IEEE Computer Graphics and Applications Vol. 34, No. 5, 58–69, 2014.

    Article  Google Scholar 

  225. [225]

    Lu, Y. F.; Wang, H.; Landis, S.; Maciejewski, R. A visual analytics framework for identifying topic drivers in media events. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 9, 2501–2515, 2018.

    Article  Google Scholar 

  226. [226]

    Luo, D. N.; Yang, J.; Krstajic, M.; Ribarsky, W.; Keim, D. A. EventRiver: Visually exploring text collections with temporal references. IEEE Transactions on Visualization and Computer Graphics Vol. 18, No. 1, 93–105, 2012.

    Article  Google Scholar 

  227. [227]

    Maciejewski, R.; Hafen, R.; Rudolph, S.; Larew, S. G.; Mitchell, M. A.; Cleveland, W. S.; Ebert, D. S. Forecasting hotspots: A predictive analytics approach. IEEE Transactions on Visualization and Computer Graphics Vol. 17, No. 4, 440–453, 2011.

    Article  Google Scholar 

  228. [228]

    Malik, A.; Maciejewski, R.; Towers, S.; McCullough, S.; Ebert, D. S. Proactive spatiotemporal resource allocation and predictive visual analytics for community policing and law enforcement. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1863–1872, 2014.

    Article  Google Scholar 

  229. [229]

    Miranda, F.; Doraiswamy, H.; Lage, M.; Zhao, K.; Goncalves, B.; Wilson, L.; Hsieh, M.; Silva, C. T. Urban pulse: Capturing the rhythm of cities. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 791–800, 2017.

    Article  Google Scholar 

  230. [230]

    Purwantiningsih, O.; Sallaberry, A.; Andary, S.; Seilles, A.; Azfie, J. Visual analysis of body movement in serious games for healthcare. In: Proceedings of the IEEE Pacific Visualization Symposium, 229–233, 2016.

  231. [231]

    Riehmann, P.; Kiesel, D.; Kohlhaas, M.; Froehlich, B. Visualizing a thinker’s life. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 4, 1803–1816, 2019.

    Article  Google Scholar 

  232. [232]

    Sacha, D.; Al-Masoudi, F.; Stein, M.; Schreck, T.; Keim, D. A.; Andrienko, G.; Janetzko, H. Dynamic visual abstraction of soccer movement. Computer Graphics Forum Vol. 36, No. 3, 305–315, 2017.

    Article  Google Scholar 

  233. [233]

    Sarikaya, A.; Correli, M.; Dinis, J. M.; O’Connor, D. H.; Gleicher, M. Visualizing co-occurrence of events in populations of viral genome sequences. Computer Graphics Forum Vol. 35, No. 3, 151–160, 2016.

    Article  Google Scholar 

  234. [234]

    Shi, C. L.; Wu, Y. C.; Liu, S. X.; Zhou, H.; Qu, H. M. LoyalTracker: Visualizing loyalty dynamics in search engines. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1733–1742, 2014.

    Article  Google Scholar 

  235. [235]

    Steiger, M.; Bernard, J.; Mittelstädt, S.; Lücke-Tieke, H.; Keim, D.; May, T.; Kohlhammer, J. Visual analysis of time-series similarities for anomaly detection in sensor networks. Computer Graphics Forum Vol. 33, No. 3, 401–410, 2014.

    Article  Google Scholar 

  236. [236]

    Stopar, L.; Skraba, P.; Grobelnik, M.; Mladenic, D. StreamStory: Exploring multivariate time series on multiple scales. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 4, 1788–1802, 2019.

    Article  Google Scholar 

  237. [237]

    Sultanum, N.; Singh, D.; Brudno, M.; Chevalier, F. Doccurate: A curation-based approach for clinical text visualization. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 142–151, 2019.

    Article  Google Scholar 

  238. [238]

    Sun, G. D.; Wu, Y. C.; Liu, S. X.; Peng, T. Q.; Zhu, J. J. H.; Liang, R. H. EvoRiver: Visual analysis of topic coopetition on social media. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1753–1762, 2014.

    Article  Google Scholar 

  239. [239]

    Sung, C. Y.; Huang, X. Y.; Shen, Y. C.; Cherng, F. Y.; Lin, W. C.; Wang, H. C. Exploring online learners’ interactive dynamics by visually analyzing their time-anchored comments. Computer Graphics Forum Vol. 36, No. 7, 145–155, 2017.

    Article  Google Scholar 

  240. [240]

    Thom, D.; Bosch, H.; Koch, S.; Wörner, M.; Ertl, T. Spatiotemporal anomaly detection through visual analysis of geolocated Twitter messages. In: Proceedings of the IEEE Pacific Visualization Symposium, 41–48, 2012.

  241. [241]

    Thom, D.; Kruger, R.; Ertl, T. Can twitter save lives? A broad-scale study on visual social media analytics for public safety. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 7, 1816–1829, 2016.

    Article  Google Scholar 

  242. [242]

    Tkachev, G.; Frey, S.; Ertl, T. Local prediction models for spatiotemporal volume visualization. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2019.2961893, 2019.

  243. [243]

    Vehlow, C.; Beck, F.; Auwärter, P.; Weiskopf, D. Visualizing the evolution of communities in dynamic graphs. Computer Graphics Forum Vol. 34, No. 1, 277–288, 2015.

    Article  Google Scholar 

  244. [244]

    Von Landesberger, T.; Brodkorb, F.; Roskosch, P.; Andrienko, N.; Andrienko, G.; Kerren, A. MobilityGraphs: Visual analysis of mass mobility dynamics via spatio-temporal graphs and clustering. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 11–20, 2016.

    Article  Google Scholar 

  245. [245]

    Wang, X.; Dou, W.; Ma, Z.; Villalobos, J.; Chen, Y.; Kraft, T.; Ribarsky, W. I-SI: Scalable architecture for analyzing latent topical-level information from social media data. Computer Graphics Forum Vol. 31, No. 3, 1275–1284, 2012.

    Article  Google Scholar 

  246. [246]

    Wang, X.; Liu, S.; Chen, Y.; Peng, T.-Q.; Su, J.; Yang, J.; Guo, B. How ideas flow across multiple social groups. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 51–60, 2016.

  247. [247]

    Wang, Y.; Haleem, H.; Shi, C. L.; Wu, Y. H.; Zhao, X.; Fu, S. W.; Qu, H. Towards easy comparison of local businesses using online reviews. Computer Graphics Forum Vol. 37, No. 3, 63–74, 2018.

    Article  Google Scholar 

  248. [248]

    Wei, F. R.; Liu, S. X.; Song, Y. Q.; Pan, S. M.; Zhou, M. X.; Qian, W. H.; Shi, L.; Tan, L.; Zhang, Q. TIARA: A visual exploratory text analytic system. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 153–162, 2010.

  249. [249]

    Wei, J.; Shen, Z.; Sundaresan, N.; Ma, K.-L. Visual cluster exploration of web clickstream data. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 3–12, 2012.

  250. [250]

    Wu, A. Y.; Qu, H. M. Multimodal analysis of video collections: Visual exploration of presentation techniques in TED talks. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 7, 2429–2442, 2020.

    Article  Google Scholar 

  251. [251]

    Wu, W.; Zheng, Y.; Cao, N.; Zeng, H.; Ni, B.; Qu, H.; Ni, L. M. MobiSeg: Interactive region segmentation using heterogeneous mobility data. In: Proceedings of the IEEE Pacific Visualization Symposium, 91–100, 2017.

  252. [252]

    Wu, Y. C.; Chen, Z. T.; Sun, G. D.; Xie, X.; Cao, N.; Liu, S. X.; Cui, W. StreamExplorer: A multi-stage system for visually exploring events in social streams. IEEE Transactions on Visualization and Computer Graphics Vol. 24, No. 10, 2758–2772, 2018.

    Article  Google Scholar 

  253. [253]

    Wu, Y. C.; Liu, S. X.; Yan, K.; Liu, M. C.; Wu, F. Z. OpinionFlow: Visual analysis of opinion diffusion on social media. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1763–1772, 2014.

    Article  Google Scholar 

  254. [254]

    Wu, Y. H.; Pitipornvivat, N.; Zhao, J.; Yang, S. X.; Huang, G. W.; Qu, H. M. egoSlider: Visual analysis of egocentric network evolution. IEEE Transactions on Visualization and Computer Graphics Vol. 22, No. 1, 260–269, 2016.

    Article  Google Scholar 

  255. [255]

    Xie, C.; Chen, W.; Huang, X. X.; Hu, Y. Q.; Barlowe, S.; Yang, J. VAET: A visual analytics approach for E-transactions time-series. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1743–1752, 2014.

    Article  Google Scholar 

  256. [256]

    Xu, J.; Tao, Y.; Lin, H.; Zhu, R.; Yan, Y. Exploring controversy via sentiment divergences of aspects in reviews. In: Proceedings of the IEEE Pacific Visualization Symposium, 240–249, 2017.

  257. [257]

    Xu, J.; Tao, Y. B.; Yan, Y. Y.; Lin, H. Exploring evolution of dynamic networks via diachronic node embeddings. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 7, 2387–2402, 2020.

    Article  Google Scholar 

  258. [258]

    Xu, P. P.; Mei, H. H.; Ren, L.; Chen, W. ViDX: Visual diagnostics of assembly line performance in smart factories. IEEE Transactions on Visualization and Computer Graphics Vol. 23, No. 1, 291–300, 2017.

    Article  Google Scholar 

  259. [259]

    Xu, P. P.; Wu, Y. C.; Wei, E. X.; Peng, T. Q.; Liu, S. X.; Zhu, J. J.; Qu. H. Visual analysis of topic competition on social media. IEEE Transactions on Visualization and Computer Graphics Vol. 19, No. 12, 2012–2021, 2013.

    Article  Google Scholar 

  260. [260]

    Yu, L.; Wu, W.; Li, X.; Li, G.; Ng, W. S.; Ng, S.-K.; Huang, Z.; Arunan, A.; Watt, H. M. iVizTRANS: Interactive visual learning for home and work place detection from massive public transportation data. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 49–56, 2015.

  261. [261]

    Garcia Zanabria, G.; Alvarenga Silveira, J.; Poco, J.; Paiva, A.; Batista Nery, M.; Silva, C. T.; de Abreu, S. F. A.; Nonato, L. G. CrimAnalyzer: Understanding crime patterns in São Paulo. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2019.2947515, 2019.

  262. [262]

    Zeng, H. P.; Shu, X. H.; Wang, Y. B.; Wang, Y.; Zhang, L. G.; Pong, T. C.; Qu, H. EmotionCues: Emotion-oriented visual summarization of classroom videos. IEEE Transactions on Visualization and Computer Graphics doi: https://doi.org/10.1109/TVCG.2019.2963659, 2020.

  263. [263]

    Zeng, H. P.; Wang, X. B.; Wu, A. Y.; Wang, Y.; Li, Q.; Endert, A.; Qu, H. EmoCo: Visual analysis of emotion coherence in presentation videos. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 927–937, 2019.

    Google Scholar 

  264. [264]

    Zeng, W.; Fu, C. W.; Müller Arisona, S.; Erath, A.; Qu, H. Visualizing waypoints-constrained origin-destination patterns for massive transportation data. Computer Graphics Forum Vol. 35, No. 8, 95–107, 2016.

    Article  Google Scholar 

  265. [265]

    Zhang, J. W.; Ahlbrand, B.; Malik, A.; Chae, J.; Min, Z. Y.; Ko, S.; Ebert, D. S. A visual analytics framework for microblog data analysis at multiple scales of aggregation. Computer Graphics Forum Vol. 35, No. 3, 441–450, 2016.

    Article  Google Scholar 

  266. [266]

    Zhang, J. W.; E, Y. L.; Ma, J.; Zhao, Y. H.; Xu, B. H.; Sun, L. T.; Chen, J.; Yuan, X. Visual analysis of public utility service problems in a metropolis. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1843–1852, 2014.

    Article  Google Scholar 

  267. [267]

    Zhao, J.; Cao, N.; Wen, Z.; Song, Y. L.; Lin, Y. R.; Collins, C. #FluxFlow: Visual analysis of anomalous information spreading on social media. IEEE Transactions on Visualization and Computer Graphics Vol. 20, No. 12, 1773–1782, 2014.

    Article  Google Scholar 

  268. [268]

    Zhao, Y.; Luo, X. B.; Lin, X. R.; Wang, H. R.; Kui, X. Y.; Zhou, F. F.; Wang, J.; Chen, Y.; Chen, W. Visual analytics for electromagnetic situation awareness in radio monitoring and management. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 1, 590–600, 2020.

    Article  Google Scholar 

  269. [269]

    Zhou, Z. G.; Meng, L. H.; Tang, C.; Zhao, Y.; Guo, Z. Y.; Hu, M. X.; Chen, W. Visual abstraction of large scale geospatial origin-destination movement data. IEEE Transactions on Visualization and Computer Graphics Vol. 25, No. 1, 43–53, 2019.

    Article  Google Scholar 

  270. [270]

    Zhou, Z. G.; Ye, Z. F.; Liu, Y. N.; Liu, F.; Tao, Y. B.; Su, W. H. Visual analytics for spatial clusters of air-quality data. IEEE Computer Graphics and Applications Vol. 37, No. 5, 98–105, 2017.

    Article  Google Scholar 

  271. [271]

    Tian, T.; Zhu, J. Max-margin majority voting for learning from crowds. In: Proceedings of the Advances in Neural Information Processing Systems, 1621–1629, 2015.

  272. [272]

    Ng, A. Machine learning and AI via brain simulations. 2013. Available at https://ai.stanford.edu/~ang/slides/DeepLearning-Mar2013.pptx.

  273. [273]

    Nilsson, N. J. Introduction to Machine Learning: An Early Draft of a Proposed Textbook. 2005. Available at https://ai.stanford.edu/~nilsson/MLBOOK.pdf.

  274. [274]

    Lakshminarayanan, B.; Pritzel, A.; Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In: Proceedings of the Advances in Neural Information Processing Systems, 6402–6413, 2017.

  275. [275]

    Lee, K.; Lee, H.; Lee, K.; Shin, J. Training confidence-calibrated classifiers for detecting ut-of-distribution samples. arXiv preprint arXiv:1711.09325, 2018.

  276. [276]

    Liu, M. C.; Jiang, L.; Liu, J. L.; Wang, X. T.; Zhu, J.; Liu, S. X. Improving learning-from-crowds through expert validation. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence, 2329–2336, 2017.

  277. [277]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision Vol. 115, No. 3, 211–252, 2015.

    MathSciNet  Article  Google Scholar 

  278. [278]

    Chandrashekar, G.; Sahin, F. A survey on feature selection methods. Computers & Electrical Engineering Vol. 40, No. 1, 16–28, 2014.

    Article  Google Scholar 

  279. [279]

    Brooks, M.; Amershi, S.; Lee, B.; Drucker, S. M.; Kapoor, A.; Simard, P. FeatureInsight: Visual support for error-driven feature ideation in text classification. In: Proceedings of the IEEE Conference on Visual Analytics Science and Technology, 105–112, 2015.

  280. [280]

    Tzeng, F.-Y.; Ma, K.-L. Opening the black box—Data driven visualization of neural networks. In: Proceedings of the IEEE Conference on Visualization, 383–390, 2005.

  281. [281]

    Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G. S.; Davis, A.; Dean, J.; Devin, M. et al. TensorFlow: Large-scale machine learning on heterogeneous distributed systems, arXiv preprint arXiv:1603.04467, 2015.

  282. [282]

    Ming, Y.; Xu, P. P.; Qu, H. M.; Ren, L. Interpretable and steerable sequence learning via prototypes. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 903–913, 2019.

  283. [283]

    Liu, S. X.; Cui, W. W.; Wu, Y. C.; Liu, M. C. A survey on information visualization: Recent advances and challenges. The Visual Computer Vol. 30, No. 12, 1373–1393, 2014.

    Article  Google Scholar 

  284. [284]

    Ma, Z.; Dou, W.; Wang, X.; Akella, S. Taglatent Dirichlet allocation: Understanding hashtags and their relationships. In: Proceedings of the IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technologies, 260–267, 2013.

  285. [285]

    Kosara, R.; Bendix, F.; Hauser, H. Parallel sets: Interactive exploration and visual analysis of categorical data. IEEE Transactions on Visualization and Computer Graphics Vol. 12, No. 4, 558–568, 2006.

    Article  Google Scholar 

  286. [286]

    Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; Dean, J. Distributed representations of words and phrases and their compositionality. In: Proceedings of the Advances in Neural Information Processing Systems, 3111–3119, 2013.

  287. [287]

    Blei, D. M.; Ng, A. Y.; Jordan, M. I. Latent Dirichlet allocation. Journal of Machine Learning Research Vol. 3, 993–1022, 2003.

    MATH  Google Scholar 

  288. [288]

    Teh, Y. W.; Jordan, M. I.; Beal, M. J.; Blei, D. M. Hierarchical dirichlet processes. Journal of the American Statistical Association Vol. 101, No. 476, 1566–1581, 2006.

    MathSciNet  MATH  Article  Google Scholar 

  289. [289]

    Wang, X. T.; Liu, S. X.; Song, Y. Q.; Guo, B. N. Mining evolutionary multi-branch trees from text streams. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 722–730, 2013.

  290. [290]

    Li, Y. F.; Guo, L. Z.; Zhou, Z. H. Towards safe weakly supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence doi: https://doi.org/10.1109/TPAMI.2019.2922396, 2019.

  291. [291]

    Li, Y. F.; Wang, S. B.; Zhou, Z. H. Graph quality judgement: A large margin expedition. In: Proceedings of the International Joint Conference on Artificial Intelligence, 1725–1731, 2016.

  292. [292]

    Zhou, Z. H. A brief introduction to weakly supervised learning. National Science Review Vol. 5, No. 1, 44–53, 2018.

    Article  Google Scholar 

  293. [293]

    Foulds, J.; Frank, E. A review of multi-instance learning assumptions. The Knowledge Engineering Review Vol. 25, No. 1, 1–25, 2010.

    Article  Google Scholar 

  294. [294]

    Zhou, Z. H. Multi-instance learning from supervised view. Journal of Computer Science and Technology Vol. 21, No. 5, 800–809, 2006.

    MathSciNet  Article  Google Scholar 

  295. [295]

    Donahue, J.; Jia, Y.; Vinyals, O.; Hofiman, J.; Zhang, N.; Tzeng, E.; Darrell, T. DeCAF: A deep convolutional activation feature for generic visual recognition. In: Proceedings of the International Conference on Machine Learning, 647–655, 2014.

  296. [296]

    Wang, Q. W.; Yuan, J.; Chen, S. X.; Su, H.; Qu, H. M.; Liu, S. X. Visual genealogy of deep neural networks. IEEE Transactions on Visualization and Computer Graphics Vol. 26, No. 11, 3340–3352, 2020.

    Article  Google Scholar 

  297. [297]

    Ayinde, B. O.; Zurada, J. M. Building efficient ConvNets using redundant feature pruning. arXiv preprint arXiv:1802.07653, 2018.

  298. [298]

    Baltrusaitis, T.; Ahuja, C.; Morency, L. P. Multimodal machine learning: A survey and taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence Vol. 41, No. 2, 423–443, 2019.

    Article  Google Scholar 

  299. [299]

    Lu, J.; Batra, D.; Parikh, D.; Lee, S. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Proceedings of the Advances in Neural Information Processing Systems, 13–23, 2019.

  300. [300]

    Lu, J.; Liu, A. J.; Dong, F.; Gu, F.; Gama, J.; Zhang, G. Q. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering Vol. 31, No. 12, 2346–2363, 2018.

    Google Scholar 

  301. [301]

    Yang, W.; Li, Z.; Liu, M.; Lu, Y.; Cao, K.; Maciejewski, R.; Liu, S. Diagnosing concept drift with visual analytics. arXiv preprint arXiv:2007.14372, 2020.

  302. [302]

    Wang, X.; Chen, W.; Xia, J.; Chen, Z.; Xu, D.; Wu, X.; Xu, M.; Schreck, T. Conceptexplorer: Visual analysis of concept drifts in multi-source time-series data. arXiv preprint arXiv:2007.15272, 2020.

  303. [303]

    Liu, S.; Andrienko, G.; Wu, Y.; Cao, N.; Jiang, L.; Shi, C.; Wang, Y.-S.; Hong, S. Steering data quality with visual analytics: The complexity challenge. Visual Informatics Vol. 2, No. 4, 191–197, 2018.

    Article  Google Scholar 

Download references

Acknowledgements

This research is supported by the National Key R&D Program of China (Nos. 2018YFB1004300 and 2019YFB1405703), the National Natural Science Foundation of China (Nos. 61761136020, 61672307, 61672308, and 61936002), TC190A4DA/3, and in part by Tsinghua-Kuaishou Institute of Future Media Data.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Shixia Liu.

Additional information

Jun Yuan is currently a Ph.D. student at Tsinghua University. His research interests are in explainable artificial intelligence. He received his B.S. degree from Tsinghua University.

Changjian Chen is now a Ph.D. student at Tsinghua University. His research interests are in interactive machine learning. He received his B.S. degree from the University of Science and Technology of China.

Weikai Yang is a graduate student at Tsinghua University. His research interest is in visual text analytics. He received his B.S. degree from Tsinghua University.

Mengchen Liu is a senior researcher at Microsoft. His research interests include explainable AI and computer vision. He received his B.S. degree in electronics engineering and his Ph.D. degree in computer science from Tsinghua University. He has served as a PC member and reviewer for various conferences and journals.

Jiazhi Xia is an associate professor in the School of Computer Science and Engineering at Central South University. He received his Ph.D. degree in computer science from Nanyang Technological University, Singapore in 2011 and obtained his M.S. and B.S. degrees in computer science and technology from Zhejiang University in 2008 and 2005, respectively. His research interests include data visualization, visual analytics, and computer graphics.

Shixia Liu is an associate professor at Tsinghua University. Her research interests include visual text analytics, visual social analytics, interactive machine learning, and text mining. She has worked as a research staff member at IBM China Research Lab and a lead researcher at Microsoft Research Asia. She received her B.S. and M.S. degree from Harbin Institute of Technology, and her Ph.D. degree from Tsinghua University. She is an Associate Editor-in-Chief of IEEE Trans. Vis. Comput. Graph.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.

The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Yuan, J., Chen, C., Yang, W. et al. A survey of visual analytics techniques for machine learning. Comp. Visual Media (2020). https://doi.org/10.1007/s41095-020-0191-7

Download citation

Keywords

  • visual analytics
  • machine learning
  • data quality
  • feature selection
  • model understanding
  • content analysis