Advertisement

The Visual Computer

, Volume 34, Issue 9, pp 1189–1207 | Cite as

VIAL: a unified process for visual interactive labeling

  • Jürgen Bernard
  • Matthias Zeppelzauer
  • Michael Sedlmair
  • Wolfgang Aigner
Original Article
  • 315 Downloads

Abstract

The assignment of labels to data instances is a fundamental prerequisite for many machine learning tasks. Moreover, labeling is a frequently applied process in visual interactive analysis approaches and visual analytics. However, the strategies for creating labels usually differ between these two fields. This raises the question whether synergies between the different approaches can be attained. In this paper, we study the process of labeling data instances with the user in the loop, from both the machine learning and visual interactive perspective. Based on a review of differences and commonalities, we propose the “visual interactive labeling” (VIAL) process that unifies both approaches. We describe the six major steps of the process and discuss their specific challenges. Additionally, we present two heterogeneous usage scenarios from the novel VIAL perspective, one on metric distance learning and one on object detection in videos. Finally, we discuss general challenges to VIAL and point out necessary work for the realization of future VIAL approaches.

Keywords

Information visualization Visual analytics Machine learning Labeling Active learning Classification Regression Similarity search Visual interactive labeling Labeling 

Notes

Acknowledgements

This work is an extended version of a previous EuroVA paper [20] entitled “ A Unified Process for Visual Interactive Labeling.” This work was supported by the Deutsche Forschungsgemeinschaft (DFG), Project No. I 2850, Lead Agency Verfahren (DACH) “Visual Segmentation and Labeling of Multivariate Time Series (VISSECT),” the Austrian Research Promotion Agency (FFG), Project Nos. 7179681, 7189193, the Austrian Ministry for Transport, Innovation and Technology under the initiative “ICT of the future” via the project “VALiD” (Project No. 845598), the Austrian Research Fund (FWF) via the projects “KAVA-Time” (Project No. P25489-N23), and “VisOnFire” (Project No. P27975-NBL), as well as by the Lower Austrian Research and Education Company and the Provincial Government of Lower Austria (NFB), Department of Science and Research via the project “IntelliGait” (Project No. LSC14-005).

References

  1. 1.
    Attenberg, J., Provost, F.: Inactive learning? Difficulties employing active learning in practice. SIGKDD Explor. Newsl. 12(2), 36–41 (2011).  https://doi.org/10.1145/1964897.1964906 CrossRefGoogle Scholar
  2. 2.
    Bengio, Y., Courville, A., Vincent, P.: Representation learning: a review and new perspectives. IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798–1828 (2013)CrossRefGoogle Scholar
  3. 3.
    Bernard, J., Daberkow, D., Fellner, D., Fischer, K., Koepler, O., Kohlhammer, J., Runnwerth, M., Ruppert, T., Schreck, T., Sens, I.: Visinfo: a digital library system for time series research data based on exploratory search—a user-centered design approach. Int. J. Digit. Libr. (IJoDL) 1, 37–59 (2015).  https://doi.org/10.1007/s00799-014-0134-y CrossRefGoogle Scholar
  4. 4.
    Bernard, J., Dobermann, E., Vögele, A., Krüger, B., Kohlhammer, J., Fellner, D.: Visual-interactive semi-supervised labeling of human motion capture data. In: Visualization and Data Analysis (VDA) (2017).  https://doi.org/10.2352/ISSN.2470-1173.2017.1.VDA-387
  5. 5.
    Bengio, Y.: Deep learning of representations for unsupervised and transfer learning. In: ICML Workshop on Unsupervised and Transfer Learning, pp. 17–36 (2012)Google Scholar
  6. 6.
    Bernard, J.: Exploratory Search in Time-Oriented Primary Data. Dissertation, Ph.D. Technische Universität Darmstadt, Graphisch-Interaktive Systeme (GRIS), Darmstadt (2015). http://tuprints.ulb.tu-darmstadt.de/5173/
  7. 7.
    Bellet, A., Habrard, A., Sebban M.: A Survey on Metric Learning for Feature Vectors and Structured Data. CoRR arXiv:1306.6709 (2013)
  8. 8.
    Bernard, J., Hutter, M., Zeppelzauer, M., Fellner, D., Sedlmair, M.: Comparing visual-interactive labeling with active learning: an experimental study. IEEE Trans. Vis. Comput. Graph. (TVCG) (2017).  https://doi.org/10.1109/TVCG.2017.2744818 Google Scholar
  9. 9.
    Buhrmester, M., Kwang, T., Gosling, S.D.: Amazon’s mechanical turk: a new source of inexpensive, yet high-quality, data? Perspect. Psychol. Sci. 6(1), 3–5 (2011)CrossRefGoogle Scholar
  10. 10.
    Blascheck, T., Kurzhals, K., Raschke, M., Burch, M., Weiskopf, D., Ertl, T.: State-of-the-art of visualization for eye tracking data. In: EuroVis (STAR) (2014), Eurograph.  https://doi.org/10.2312/eurovisstar.20141173
  11. 11.
    Behrisch, M., Korkmaz, F., Shao, L., Schreck, T.: Feedback-driven interactive exploration of large multidimensional data supported by visual classifier. In: IEEE Visual Analytics Science and Technology (VAST), pp. 43–52 (2014)Google Scholar
  12. 12.
    Brown, E.T., Liu, J., Brodley, C.E., Chang, R.: Dis-function: Learning distance functions interactively. In: IEEE Visual Analytics Science and Technology (VAST), pp. 83–92. IEEE (2012)Google Scholar
  13. 13.
    Bernard, J., Ruppert, T., Goroll, O., May, T., Kohlhammer, J.: Visual-interactive preprocessing of time series data. In: SIGRAD, Swedish Chapter of Eurographics, vol. 81 of Linköping Electronic Conference Proceedings, Linköping University Electronic Press, pp. 39–48 (2012). http://www.ep.liu.se/ecp_article/index.en.aspx?issue=081;article=006
  14. 14.
    Bernard, J., Ruppert, T., Scherer, M., Schreck, T., Kohlhammer, J.: Guided discovery of interesting relationships between time series clusters and metadata properties. In: Knowledge Management and Knowledge Technologies (i-KNOW), pp. 22:1–22:8. ACM (2012).  https://doi.org/10.1145/2362456.2362485
  15. 15.
    Bernard, J., Ritter, C., Sessler, D., Zeppelzauer, M., Kohlhammer, J., Fellner, D.: Visual-interactive similarity search for complex objects by example of soccer player analysis. In: IVAPP, VISIGRAPP, pp. 75–87 (2017).  https://doi.org/10.5220/0006116400750087
  16. 16.
    Bernard, J., Sessler, D., Berisch, M., Hutter, M., Schreck, T., Kohlhammer, J.: Towards a user-defined visual-interactive definition of similarity functions for mixed data. In: IEEE Visual Analytics Science and Technology (Poster Paper) (2014).  https://doi.org/10.1109/VAST.2014.7042503
  17. 17.
    Bernard, J., Sessler, D., Bannach, A., May, T., Kohlhammer, J.: A visual active learning system for the assessment of patient well-being in prostate cancer research. In: VIS Workshop on Visual Analytics in Healthcare, pp. 1–8. ACM (2015).  https://doi.org/10.1145/2836034.2836035
  18. 18.
    Bernard, J., Sessler, D., Ruppert, T., Davey, J., Kuijper, A., Kohlhammer, J.: User-based visual-interactive similarity definition for mixed data objects-concept and first implementation. J. WSCG 22, 329–338 (2014)Google Scholar
  19. 19.
    Baeza-Yates, R.A., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Longman (1999)Google Scholar
  20. 20.
    Bernard, J., Zeppelzauer, M., Sedlmair, M., Aigner, W.: A unified process for visual-interactive labeling. In: Sedlmair, M., Tominski, C. (eds.) EuroVis Workshop on Visual Analytics (EuroVA), Eurographics (2017).  https://doi.org/10.2312/eurova.20171123
  21. 21.
    Chen, M., Golan, A.: What may visualization processes optimize? IEEE Trans. Vis. Comput. Graph. (TVCG) 22(12), 2619–2632 (2016).  https://doi.org/10.1109/TVCG.2015.2513410 CrossRefGoogle Scholar
  22. 22.
    Card, S.K., Mackinlay, J.D., Shneiderman, B. (eds.): Readings in Information Visualization: Using Vision to Think. Morgan Kaufmann, San Francisco (1999)Google Scholar
  23. 23.
    Choo, J., Park, H.: Customizing computational methods for visual analytics with big data. IEEE Comput. Graph. Appl. (CG&A) 33(4), 22–28 (2013)CrossRefGoogle Scholar
  24. 24.
    Craik, K. (ed.): The Nature of Explanation. Cambridge University Press, Cambridge (1943)Google Scholar
  25. 25.
    Card, S.K., Robertson, G.G., Mackinlay, J.D.: The information visualizer, an information workspace. In: SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM, pp. 181–186 (1991).  https://doi.org/10.1145/108844.108874
  26. 26.
    Chapelle, O., Schölkopf, B., Zien, A.: Semi-Supervised Learning. Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, MA (2006)CrossRefGoogle Scholar
  27. 27.
    Dagli, C.K., Rajaram, S., Huang, T.S.: Leveraging active learning for relevance feedback using an information theoretic diversity measure. In: Conference on Image and Video Retrieval, pp. 123–132. Springer, Berlin (2006).  https://doi.org/10.1007/11788034_13
  28. 28.
    Elmqvist, N., Fekete, J.-D.: Hierarchical aggregation for information visualization: overview, techniques, and design guidelines. IEEE Trans. Vis. Comput. Graph. (TVCG) 16(3), 439–454 (2010).  https://doi.org/10.1109/TVCG.2009.84 CrossRefGoogle Scholar
  29. 29.
    Endert, A., Fiaux, P., North, C.: Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Trans. Vis. Comput. Graph. 18(12), 2879–2888 (2012).  https://doi.org/10.1109/TVCG.2012.260 CrossRefGoogle Scholar
  30. 30.
    Endert, A., Fiaux, P., North, C.: Semantic interaction for visual text analytics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, pp. 473–482. ACM, New York (2012).  https://doi.org/10.1145/2207676.2207741
  31. 31.
    Endert, A., Han, C., Maiti, D., House, L., Leman, S., North, C.: Observation-level interaction with statistical models for visual analytics. In: 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 121–130 (2011).  https://doi.org/10.1109/VAST.2011.6102449
  32. 32.
    Endert, A., Ribarsky, W., Turkay, C., Wong, B.W., Nabney, I., Blanco, I.D., Rossi, F.: The state of the art in integrating machine learning into visual analytics. In: Computer Graphics Forum (CGF) (2017).  https://doi.org/10.1111/cgf.13092
  33. 33.
    Gleicher, M., Albers, D., Walker, R., Jusufi, I., Hansen, C.D., Roberts, J.C.: Visual comparison for information visualization. Inf. Vis. 10(4), 289–309 (2011).  https://doi.org/10.1177/1473871611416549 CrossRefGoogle Scholar
  34. 34.
    Grabner, H., Bischof, H.: On-line boosting and vision. In: Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on, vol. 1, pp. 260–267. IEEE (2006)Google Scholar
  35. 35.
    Gschwandtner, T., Gärtner, J., Aigner, W., Miksch, S.: A Taxonomy of Dirty Time-Oriented Data, pp. 58–72. Springer, Berlin (2012).  https://doi.org/10.1007/978-3-642-32498-7_5 Google Scholar
  36. 36.
    Gleicher, M.: A framework for considering comprehensibility in modeling. Big Data 4(2), 75–88 (2016).  https://doi.org/10.1089/big.2016.0007 CrossRefGoogle Scholar
  37. 37.
    Hoi, S.C., Jin, R., Lyu, M.R.: Large-scale text categorization by batch mode active learning. In: World Wide Web. ACM, pp. 633–642 (2006).  https://doi.org/10.1145/1135777.1135870.3
  38. 38.
    Heimerl, F., Koch, S., Bosch, H., Ertl, T.: Visual classifier training for text document retrieval. IEEE Trans. Vis. Comput. Graph. (TVCG) 18(12), 2839–2848 (2012)CrossRefGoogle Scholar
  39. 39.
    Höferlin, B., Netzel, R., Höferlin, M., Weiskopf, D., Heidemann, G.: Inter-active learning of ad-hoc classifiers for video visual analytics. In: IEEE Visual Analytics Science and Technology (VAST). IEEE, pp. 23–32 (2012).  https://doi.org/10.1109/VAST.2012.6400492
  40. 40.
    Janetzko, H., Sacha, D., Stein, M., Schreck, T., Keim, D.A., Deussen, O.: Feature-driven visual analytics of soccer data. In: 2014 IEEE Conference on Visual Analytics Science and Technology (VAST), pp. 13–22 (2014).  https://doi.org/10.1109/VAST.2014.7042477
  41. 41.
    Keim, D., Andrienko, G., Fekete, J.-D., Görg, C., Kohlhammer, J., Melançon, G.: Visual Analytics: Definition, Process, and Challenges, pp. 154–175. Springer, Berlin (2008).  https://doi.org/10.1007/978-3-540-70956-5_7 Google Scholar
  42. 42.
    Kandel, S., Heer, J., Plaisant, C., Kennedy, J., van Ham, F., Riche, N.H., Weaver, C., Lee, B., Brodbeck, D., Buono, P.: Research directions in data wrangling: visualizations and transformations for usable and credible data. Inf. Vis. 10(4), 271–288 (2011).  https://doi.org/10.1177/1473871611415994 CrossRefGoogle Scholar
  43. 43.
    Karpinski, M., Macintyre, A.: Polynomial bounds for VC dimension of sigmoidal and general Pfaffian neural networks. J. Comput. Syst. Sci. 54(1), 169–176 (1997).  https://doi.org/10.1006/jcss.1997.1477 MathSciNetCrossRefzbMATHGoogle Scholar
  44. 44.
    Krause, J., Perer, A., Bertini, E.: Infuse: Interactive feature selection for predictive modeling of high dimensional data. IEEE Trans. Vis. Comput. Graph. (TVCG) 20(12), 1614–1623 (2014).  https://doi.org/10.1109/TVCG.2014.2346482 CrossRefGoogle Scholar
  45. 45.
    Lewis, J.M., Ackerman, M., de Sa, V.R.: Human cluster evaluation and formal quality measures: a comparative study. In: Annual Meeting of the Cognitive Science Society (CogSci), pp. 1870–1875 (2012)Google Scholar
  46. 46.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar
  47. 47.
    Losing, V., Hammer, B., Wersing, H.: Incremental on-line learning: a review and comparison of state of the art algorithms. Neurocomputing 275, 1261–1274 (2017).  https://doi.org/10.1016/j.neucom.2017.06.084 CrossRefGoogle Scholar
  48. 48.
    Liu, T.-Y.: Learning to rank for information retrieval. Found. Trends Inf. Retr. 3(3), 225–331 (2009).  https://doi.org/10.1561/1500000016 CrossRefGoogle Scholar
  49. 49.
    Liu, Z., Stasko, J.: Mental models, visual reasoning and interaction in information visualization: a top-down perspective. IEEE Trans. Vis. Comput. Graph. 16(6), 999–1008 (2010).  https://doi.org/10.1109/TVCG.2010.177 CrossRefGoogle Scholar
  50. 50.
    Mamitsuka, N.A.H.: Query learning strategies using boosting and bagging. In: Shavlik, J.W. (ed.) International Conference on Machine Learning (ICML), vol. 1, pp. 1–9. Morgan Kaufmann, Los Altos (1998)Google Scholar
  51. 51.
    Möhrmann, J., Bernstein, S., Schlegel, T., Werner, G., Heidemann, G.: Improving the usability of interfaces for the interactive semi-automatic labeling of large image data sets. In: Jacko, J.A. (ed.) Human-Computer Interaction. Design and Development Approaches, pp. 618–627. Springer, Berlin (2011)Google Scholar
  52. 52.
    Mamani, G.M.H., Fatore, F.M., Nonato, L.G., Paulovich, F.V.: User-driven feature space transformation. Comput. Graph. Forum (CGF) 32(3), 291–299 (2013).  https://doi.org/10.1111/cgf.12116 CrossRefGoogle Scholar
  53. 53.
    Mühlbacher, T., Piringer, H.: A partition-based framework for building and validating regression models. IEEE Trans. Vis. Comput. Graph. (TVCG) 19(12), 1962–1971 (2013).  https://doi.org/10.1109/TVCG.2013.125 CrossRefGoogle Scholar
  54. 54.
    Mühlbacher, T., Piringer, H., Gratzl, S., Sedlmair, M., Streit, M.: Opening the black box: strategies for increased user involvement in existing algorithm implementations. IEEE Trans. Vis. Comput. Graph. 20(12), 1643–1652 (2014)CrossRefGoogle Scholar
  55. 55.
    Mitrović, D., Zeppelzauer, M., Breiteneder, C.: Features for content-based audio retrieval. Adv. Comput. 78, 71–150 (2010)CrossRefGoogle Scholar
  56. 56.
    Norman, D.A.: The Design of Everyday Things, reprint, paperback edn. Basic Books, New York (2002)Google Scholar
  57. 57.
    Olsson, F.: A Literature Survey of Active Machine Learning in the Context of Natural Language Processing, Technical report. Swedish Institute of Computer Science (2009)Google Scholar
  58. 58.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)CrossRefGoogle Scholar
  59. 59.
    Qi, G.-J., Hua, X.-S., Rui, Y., Tang, J., Zhang, H.-J.: Two-dimensional multilabel active learning with an efficient online adaptation model for image classification. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 31(10), 1880–1897 (2009).  https://doi.org/10.1109/TPAMI.2008.218 CrossRefGoogle Scholar
  60. 60.
    Rauber, P.E., Fadel, S.G., Falcao, A.X., Telea, A.C.: Visualizing the hidden activity of artificial neural networks. IEEE Trans. Vis. Comput. Graph. 23(1), 101–110 (2017)CrossRefGoogle Scholar
  61. 61.
    Riek, L.D., OŠconnor, M.F., Robinson, P.: Guess what? a game for affective annotation of video using crowd sourcing. In: International Conference on Affective Computing and Intelligent Interaction, pp. 277–285. Springer, Berlin (2011)Google Scholar
  62. 62.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: Labelme: a database and web-based tool for image annotation. Int. J. Comput. Vis. 77(1), 157–173 (2008)CrossRefGoogle Scholar
  63. 63.
    Sedlmair, M., Aupetit, M.: Data-driven evaluation of visual quality measures. Comput. Graph. Forum (CGF) 34(3), 201–210 (2015).  https://doi.org/10.1111/cgf.12632 CrossRefGoogle Scholar
  64. 64.
    Shurkhovetskyy, G., Andrienko, N., Andrienko, G., Fuchs, G.: Data abstraction for visualizing large time series. Comput. Graph. Forum (CGF) (2017).  https://doi.org/10.1111/cgf.13237 Google Scholar
  65. 65.
    Seifert, C., Aamir, A., Balagopalan, A., Jain, D., Sharma, A., Grottel, S., Gumhold, S.: Visualizations of Deep Neural Networks in Computer Vision: A Survey, pp. 123–144. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-54024-5_6 Google Scholar
  66. 66.
    Salton, G., Buckley, C.: Improving retrieval performance by relevance feedback. Read. Inf. Retr. 24, 5 (1997).  https://doi.org/10.1002/(SICI)1097-4571(199006)41:4%3c288::AID-ASI8%3e3.0.CO;2-H Google Scholar
  67. 67.
    Sessler, D., Bernard, J., Kuijper, A., Kohlhammer, J.: Adopting Mental Similarity Notions of Categorical Data Objects to Algorithmic Similarity functions. (2014). Poster Paper. http://www.vmv2014.gcc.tu-darmstadt.de/sites/program.html
  68. 68.
    Schreck, T., Bernard, J., Von Landesberger, T., Kohlhammer, J.: Visual cluster analysis of trajectory data with interactive kohonen maps. Inf. Vis. 8(1), 14–29 (2009).  https://doi.org/10.1057/ivs.2008.29 CrossRefGoogle Scholar
  69. 69.
    Settles, B., Craven, M.: An analysis of active learning strategies for sequence labeling tasks. In: Empirical Methods in Natural Language Processing, Computational Linguistics, pp. 1070–1079 (2008)Google Scholar
  70. 70.
    Settles, B., Craven, M., Ray, S.: Multiple-instance active learning. In: Advances in Neural Information Processing Systems, pp. 1289–1296 (2008)Google Scholar
  71. 71.
    Settles, B.: Active Learning Literature Survey, Technical Report 1648. University of Wisconsin–Madison (2009)Google Scholar
  72. 72.
    Settles, B.: Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In: Conference on Empirical Methods in Natural Language Processing (EMNLP), Computational Linguistics, pp. 1467–1478 (2011)Google Scholar
  73. 73.
    Settles, B.: Active learning. Synth. Lect. Artif. Intell. Mach. Learn. 6(1), 1–114 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  74. 74.
    Seifert, C., Granitzer, M.: User-based active learning. In: IEEE International Conference on Data Mining Workshops, pp. 418–425 (2010).  https://doi.org/10.1109/ICDMW.2010.181
  75. 75.
    Stasko, J., Görg, C., Liu, Z.: Jigsaw: supporting investigative analysis through interactive visualization. Inf. Vis. 7(2), 118–132 (2008).  https://doi.org/10.1145/1466620.1466622 CrossRefGoogle Scholar
  76. 76.
    Sedlmair, M., Heinzl, C., Bruckner, S., Piringer, H., Möller, T.: Visual parameter space analysis: a conceptual framework. IEEE Trans. Vis. Comput. Graph. (TVCG) 20(12), 2161–2170 (2014).  https://doi.org/10.1109/TVCG.2014.2346321 CrossRefGoogle Scholar
  77. 77.
    Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  78. 78.
    Sedlmair, M., Meyer, M., Munzner, T.: Design study methodology: reflections from the trenches and the stacks. IEEE Trans. Vis. Comput. Graph (TVCG) 18(12), 2431–2440 (2012).  https://doi.org/10.1109/TVCG.2012.213 CrossRefGoogle Scholar
  79. 79.
    Seung, H.S., Opper, M., Sompolinsky, H.: Query by committee. In: Workshop on Computational Learning Theory (COLT), pp. 287–294. ACM, New York (1992).  https://doi.org/10.1145/130385.130417
  80. 80.
    Stolper, C.D., Perer, A., Gotz, D.: Progressive visual analytics: user-driven visual exploration of in-progress analytics. IEEE Trans. Vis. Comput. Graph. 20(12), 1653–1662 (2014)CrossRefGoogle Scholar
  81. 81.
    Sarkar, A., Spott, M., Blackwell, A.F., Jamnik, M.: Visual discovery and model-driven explanation of time series patterns. In: Visual Languages and Human-Centric Computing (VL/HCC). IEEE, pp. 78–86 (2016).  https://doi.org/10.1109/VLHCC.2016.7739668
  82. 82.
    Seebacher, D., Stein, M., Janetzko, H., Keim, D.A.: Patent retrieval: a multi-modal visual analytics approach. In: EuroVis Workshop on Visual Analytics (EuroVA), Eurographics, pp. 013–017 (2016)Google Scholar
  83. 83.
    Sacha, D., Stoffel, A., Stoffel, F., Kwon, B.C., Ellis, G.P., Keim, D.A.: Knowledge generation model for visual analytics. IEEE Trans. Vis. Comput. Graph. (TVCG) 20(12), 1604–1613 (2014).  https://doi.org/10.1109/TVCG.2014.2346481 CrossRefGoogle Scholar
  84. 84.
    Sacha, D., Sedlmair, M., Zhang, L., Lee, J.A., Weiskopf, D., North, S.C., Keim, D.A.: Human-centered machine learning through interactive visualization: review and open challenges. In: Artificial Neural Networks, Computational Intelligence and Machine Learning (2016)Google Scholar
  85. 85.
    Sacha, D., Sedlmair, M., Zhang, L., Lee, J.A., Peltonen, J., Weiskopf, D., North, S.C., Keim, D.A.: What you see is what you can change: human-centered machine learning by interactive visualization. Neurocomputing (2017).  https://doi.org/10.1016/j.neucom.2017.01.105. ISSN = 0925-2312
  86. 86.
    Sacha, D., Zhang, L., Sedlmair, M., Lee, J.A., Peltonen, J., Weiskopf, D., North, S.C., Keim, D.A.: Visual interaction with dimensionality reduction: a structured literature analysis. IEEE Trans. Vis. Comput. Graph. (TVCG) 23(01), 241–250 (2016).  https://doi.org/10.1109/TVCG.2016.2598495 CrossRefGoogle Scholar
  87. 87.
    Turkay, C., Kaya, E., Balcisoy, S., Hauser, H.: Designing progressive and interactive analytics processes for high-dimensional data analysis. IEEE Trans. Vis. Comput. Graph. (TVCG) 23(1), 131–140 (2017)CrossRefGoogle Scholar
  88. 88.
    Tuia, D., Volpi, M., Copa, L., Kanevski, M., Munoz-Mari, J.: A survey of active learning algorithms for supervised remote sensing image classification. IEEE J. Sel. Top. Signal Process. 5(3), 606–617 (2011)CrossRefGoogle Scholar
  89. 89.
    Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Conference on Human Factors in Computing Systems (SIGCHI), pp. 319–326. ACM (2004)Google Scholar
  90. 90.
    Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Berlin (2013)zbMATHGoogle Scholar
  91. 91.
    van der Corput, P., van Wijk, J.J.: Comparing personal image collections with picturevis. Comput. Graph. Forum (CGF) 36(3), 295–304 (2017).  https://doi.org/10.1111/cgf.13188 CrossRefGoogle Scholar
  92. 92.
    van den Elzen, S., van Wijk, J.J.: Baobabview: interactive construction and analysis of decision trees. In: IEEE Visual Analytics Science and Technology (VAST), pp. 151–160 (2011).  https://doi.org/10.1109/VAST.2011.6102453
  93. 93.
    Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)CrossRefGoogle Scholar
  94. 94.
    Vendrig, J., Patras, I., Snoek, C., Worring, M., den Hartog, J., Raaijmakers, S., van Rest, J., van Leeuwen, D.A.: Trec feature extraction by active learning. In: TREC (2002)Google Scholar
  95. 95.
    Visentini, I., Snidaro, L., Foresti, G.L.: On-line boosted cascade for object detection. In: Pattern Recognition, 2008. ICPR 2008. 19th International Conference on, pp. 1–4. IEEE (2008)Google Scholar
  96. 96.
    van Wijk, J.J.: The value of visualization. In: VIS 05. IEEE Visualization, 2005, pp. 79–86 (2005).  https://doi.org/10.1109/VISUAL.2005.1532781
  97. 97.
    Wall, E., Das, S., Chawla, R., Kalidindi, B., Brown, E.T., Endert, A.: Podium: ranking data using mixed-initiative visual analytics. IEEE Trans. Vis. Comput. Graph. 24(1), 288–297 (2018)CrossRefGoogle Scholar
  98. 98.
    Wang, M., Hua, X.-S.: Active learning in multimedia annotation and retrieval: a survey. CM Trans. Intell. Syst. Technol. 2(2), 10:1–10:21 (2011).  https://doi.org/10.1145/1899412.1899414 Google Scholar
  99. 99.
    Wu, Y., Kozintsev, I., Bouguet, J.-Y., Dulong, C.: Sampling strategies for active learning in personal photo retrieval. In: IEEE International Conference on Multimedia and Expo. IEEE, pp. 529–532 (2006).  https://doi.org/10.1109/ICME.2006.262442
  100. 100.
    Wenskovitch, J., North, C.: Observation-level interaction with clustering and dimension reduction algorithms. In: Workshop on Human-In-the-Loop Data Analytics (HILDA). ACM, pp. 14:1–14:6 (2017).  https://doi.org/10.1145/3077257.3077259
  101. 101.
    Wongsuphasawat, K., Smilkov, D., Wexler, J., Wilson, J., Mané, D., Fritz, D., Krishnan, D., Viégas, F.B., Wattenberg, M.: Visualizing dataflow graphs of deep learning models in tensorflow. IEEE Trans. Vis. Comput. Graph. 24(1), 1–12 (2018).  https://doi.org/10.1109/TVCG.2017.2744878 CrossRefGoogle Scholar
  102. 102.
    Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 27, pp. 3320–3328. Curran Associates Inc, New York (2014)Google Scholar
  103. 103.
    Yosinski, J., Clune, J., Nguyen, A., Fuchs, T., Lipson, H.: Understanding Neural Networks Through Deep Visualization (2015). arXiv preprint arXiv:1506.06579
  104. 104.
    Yang, L., Jin, R.: Distance metric learning: a comprehensive survey. Mich. State Univ. 2, 2 (2006)Google Scholar
  105. 105.
    Zeiler, M.D., Fergus, R.: Visualizing and Understanding Convolutional Networks, pp. 818–833. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10590-1_53 Google Scholar
  106. 106.
    Zhu, Q., Keogh, E.J.: Using captchas to index cultural artifacts. In: International Symposium on Advances in Intelligent Data Analysis IX, pp. 245–257. Springer, Berlin (2010)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.TU DarmstadtDarmstadtGermany
  2. 2.St. Pölten University of Applied SciencesSt. PöltenAustria
  3. 3.Jacobs UniversityBremenGermany
  4. 4.IGDFraunhofer Institute for Computer Graphics ResearchDarmstadtGermany
  5. 5.TU WienViennaAustria

Personalised recommendations