Advertisement

Evaluating Information Visualizations

  • Sheelagh Carpendale
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4950)

Abstract

Information visualization research is becoming more established, and as a result, it is becoming increasingly important that research in this field is validated. With the general increase in information visualization research there has also been an increase, albeit disproportionately small, in the amount of empirical work directly focused on information visualization. The purpose of this chapter is to increase awareness of empirical research in general, of its relationship to information visualization in particular; to emphasize its importance; and to encourage thoughtful application of a greater variety of evaluative research methodologies in information visualization.

Keywords

Empirical Research Qualitative Method Visualization Technique Sage Publication Information Visualization 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Amar, R.A., Stasko, J.T.: Knowledge Precepts for Design and Evaluation of Information Visualizations. IEEE Transactions on Visualization and Computer Graphics 11(4), 432–442 (2005)CrossRefGoogle Scholar
  2. 2.
    Andrews, K.: Evaluating Information Visualisations. In: Proceedings of the 2006 AVI Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, pp. 1–5 (2006)Google Scholar
  3. 3.
    Auerbach, C.: Qualitative Data: An Introduction to Coding and Analysis. University Press, New York (2003)Google Scholar
  4. 4.
    Baker, K., Greenberg, S., Gutwin, C.: Empirical Development of a Heuristic Evaluation Methodology for Shared Workspace Groupware. In: Proceedings of the ACM Conference on Computer Supported Cooperative Work, pp. 96–105. ACM Press, New York (2002)CrossRefGoogle Scholar
  5. 5.
    Baldonado, M., Woodruff, A., Kuchinsky, A.: Guidelines for Using Multiple Views in Information Visualization. In: Proeedings of the Conference on Advanced Visual Interfaces (AVI), pp. 110–119. ACM Press, New York (2000)CrossRefGoogle Scholar
  6. 6.
    Barzun, J., Graff, H.: The Modern Researcher, 3rd edn. Harcourt Brace Jovanvich, New York (1977)Google Scholar
  7. 7.
    BELIV 2006, accessed http://www.dis.uniroma1.it/~beliv06/ (February 4, 2008)
  8. 8.
    Bertin, J.: Semiology of Graphics (Translation: William J. Berg). University of Wisconsin Press (1983)Google Scholar
  9. 9.
    Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J., Wixon, W.: The “Magic Number 5”: Is It Enough for Web Testing? In: CHI Extended Abstracts, pp. 698–699. ACM Press, New York (2003)Google Scholar
  10. 10.
    Boyatzis, R.: Transforming Qualitative Information: Thematic Analysis and Code Development. Sage Publications, London (1998)Google Scholar
  11. 11.
    Brereton, M., McGarry, B.: An Observational Study of How Objects Support Engineering Design Thinking and Communication: Implications for the Design of Tangible Media. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’00), pp. 217–224. ACM Press, New York (2000)CrossRefGoogle Scholar
  12. 12.
    Chen, C., Czerwinski, M.: Introduction to the Special Issue on Empirical Evaluation of Information Visualizations. International Journal of Human-Computer Studies 53(5), 631–635 (2000)CrossRefGoogle Scholar
  13. 13.
    Corbin, J., Strauss, A.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 3rd edn. Sage Publications, Los Angeles (2008)Google Scholar
  14. 14.
    Creswell, J.: Qualitative Inquiry and Research Design: Choosing Among Five Traditions. Sage Publications, London (1998)Google Scholar
  15. 15.
    Dix, A., Finlay, J., Abowd, G., Beale, R.: Human Computer Interaction, 2nd edn. Prentice-Hall, Englewood Cliffs (1998)Google Scholar
  16. 16.
    Dumais, S., Cutrell, E., Chen, H.: Optimizing Search by Showing Results In Context. In: Proc. CHI’01, pp. 277–284. ACM Press, New York (2001)Google Scholar
  17. 17.
    Eberts, R.E.: User Interface Design. Prentice-Hall, Englewood Cliffs (1994)zbMATHGoogle Scholar
  18. 18.
    Ellis, E., Dix, A.: An Explorative Analysis of User Evaluation Studies in Information Visualization. In: Proceedings of the Workshop on Beyond Time and Errors: Novel Evaluation Methods for Information Visualization, BELIV (2006)Google Scholar
  19. 19.
    Ericsson, K., Simon, H.: Verbal Reports as Data. Psychological Review 87(3), 215–251 (1980)CrossRefGoogle Scholar
  20. 20.
    Ericsson, K., Simon, H.: Verbal Reports on Thinking. In: Faerch, C., Kasper, G. (eds.) Introspection in Second Language Research, pp. 24–54. Multilingual Matters, Clevedon, Avon (1987)Google Scholar
  21. 21.
    Ericsson, K., Simon, H.: Protocol Analysis: Verbal Reports as Data, 2nd edn. MIT Press, Boston (1993)Google Scholar
  22. 22.
    Fall, J., Fall, A.: SELES: A Spatially Explicit Landscape Event Simulator. In: Proceedings of GIS and Environmental Modeling, pp. 104–112. National Center for Geographic Information and Analysis (1996)Google Scholar
  23. 23.
    Forlines, C., Shen, C., Wigdor, D., Balakrishnan, R.: Exploring the effects of group size and display configuration on visual search. In: Computer Supported Cooperative Work 2006 Conference Proceedings, pp. 11–20 (2006)Google Scholar
  24. 24.
    Garfinkel, H.: Studies in Ethnomethodology. Polity Press, Cambridge (1967)Google Scholar
  25. 25.
    Gonzalez, V., Kobsa, A.: A Workplace Study of the Adoption of Information Visualization systems. In: Proceedings of the International Conference on Knowledge Management, pp. 92–102 (2003)Google Scholar
  26. 26.
    Gorard, S.: Combining Methods in Educational Research. McGraw-Hill, New York (2004)Google Scholar
  27. 27.
    Greenberg, S., Buxton, B.: Usability Evaluation Considered Harmful (Some of the Time). In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2008)Google Scholar
  28. 28.
    Greene, J., Caracelli, V., Graham, W.: Toward a Conceptual Framework for Mixed-Method Evaluation Design. Educational Evaluation and Policy Analysis 11(3), 255–274 (1989)Google Scholar
  29. 29.
    Gutwin, C., Greenberg, S.: The Mechanics of Collaboration: Developing Low Cost Usability Evaluation Methods for Shared Workspaces. In: Proceedings WETICE, pp. 98–103. IEEE Computer Society Press, Los Alamitos (2000)Google Scholar
  30. 30.
    Healey, C.G.: On the Use of Perceptual Cues and Data Mining for Effective Visualization of Scientific Datasets. In: Proceedings of Graphics Interface, pp. 177–184 (1998)Google Scholar
  31. 31.
    Heer, J., Viegas, F., Wattenberg, M.: Voyagers and Voyeurs: Supporting Asynchronous Collaborative Information Visualization. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI’07), pp. 1029–1038. ACM Press, New York (2007)Google Scholar
  32. 32.
    Holtzblatt, K., Beyer, H.: Contextual Design: Defining Customer-Centered Systems. Morgan Kaufmann, San Francisco (1998)Google Scholar
  33. 33.
    Huck, S.W.: Reading Statistics and Research, 4th edn. Pearson Education Inc., Boston (2004)Google Scholar
  34. 34.
    Interrante, V.: Illustrating Surface Shape in Volume Data via Principal Direction-Driven 3D Line Integral Convolution. Computer Graphics, Annual Conference Series, pp. 109–116 (1997)Google Scholar
  35. 35.
    Isenberg, P., Carpendale, S.: Interactive Tree Comparison for Co-located Collaborative Information Visualization. IEEE Transactions on Visualization and Computer Graphics 12(5) (2007)Google Scholar
  36. 36.
    Isenberg, P., Tang, A., Carpendale, S.: An Exploratory Study of Visual Information Analysis. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI’08), ACM Press, New York (to appear, 2008)Google Scholar
  37. 37.
    Kay, J., Reiger, H., Boyle, M., Francis, G.: An Ecosystem Approach for Sustainability: Addressing the Challenge of Complexity. Futures 31(7), 721–742 (1999)CrossRefGoogle Scholar
  38. 38.
    Kim, S., Hagh-Shenas, H., Interrante, V.: Conveying Shape with Texture: Experimental Investigations of Texture’s Effects on Shape Categorization Judgments. IEEE Transactions on Visualization and Computer Graphics 10(4), 471–483 (2004)CrossRefGoogle Scholar
  39. 39.
    Kleffner, D.A., Ramachandran, V.S.: On the Perception of Shape from Shading. Perception and Psychophysics 52(1), 18–36 (1992)Google Scholar
  40. 40.
    Kobsa, A.: User Experiments with Tree Visualization Systems. In: Proceedings of the IEEE Symposium on Information Visualization, pp. 9–26 (2004)Google Scholar
  41. 41.
    Kruger, R., Carpendale, S., Scott, S.D., Greenberg, S.: Roles of Orientation in Tabletop Collaboration: Comprehension, Coordination and Communication. Journal of Computer Supported Collaborative Work 13(5–6), 501–537 (2004)CrossRefGoogle Scholar
  42. 42.
    Lewin, C. (ed.): Research Methods in the Social Sciences. Sage Publications, London (2004)Google Scholar
  43. 43.
    Lewis, C., Rieman, J.: Task-Centered User Interface Design: A Practical Introduction (1993)Google Scholar
  44. 44.
    Likert, R.: A Technique for the Measurement of Attitudes. Archives of Psychology 140, 1–55 (1932)Google Scholar
  45. 45.
    Lofland, J., Lofland, L.: Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. Wadsworth Publishing Company, CA, USA (1995)Google Scholar
  46. 46.
    Mankoff, J., Dey, A., Hsieh, G., Kientz, J., Lederer, S., Ames, A.: Heuristic Evaluation of Ambient Displays. In: Proceedings of CHI ’03, pp. 169–176. ACM Press, New York (2003)Google Scholar
  47. 47.
    Mark, G., Kobsa, A., Gonzalez, V.: Do Four Eyes See Better Than Two? Collaborative Versus Individual Discovery in Data Visualization Systems. In: Proceedings of the IEEE Conference on Information Visualization (IV’02), July 2002, pp. 249–255. IEEE Press, Los Alamitos (2002)CrossRefGoogle Scholar
  48. 48.
    Mazza, R., Berre, A.: Focus Group Methodology for Evaluating Information Visualization Techniques and Tools. In: Proceedings of the International Conference on Information Visualization IV (2007)Google Scholar
  49. 49.
    McCarthy, D.: Normal Science and Post-Normal Inquiry: A Context for Methodology (2004)Google Scholar
  50. 50.
    McGrath, J.: Methodology Matters: Doing Research in the Social and Behavioural Sciences. In: Readings in Human-Computer Interaction: Toward the Year 2000, Morgan Kaufmann, San Francisco (1995)Google Scholar
  51. 51.
    Moggridge, B.: Design Interactions. MIT Press, Cambridge (2006)Google Scholar
  52. 52.
    Morris, M.R., Ryall, K., Shen, C., Forlines, C., Vernier, F.: Beyond “Social Protocols”: Multi-User Coordination Policies for Co-located Groupware. In: Proceedings of the ACM Conference on Computer-Supported Cooperative Work (CSCW, Chicago, IL, USA), CHI Letters, November 6-10, 2004, pp. 262–265. ACM Press, New York (2004)Google Scholar
  53. 53.
    Morse, E., Lewis, M., Olsen, K.: Evaluating Visualizations: Using a Taxonomic Guide. Int. J. Human-Computer Studies 53, 637–662 (2000)zbMATHCrossRefGoogle Scholar
  54. 54.
    Nielsen, J., Mack, R.: Usability Inspection Methods. John Wiley & Sons, Chichester (1994)Google Scholar
  55. 55.
    North, C.: Toward Measuring Visualization Insight. IEEE Computer Graphics and Applications 26(3), 6–9 (2006)CrossRefGoogle Scholar
  56. 56.
    Patton, M.Q.: Qualitative Research and Evaluation Methods, 3rd edn. Sage Publications, London (2001)Google Scholar
  57. 57.
    Plaisant, C.: The Challenge of Information Visualization Evaluation. In: Proceedings of the Working Conference on Advanced Visual Interfaces, pp. 109–116 (2004)Google Scholar
  58. 58.
    Purchase, H.C., Hoggan, E., Görg, C.: How Important Is the “Mental Map”? – An Empirical Investigation of a Dynamic Graph Layout Algorithm. In: Kaufmann, M., Wagner, D. (eds.) GD 2006. LNCS, vol. 4372, pp. 184–195. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  59. 59.
    Purchase, H.C.: Effective Information Visualisation: A Study of Graph Drawing Aesthetics and Algorithms. Interacting with Computers 13(2), 477–506 (2000)CrossRefGoogle Scholar
  60. 60.
    Purchase, H.C.: Performance of Layout Algorithms: Comprehension, Not Computation. Journal of Visual Languages and Computing 9, 647–657 (1998)CrossRefGoogle Scholar
  61. 61.
    Brandenburg, F.J. (ed.): GD 1995. LNCS, vol. 1027. Springer, Heidelberg (1996)Google Scholar
  62. 62.
    Reilly, D., Inkpen, K.: White Rooms and Morphing Don’t Mix: Setting and the Evaluation of Visualization Techniques. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 111–120 (2007)Google Scholar
  63. 63.
    Saraiya, P., North, C., Duca, K.: An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. IEEE Transactions on Visualization and Computer Graphics 11(4), 443–456 (2005)CrossRefGoogle Scholar
  64. 64.
    Scott, S.D., Carpendale, S., Inkpen, K.: Territoriality in Collaborative Tabletop Workspaces. In: Proceedings of the ACM Conference on Computer-Supported Cooperative Work (CSCW, Chicago, IL, USA), CHI Letters, November 6-10, 2004, pp. 294–303. ACM Press, New York (2004)Google Scholar
  65. 65.
    Seidman, I.: Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences. Teachers’ College Press, New York (1998)Google Scholar
  66. 66.
    Shneiderman, B.: The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations. In: Proceedings of the IEEE Symposium on Visual Languages, pp. 336–343. IEEE Computer Society Press, Los Alamitos (1996)CrossRefGoogle Scholar
  67. 67.
    Shneiderman, B., Plaisant, C.: Strategies for Evaluating Information Visualization Tools: Multi-Dimensional In-Depth Long-Term Case Studies. In: Proceedings of the Workshop on BEyond Time and Errors: Novel Evaluation Methods for Information Visualization, BELIV (2006)Google Scholar
  68. 68.
    Spence, R.: Information Visualization, 2nd edn. Addison-Wesley, Reading (2007)Google Scholar
  69. 69.
    Spool, J., Schroeder, W.: Testing Web Sites: Five Users is Nowhere Near Enough. In: CHI ’01 Extended Abstracts, pp. 285–286. ACM Press, New York (2001)CrossRefGoogle Scholar
  70. 70.
    Strauss, A.L., Corbin, J.: Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Sage Publications, London (1998)Google Scholar
  71. 71.
    Tang, A., Tory, M., Po, B., Neumann, P., Carpendale, S.: Collaborative Coupling over Tabletop Displays. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI’06), pp. 1181–1290. ACM Press, New York (2006)Google Scholar
  72. 72.
    Tang, A., Carpendale, S.: An observational study on information flow during nurses’ shift change. In: Proc. of the ACM Conf. on Human Factors in Computing Systems (CHI), pp. 219–228. ACM Press, New York (2007)Google Scholar
  73. 73.
    Tang, J.C.: Findings from observational studies of collaborative work. International Journal of Man-Machine Studies 34(2), 143–160 (1991)CrossRefGoogle Scholar
  74. 74.
    Tory, M., Möller, T.: Evaluating Visualizations: Do Expert Reviews Work. IEEE Computer Graphics and Applications 25(5), 8–11 (2005)CrossRefGoogle Scholar
  75. 75.
    Tufte, E.: The Visual Display of Quantitative Information. Graphics Press, Cheshire (1986)Google Scholar
  76. 76.
    Tufte, E.: Envisioning Information. Graphics Press, Cheshire (1990)Google Scholar
  77. 77.
    Tufte, E.: Visual Explanations. Images and Quantities, Evidence and Narrative. Graphics Press, Cheshire (1997)zbMATHGoogle Scholar
  78. 78.
    Viegas, F.B., Wattenberg, M., van Ham, F., Kriss, J., McKeon, M.: Many Eyes: A Site for Visualization at Internet Scale. IEEE Transactions on Visualization and Computer Graphics (Proceedings Visualization / Information Visualization 2007) 12(5), 1121–1128 (2007)CrossRefGoogle Scholar
  79. 79.
    Ware, C.: Information Visualization: Perception for Design, 2nd edn. Morgan Kaufmann, San Francisco (2004)Google Scholar
  80. 80.
    Wigdor, D., Shen, C., Forlines, C., Balakrishnan, R.: Perception of Elementary Graphical Elements in Tabletop and Multi-surface Environments. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI’07), pp. 473–482. ACM Press, New York (2007)Google Scholar
  81. 81.
    Willett, W., Heer, J., Agrawala, M.: Scented Widgets: Improving Navigation Cues with Embedded Visualizations. In: INFOVIS 2007. IEEE Symposium on Information Visualization (2007)Google Scholar
  82. 82.
    Yost, B., North, C.: The Perceptual Scalability of Visualization. IEEE Transactions on Visualization and Computer Graphics 12(5), 837–844 (2006)CrossRefGoogle Scholar
  83. 83.
    Zuk, T.: Uncertainty Visualizations. PhD thesis. Department of Compute Science, University of Calgary (2007)Google Scholar
  84. 84.
    Zuk, T., Carpendale, S.: Theoretical Analysis of Uncertainty Visualizations. In: Proceedings of SPIE Conference Electronic Imaging, Vol. 6060: Visualization and Data Analysis (2006)Google Scholar
  85. 85.
    Zuk, T., Schlesier, L., Neumann, P., Hancock, M.S., Carpendale, S.: Heuristics for Information Visualization Evaluation. In: Proceedings of the Workshop BEyond Time and Errors: Novel Evaluation Methods for Information Visualization (BELIV 2006), held in conjunction with the Working Conference on Advanced Visual Interfaces (AVI 2006), ACM Press, New York (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Sheelagh Carpendale
    • 1
  1. 1.Department of Computer ScienceUniversity of CalgaryCalgaryCanada

Personalised recommendations