Cognitive Ergonomics in Visualization
A key factor to assure success of a visualization technique is how efficiently users perceive information using this visualization technique. This efficiency is strongly correlated with visualization parameters which we will summarize in the following with the term cognitive ergonomics. In this chapter we will motivate the reader to study cognitive ergonomics in visualization using an interdisciplinary approach. This interdisciplinary approach is based on eye tracking data visualization, ontology based visualization models, and cognitive simulations. Figure 1 shows the chapter structure, and the interdisciplinary approach we suggest.
KeywordsVisualization Technique Declarative Knowledge Graphical Element Cognitive Ergonomic Visual Search Strategy
Many user experiments have been conducted to study perception of visualization techniques. Apart from measuring accuracy rates and completion times, eye tracking experiments provide an additional technique to evaluate visualizations. Due to eye tracking devices becoming cheaper, eye tracking is a promising approach to study visualization parameters that are relevant for cognitive ergonomics. The results of eye tracking experiments allow researchers to investigate scan paths of eye fixations on the stimulus. Thereby, researchers can measure which areas on the stimulus have been focused on in which order, by which participants, and with how many fixations. Overall, eye tracking metrics allow evaluating cognitive stress by statistically analyzing fixation durations, distributions, and sequences, or cognitive workload by studying changes of pupil size of participants.
Besides using statistical algorithms to compare eye tracking metrics, visualization techniques allow to visually analyze fixation durations, distributions and sequences of several participants at one glance. However, only state of the art visualization techniques are usually used, such as scan path or heat map visualizations (cf. Fig. 2). For this reason, we are motivating readers to develop further visualization techniques for eye tracking data analysis in the first part of this chapter. We will analyze the structure of eye tracking data, and the visual analysis process of eye tracking results from an information visualization pipeline perspective. This systematic approach will help visualization developers to find new visualization techniques for graphical representations of eye tracking data. We will conclude this part with the presentation of the parallel scan path visualization technique which we have developed following this systematic approach.
Today, the development of visualizations is mostly driven by a technical perspective. The main goal of visualization research is to visualize as many data points in real time on high resolution screens as possible. However, there are now tendencies towards a user-centered design of visualizations. The user-centered design takes effects into account such as perception of graphical representations and cognitive workload. The second part of this chapter demonstrates how this user-centered design paradigm can be applied in a visualization scenario. In this scenario we annotate visualizations with semantic information to allow viewers to customize their visualizations.
The discussion of cognitive ergonomics is strongly related to the question if metrics such as fixation durations, distributions, and sequences can be modeled and then be simulated? If this question can be answered positively, the high effort which is required for the preparation, and conduction of eye tracking experiments could be reduced. In the future, interesting visual tasks and visualization parameters could be selected in advance by running a simulated experiment without conducting any experiments with real participants. This approach is mainly inspired by the successful application of cognitive simulations in human computer interaction (HCI) research. The third part of this chapter will give a brief motivation for using cognition simulations to test visualization designs. Therefore, we will discuss interesting aspects of CogTool from HCI, and the cognitive simulation framework ACT-R.
Finally, we will conclude the chapter by bringing together approaches, concepts, and techniques presented in this chapter to formulate a road map to study cognitive ergonomics of visualizations in future work.
1 Eye Tracking Data Visualization
Usually, the goal of eye tracking experiments is to find common eye movement patterns. One approach is to use visualizations which show the eye movements of several participants in an appropriate way to support finding common structures. Classic techniques to do this are heat maps and traditional scan path visualizations (cf. Fig. 2) . Enhancements to these classic techniques have been presented by Aula et al. who developed a non-overlapping scan path visualization technique . Another technique is used by eSeeTrack which combines a time line and a tree-structured visual representation to extract patterns of sequential gaze orderings. Displaying these patterns does not depend on the number of fixations on a scene . If areas of interest are available, transition matrices [5, 6] or string editing algorithms can be used [7, 8, 9]. A relatively new approach is presented by Andrienko et al. . In their work, the authors discuss the application of visual analytics techniques for the analysis of recorded eye movement data. As a follow-up work, Burch et al. demonstrate how visual analytics techniques can be used to analyze an eye tracking experiment .
One example of an eye tracking study in visualization research is the comparison of different types of graph layouts such as radial, orthogonal, and traditional by Burch et al. . Another example is the eye tracking experiment by Huang et al. Their results show that graphs are read following a geodesic-path tendency. As a result, links which go towards the target node are more likely to be searched first . Another experiment by Kim et al. investigates the influence of peripheral vision during the perception of visualizations .
However, fundamental questions of using eye tracking to test the cognitive ergonomics of visualizations remain. The most important issue about using eye tracking techniques to measure cognitive ergonomics is the question, if the recorded eye movements reflect mental processes, what often is called the “Eye-Mind Hypothesis”? We think, that this question cannot be answered with a definite “yes” or “no”, and refer to the literature [15, 16]. Our opinion is, that the answer depends on the complexity of the visualization, the visual task and the required mental processes. Another relevant point for visual analysis of eye tracking data is, that scan paths can have completely different shapes for different participants performing the same task. The question of how these different eye movement patterns could be compared with each other is still not sufficiently answered. However, we think that new visualization techniques for eye tracking data can bring benefit to scan path comparison.
In the following sub chapters, we will analyze the structure of eye tracking data from the visualization pipeline perspective to motivate the reader to develop further visualization techniques. In a second sub chapter we will present the parallel scan path visualization technique as a result of this analysis.
1.1 New Visualization Techniques for Eye Tracking Data
The visualization pipeline defines four steps for deriving a graphical representation from raw data. In the original work of Naber and McNabb these four steps are: data analysis, filtering, mapping, and rendering . In the following, we will formulate a concept for developing new visualization techniques for visual eye tracking data analysis.
Step 1: Analysis of Eye Tracking Data
Current eye tracking software systems generate large amounts of data representing the output of eye tracker sensors. The most important types of the data sets are: various types of timestamps, fixation point information for left/right/both eye(s), pupil size for left and right eye, software meta data.
Step 2: Filtering
Step 3: Mapping
The data selection from step two has to be mapped to geometrical shapes. At first, we have to choose a visualization concept. Even though it seems to be too simple, we can visualize eye tracking data with one dimension using a number line. In this visualization scenario one number line could represent the temporal characteristics of this data. If we want to visualize more information about participants’ eye movements on a screen, two dimensional diagrams can be used. For example, heat map and scan path visualizations use Cartesian coordinate systems to display the positions of the eye movement elements. We can get inspirations from visualization collections like http://www.visualcomplexity.com or http://infosthetics.com/ to find an adequate visualization technique.
Besides geometrical dimensions of visualizations, colors can indicate additional characteristic of the eye tracking data. Scan paths from different participants can be distinguished using a color table, or can be colored differently when intersecting with areas of interest. Alternatively, interesting characteristics of the eye movements, like high eye movement frequencies, can be displayed using color gradients. Other data dimensions can be mapped to different types of symbols.
Step 4: Rendering
Finally, the filtered and mapped data is rendered to the screen. Thereby, existing rendering libraries for information visualizations can be used.
1.2 Parallel Scan Path Visualization (PSP)
Key feature of the parallel scan path visualization (PSP) is the visualization of eye movements of many participants in a single visualization with a parallel layout containing various levels of detail, such as fixations, gaze durations, eye shift frequencies, and time.
Figure 4 top right shows an example stimulus together with AOIs of an experiment where participants had to answer the question “Why is the road wet?”. Figure 4 bottom left shows one fixation sequence using the parallel scan-path visualization, Fig. 4 bottom right the traditional scan path visualization of the same fixation sequence. A fixation sequence could be to first focus on the road (1), then on the puddle (2), on the cloud (3), on the sun (4), and on the fire hydrant (5). Finally, the attention would move to the puddle again (6). Using the PSP visualization changes of participant’s attention can be studied by following the fixation sequence line in the visualization.
2 Ontology Based Visualization Models
Visualizations often don’t have a unique meaning. They can be interpreted differently by several viewers due to different start conditions of their interpretation, such as cultural or intellectual differences. Also a different context of use can lead to different interpretations of the information which is represented graphically. To avoid these misunderstandings this chapter describes a method to annotate visualizations and their graphical elements with semantic information1. We annotate visualizations on two levels: the visualization concept level and the graphical elements. Every graphical element represents a piece of graphically encoded information. We propose to link every graphical element with a semantic web resource. This concept will allow viewers to customize their visualizations and thus, to close the user viewer gap discussed by Norman .
Graphical representations often don’t have a unique meaning. This chapter describes a method to avoid these misunderstandings. Therefore, we propose to annotate visualizations on the visualization concept level and on the graphical elements level. Finally, we will show how this annotation will allow users to individually optimize their visualizations.
Our literature research has shown that one of the most applications of annotated visualizations is to better find graphical elements inside a visualization . Only few approaches deal with the question how semantic annotations can improve the understanding of visualized information. For example, Janeck and Pu show how annotations can be used to find related information about the presented graphics . Other approaches use annotations to allow a semantic filtering  or an intelligent zooming .
2.1 User Viewer Gap
2.2 Interaction Model for a User Centered Visualization Optimization
The visualization designer annotates both graphical elements and the visualization concept with resources from domain ontologies and resources from graphic ontologies (cf. Fig. 6 left side). Every graphical element is assigned one or more URIs (Unified Resource Identifier).
The visualization viewer can explore the visualization using the assigned annotations (cf. Fig. 6 right side). The viewer can replace graphical elements or change the visualization concept. Dependencies and restrictions of the ontologies guarantee a simultaneous persistence of the meaning of the visualization and its graphical elements.
2.3 Future Questions
As described in Sect. 1.1 the visualization process can systematically be divided up into several steps. The last step of every visualization presentation is the rendering step. Thereby, the rendering is always based on parameters such as the geometrical layout, shapes, and colors from the steps before. These parameters are defined by the visualization designer. During the implementation we asked ourself how a renderer which is based on semantic information could look like? What are important input parameters from the semantic models to the renderer algorithms? How can a visualization layout be described in a semantic model?
We developed a WIMP (Windows, Icons, Menus and Pointer) prototype, where different visualization concepts and graphical elements could be chosen via pull down menus. Due to that, the prototype provided a very simple interaction concept, the question remains how more powerful HCI concepts can be used to improve the user center optimization of visualizations?
One important drawback of our ontology based concept is, that there does not exist a well defined comprehensive ontology for visualizations and their graphical elements. How could an ontology look like? What are important semantic elements of a visualization? What are their relations? One starting point could be the VISO ontology .
And finally, what are other possible applications of semantic annotated visualizations?
3 Cognitive Simulations
Cognitive simulation frameworks provide a promising simulation technique to study formalized cognitive processes during the perception of visualizations. In general, cognitive scientists who are using cognitive simulations aim at using results of psychological experiments to develop models for mental processes which are processes by these simulation frameworks. Cognitive simulations are used to model a wide field of human behavior from problem solving, planning, learning, knowledge representation over natural language processing, perception, expert systems, psychological modeling, to robotics, and human computer interaction. This section motivates using the simulation framework ACT-R in visualization research to model visual search, and the perception of graphics. This section concludes with a brief presentation of results from the successful application of ACT-R in the HCI simulation tool CogTool.
This section presents the basic concept of the cognition simulation framework ACT-R and motivates for using this framework to study aspects of cognitive ergonomics in visualization.
3.1 Brief Introduction to the Adaptive Control of Thought-Rational Simulation Framework (ACT-R)
ACT-R is a modular cognitive architecture, using a production system to operate on symbolic representations of declarative memory2 . In its core the ACT-R system comes with a visual module for the identification of objects in the visual field, a manual module for hand control, a declarative module for retrieving information from the memory, and an intentional module for the current action goals and intentions. All modules are coordinated through a central production system, which can respond to a limited amount of information in the buffers of the visual, manual, declarative, or goal module. This central production system can recognize patterns in these buffers, and make changes to theses buffers. The buffers form one of the fundamental parts of the ACT-R framework, and are noted to cortical regions.
The ACT-R architecture divides knowledge up into two categories: declarative knowledge and procedural knowledge. The declarative knowledge represents factual knowledge. For example, declarative knowledge describes what the parts of a bicycle are. Declarative knowledge is processed with so called chunks. The procedural knowledge describes actions, for example how parts of a bicycle have to be used in order to drive it. Procedural knowledge is describe by production rules. Pattern matching algorithms allow the production system to find appropriate production rules for declarative knowledge chunks in the retrieval buffer considering a given goal in the goal buffer.
The framework is mainly written in a LISP dialect, and uses several modules which represent different brain areas. The modules are connected via buffers. Information between the modules is exchanged trough these buffers. ACT-R uses several metrics to measure cognitive activities. These metrics allow the comparison of the simulation results with results from psychological experiments or fMRI images. A comfortable graphical user interface allows to set up all simulation parameters, and to view the simulation results.
Additional to the built-in visual module of ACT-R Salvucci et al. have developed the “Eye Movements and Movement of Attention” (EMMA) module . This ACT-R module is used to calculate fixation positions during processing a visual search task. EMMA extends the built-in visual module by taking into account effects of fixation frequencies and foveal eccentricity when encoding visual objects. EMMA can predict timings and positions of when and where eyes move, and hence serves to relate high-level cognitive processes with low-level eye movement behavior.
3.2 CogTool - Simulation of Human Computer Interaction
CogTool describes the graphical user interface of applications by frames which are views of an application (this could be a dialoque or a complete window). Changes between different frames are called transitions which describe interactions leading to a transition. Standard transitions are keystrokes or mouse actions. Therefore, CogTool uses an enhanced keystroke level model (KLM) . Perception and visual search is modeled via EMMA. CogTool provides operators for eye movement preparation, eye movement execution, vision encoding, system wait, cognition, key presses, cursor moves, mouse clicks, and simple hand movements. These operators are lined up in parallel. For example, the user can move the mouse and can think in parallel. The computation of the duration times of the single operators is done via ACT-R. Thus, duration times are not fix like in the KLM model. They can be different depending on their point in time during the interaction process.
4 Bringing Everything Together - Roadmap to Study Cognitive Ergonomics of Visualization
We think that by combining results from eye tracking data analysis, semantic models, and cognitive simulation frameworks the cost of constructing cognitive models and running simulations of visualization perception can be reduced.
Once valid models are available the overall effort for conducting user experiments can be reduced by running simulated pre-studies. To reach this goal, we propose to develop a simulation tool similar to CogTool in visualization research. Figure 8 shows a sketch of how the three presented topics can be combined. Eye tracking is used to analyze scan paths from user studies. This analysis is done both with respect to the fixation distribution on the screen (WHERE space) and with respect to the semantic structure of the scan paths (WHAT space). The analysis in both spaces leads to results in two different directions: first, time durations of different visual tasks, and common visual search strategy patterns of participants; second, a model for knowledge processing of visual elements by the temporal order of focused semantic entities, their relations, and meanings. Based on these two results, ACT-R or other cognition simulation frameworks can be used to simulate cognitive and perceptual processes that lead to visual search strategies.
We propose to use results of eye tracking experiments in the WHERE and WHAT space to formulate cognitive models for simulating visual search strategies. These simulations will lead to a better understanding of visualization parameters leading to optimize cognitive ergonomics of visualizations.
Most visualization techniques as well as the parallel scan path visualization technique allow to analyze eye movements in the WHERE space; they graphically represent participants’ scan paths on the screen. We proposed to annotate graphical elements of a visualization with semantic information. Using this annotation the “WHAT” space of scan paths can be studied.
To study the WHAT space, a sufficient visualization technique of semantic attention, and declarative knowledge processing has to be developed.
We proposed to use results from eye tracking experiments in the WHAT and WHERE space to formulate both a cognitive model of visualization perception, and to simulate visual search strategies. How could such a model look like? What are the development strategies for such as model? Would it be based on a KLM approach with operators  or on a declarative knowledge processing simulation? Would it be possible to combine both approaches?
We firstly presented this concept during the workshop “Interaktion und Visualisierung im Daten-Web (IVDW 2011)” (Interaction with and Visualization of Data in the Semantic Web) at the Informatik 2011 conference .
ACT-R is available at http://act-r.psy.cmu.edu/ together with a large number of articles, conference papers, examples, and the programming documentation.
CogTool is an open source project released under LGPL. Executables, documentary, examples, and source code is available at: http://cogtool.hcii.cs.cmu.edu/.
We would like to thank our students for their contribution: Tanja Agapkin, Christoph Bergmann, Xuemei Chen, Alex Duscheck, Christian Dittrich, Tobias Eikmeier, Stephan Engelhardt, Dominik Herr, Paul Hummel, Michael Kircher, Andreas Neupert, Hannes Pfannkuch, Sven Plohmer, Nico Ploner, Edwin Püttman, Oliver Schmidtmer, and Stefan Strohmaier (in alphabetical order).
- 1.Tory, M., Staub-French, S.: Qualitative analysis of visualization: a building design field study. In: Proceedings of the 2008 Workshop on BEyond Time and Errors: Novel evaLuation Methods for Information Visualization, pp. 1–8 (2008)Google Scholar
- 2.Spakov, O., Miniotas, D.: Visualization of eye gaze data using heat maps. Electron. Electr. Eng./Elektronika ir Elektrotechnika 2(74), 55–58 (2007)Google Scholar
- 8.Duchowski, A.T., Driver, J., Jolaoso, S., Tan, W., Ramey, B.N., Robbins, A.: Scanpath comparison revisited. In: Proceedings of the 2010 Symposium on Eye-Tracking Research and Applications (ETRA), pp. 219–226 (2010)Google Scholar
- 9.Heminghous, J., Duchowski, A.T.: iComp: a tool for scanpath visualization and comparison. In: Proceedings of the 3rd Symposium on Applied Perception in Graphics and Visualization, pp. 152–152 (2006)Google Scholar
- 11.Burch, M., Andrienko, G.L., Andrienko, N.V., Höferlin, M., Raschke, M., Weiskopf, D.: Visual task solution strategies in tree diagrams. In: Proceedings of IEEE PacificVIS 2013, pp. 169–176 (2013)Google Scholar
- 13.Huang, W., Eades, P., Seok-Hee, H.: A graph reading behavior: geodesic-path tendency. In: Proceedings IEEE PacificVIS 2009, pp. 137–144 (2009)Google Scholar
- 17.Haber, R., McNabb, D.A.: Visualization idioms: a conceptual model for scientific visualization systems. In: Nielson, G.M., Shriver, B., Rosenblum, L. (eds.) Visualization in Scientific Computing, pp. 74–93. IEEE Computer Society Press, Los Alamitos (1990)Google Scholar
- 18.Raschke, M., Xuemei, C., Ertl, T.: Parallel scan-path visualization. In: Proceedings of the 2012 Symposium on Eye-Tracking Research and Applications (ETRA), pp. 165–168 (2012)Google Scholar
- 19.Raschke, M., Heim, P., Ertl, T.: Interaktive verständnisorientierte Optimierung von semantisch-annotierten Visualisierungen. INFORMATIK 2011: Informatik schafft Communities; 41. Jahrestagung der Gesellschaft für Informatik e.V. (GI) (2011)Google Scholar
- 20.Norman, D.: Some observations on mental models. In: Gentner, D., Stevens, A.L. (eds.) Mental Models, pp. 7–14. Lawrence Erlbaum Associates, Hillsdale (1983)Google Scholar
- 22.Janecek, P., Pu, P.: Searching with semantics: an interactive visualization technique for exploring an annotated image collection. In: Meersman, R. (ed.) OTM-WS 2003. LNCS, vol. 2889, pp. 185–96. Springer, Heidelberg (2003)Google Scholar
- 23.Kalogerakis, E., Moumoutzis, N. Christodoulakis, S.: Coupling ontologies with graphics content for knowledge driven visualization. In: Proceedings of the IEEE Virtual Reality Conference (IEEE VR’06), pp. 43–50 (2006)Google Scholar
- 25.http://www-st.inf.tu-dresden.de/smtvis/blog/. last Accessed 20 Aug 2013
- 26.Anderson, J.R., Lebiere, C.: The Atomic Components Thought. Lawrence Erlbaum Associates, Hillsdale (1998)Google Scholar
- 27.Salvucci, D.D.: A model of eye movements and visual attention. In: Proceedings of the Third International Conference on Cognitive Modeling, pp. 252–259 (2000)Google Scholar
- 28.John, B.E., Prevas, K., Salvucci, D.D., Koedinger, K.: Predictive human performance modeling made easy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 455–462 (2004)Google Scholar