Keywords

1 Introduction

Usability testing is performed by HCI experts to identify usability problems of interactive software systems. During a test, data like audio/video, PC logs, questionnaire results are gathered and analyzed to detect usability problems. This technique stands out positively from others (e.g. inspection techniques) since users are involved, critical tasks are executed, and a lot of qualitative and quantitative data can be collected and analyzed. It is considered expensive, especially due to the time needed to analyze data.

Some researchers propose the use of visualization techniques to help, in different ways, the evaluator discovering usability problems. For example, WebQuilt is a usability logging and visualization tool that presents logged data in an interactive graph, whose nodes are web pages visited by the users [1]. Each node details usability issues that occurred at that page. In [2], the user’s actions during a task are shown on a timeline, and the optimal actions to perform a task are also represented; this should help the evaluator to detect deviations of the performed actions from the correct ones, speeding up the time to investigate something that occurred at a certain time.

In our research, we are primarily working on usability evaluation of web sites and we aim at supporting the evaluator on easily identifying the web pages presenting usability issues. Thus, we propose two techniques that visualize a graph whose nodes are web pages. The presented graphs also show the optimal path to perform a task, since the visual comparison with the optimal path provides useful indications of something that the user could not correctly perform (this was inspired by [2]). In the following, we describe the proposed interactive visualizations, and briefly report on an evaluation study that compared the two techniques.

2 Two Visualization Techniques to Identify Usability Problems

In usability tests, the identification of sources of usability problems often needs the triangulation of different quantitative and qualitative methods. Quantitative data, like average task time and task success rate, provide indications on tasks that caused evident problems but not where the problem occurred. To get this information, evaluators should look at the videos, paying attention to which pages led participants to fail the tasks or too much longer routes to complete the task.

To support evaluators we propose two interactive visualization techniques that depict, for each task, the routes followed by participants, as well as the ideal path that participants would follow to successfully complete a task. The problematic web pages are the ones that mostly confused the participants, leading them to follow wrong paths.

The first visualization is based on the Arc Diagram [3], which places vertices along a horizontal line and arcs represent connections between nodes. Figure 1a, shows the adaptation of Arc Diagram to support the identification of web pages that caused problems during the execution of a task by all participants to the test. Nodes represent web pages visited by participants, their size is directly proportional to the number of visits. Blue node refers to the task starting page, red if participants wrongly terminated the task at that page, green node refers to the task ending page, black for the other pages. Arc thickness is proportional to the frequency of movement between connected pages. Green arc is the ideal path, otherwise it is light grey. Violet arcs represented below the node line indicate backward paths. The graph in Fig. 1a shows that 13 users correctly moved from the task starting page to the next one (the green arc with ‘13’ label); then only 9 users correctly moved to the next page, and eventually all 15 users moved toward the page that completed the task. Only 1 user wrongly terminated the task, as shown by the red node. One of the most problematic web pages is the one where the arc with 13 users ends, since only 9 users were able to correctly move from this page to the next correct page.

Fig. 1.
figure 1

Visualizations of participants paths inspired by the Arc Diagram (a) and Page Tree (b) (Color figure online)

Full size image

The second visualization is called Page Tree because it shows a tree-like structure that better recalls the ‘hierarchical’ organization of a Website (see Fig. 1b). Nodes and arcs (branches) have the same meaning as other visualization, both in term of colors and dimensions/thickness. In this visualization, the backward paths are not explicitly shown. Here the 47% of participants correctly moved from the task starting page toward the final task page (percentage or absolute values can be switched), while the other participants followed wrong paths (in grey), sometimes wrongly terminating the task (to the red nodes) and sometimes correctly finishing the task (the other green nodes). Lack of space prevents us from detailing the motivation of the choice of these visualizations and reporting all their details and functionalities.

3 Evaluation of the Proposed Visualizations

The design and development of the two visualizations followed the User-Centered Design methodology. Initial ideas were sketched and discussed among designers and evaluation experts. Prototypes of increasing complexity (starting from paper-based ones) were tested with few evaluators using the thinking aloud protocol. They allowed to detect various problems that were fixed in the next prototype. Figure 1a and b refer to the current interactive prototypes, implemented using D3 JavaScript library. These prototypes were used in an experimental study to understand to what extent the visualizations support HCI experts in identifying problematic Webpages. We recruited 15 participants (4 females, 11 males) among the students of the Computer Science and Digital Communication curriculum at our university, who completed a course on Human-Computer Interaction and had some basic experience about performing a usability test. Their average age was 26 years (min = 23, max = 29). The study was organized according to a within-subject design, with visualization as an independent variable and as within-subject factors the two visualizations. Each visualization showed data gathered in a study conducted with 15 employees of an Italian Administration, who used a Web platform for onsite and remote usability tests [4, 5]. With each system, the participants performed the following tasks: 1. Identify the page that caused most problems; 2. Identify the path(s) that led to the task’s failure; 3. Identify the path that led to the task success following the ideal path, 4. Identify the paths that led to the task success following alternative paths; 5. Identify backward paths, if possible.

Visualizations and order of tasks were balanced according to a Latin-square design. At the end of all tasks with each visualization, participants filled the UES-short form questionnaire.

4 Results and Conclusion

Visualizations have been analyzed and compared according to their effectiveness, efficiency and satisfaction in discovery usability problems. The main findings are mentioned here.

Efficiency was evaluated considering the time that participants needed to complete the assigned tasks. Participants were slower when using Arc Diagram (x = 34.93, SD = 5.77) than Page Tree (X = 27.76, SD = 5.13). Paired sample t-test showed that this difference is statistically significant (t(14) = −5.287, p = .000).

Effectiveness was analyzed calculating the task success rate. Page Tree had a lower rate (x = 86,56%) than Arc Diagram (X = 93,00%). Even in this case, paired sample t-test revealed a significant difference (t(14) = −2.512, p = .025).

Satisfaction was analyzed calculating the UES score of the two visualizations (Arc Diagram = 4.61/5; Page Tree = 4.78/5). However, paired sample t-test did not reveal any significant difference (t(14) = −1.878, p = .081).

In conclusion, this on-going work proposes two interactive visualization techniques to support usability evaluators in identifying web pages that cause usability problems, speeding up data analysis and encouraging the adoption of usability test in software development lifecycle. Results have shown that both visualizations are adequate to identify web pages that cause usability errors. Page Tree resulted better in term of time to find errors and success rate, while no differences emerged in user engagement. Studies are in progress to investigate other visualization techniques to focus on other aspects of the user flow during task execution or to visualize user activities [6].