Keywords

1 Importance of Forms

Business forms and disclosures have become central to customer relationship management. Forms are used to solicit information from the customer in a standardized manner, and disclosures are used to communicate rights, facts, risks, and other important information to the customer. Anyone who has ever completed a medical history form at a hospital or accepted the terms and conditions of a new software download has interacted with a business form or disclosure and, more than likely, the unnecessary jargon, redundancy, ambiguity, and obscurity associated with many of these documents. It should therefore come as no surprise to learn that most consumers are unable or unwilling to read these documents [1].

2 Importance of Usability Testing for Forms

Although some may speculate that businesses deliberately make forms opaque, there are just as many benign causes related to the difficulty in organizing and presenting large amounts of information to a wide variety of audiences. In fact, this situation led the U.S. Government to pass the Plain Writing Act of 2010 [10], which requires federal executive agencies to use plain writing in all documents agencies issue to enhance citizen access of Government information and services.Footnote 1 The latter part of this requirement is essential because it highlights the need to collect and assess information from users on their experience with forms and their language.

Typically, when we consider the application of usability testing, it is in regard to complex technologies such as websites, software, or applications, but usability can apply to any context where specific users are interacting with a product or information in order to reach a specific goal [5]. In this respect, forms and disclosures are no different from applications or websites. In general, the goals of forms and disclosures are to extract accurate information from users and convey important, necessary information. In order to meet these goals, these documents must comply with the same usability principles that are used in more complex applications. Peter Moorville [8], in particular, expanded upon the concept of usability and illustrated the facets of user experience. Table 1 shows each facet, its general application, and how it applies to form usability.

Table 1. Facets of the user experience with products
Full size table

3 Role of Eye Tracking in Usability Tests of Forms

Because the user experience is multi-faceted and complex, the use of multiple metrics is often advocated for when conducting a usability test [14]. The two most common group of metrics are self-report (e.g., satisfaction questionnaire ratings, verbal comments) and performance (e.g., time on task, task accuracy). The third, and less often discussed, group of metrics is implicit (e.g., eye tracking, pupil dilation, electrodermal activity). Some aspects of form usability and findability are directly related to users’ attentional capture and engagement, making eye tracking an attractive tool to capture these behaviors. These same data gathered through other means may be biased or unreliable, given that users are not fully conscious of their attentional focus. Eye tracking has been used extensively in many other usability contexts, but has only recently been adopted for use with physical forms and notices [7]. Eye-tracking data informs us about the allocation of attention on design elements as well as the language in the form. It can be used to provide an additional level of insight—over and above self-report and performance metrics—into the optimal design and language.

3.1 Web Forms

Forms can take on two mediums: electronic and physical. Both mediums are used for a multitude of purposes. Electronic forms are used for site registration, email and service subscriptions, customer feedback, checkout, and data input to search or share information [6]. Electronic forms are critical to e-product success because poor design would likely result in lost data, lost conversions, and uninterested users. Much of the usability testing literature has focused on optimizing the design of web forms and, in a few examples, eye tracking has also been incorporated [7].

One such study tested the effectiveness of twenty web-form design heuristics [2, 13]. The researchers selected forms from actual commercial websites and applied their twenty heuristics to improve upon the design. The primary aim of the study was to examine the differences in usability between the original and redesigned versions of the forms. Eye tracking was included to provide further insight on users’ processing and comprehension. The original version of one of the forms displayed the labels on the side of the open fields, while the redesigned version contained labels directly above the text fields. Usage of the original version of the form with side-by-side labels and text fields resulted in more fixations, longer total fixation duration, and longer total saccade duration than in the redesigned version. These results suggest that completing the redesigned form was more efficiently done and less cognitively demanding. Furthermore, not only did performance-based outcomes improve, but researchers also saw improvement in self-report-based outcomes such as satisfaction questionnaire ratings and verbal responses during follow-up interviews.

In a separate study, Redline and Lankford [11] tracked participants’ eyes while they completed a complex questionnaire with extensive branching between questions. The researchers found that many completion errors could be attributed to participants failing to read branching instructions, especially when instructions were presented long before or after participants responded to the origin question. In addition, they found that participants did not read linearly and skipped around looking at the survey. In this case, eye tracking provided insight into the sequential processing undertaken by participants in addition to their overall performance.

3.2 Physical Forms

Though there is some literature related to the usability of physical forms [6] —mostly voting ballots—the vast majority of work has concerned their electronic counterparts. Furthermore, this disparity is even more apparent regarding the application and value derived from eye tracking in usability tests of physical forms and notices [7]. Nonetheless, due to the ubiquity and importance of physical forms, it is essential to update the methods and tools that improve their efficiency and effectiveness.

Eye tracking of users while they interact with a physical form provides researchers access to important cues about where users are in the process that would be otherwise unavailable. For example, when users complete a web form, they make frequent mouse movements and clicks that cue the researcher about progress they made on the task. However, when users complete a paper form, there are large gaps and absences of their observable behavior. During this time, the researcher is unaware of progress made on completing the form, making it difficult to ascertain the level of difficulty that is being experienced. These gaps in user behavior are precisely where eye tracking provides critical insight and value. During this time interval, the researcher can observe whether certain areas of the form are fixated on as well as the frequency and duration of those fixations.

Challenges with Eye Tracking Physical Forms. Before introducing eye tracking into the usability test of a form, researchers must determine whether the goals of the study merit its use, because many research questions can be addressed more efficiently and directly through alternative measures [3]. Further, using eye tracking for exploratory purposes can be a rather long and arduous fishing expedition given the sheer quantity of data collected by most contemporary tracking equipment. However, once specific hypotheses and outcomes related to users’ visual behavior have been established, the practical and methodological concerns can be minimized.

The demands of extracting eye-tracking data from users interacting with paper forms are rather different from those of electronic forms. For example, paper forms are manually manipulated and often require interaction with other environmental elements, making their eye tracking more complicated than that of well-contained electronic forms. However, in order to maintain ecological validity, it is important to try to reflect the natural context in which physical forms are completed during the usability test. Because there is no human-computer interface for the eye tracker to communicate with, an external scene camera must be used to map the eye movements to the environment.

There are two tools to choose from when eye tracking with paper: one is a fixed-position setup and the other is a head-mounted setup. With the fixed-position setup, the form is affixed to a mounted stand and the eye tracker is mounted below it. The stand keeps the scene camera and form in the same positions for each participant in the study. Because the form and eye tracker remain in the same position, the frame of reference never changes. As a result, eye-tracking data points are mapped to the same coordinates. This greatly simplifies data aggregation and analysis after the study is completed. However, the fixed-position configuration comes with a significant trade-off in external generalizability. The stand prevents participants from being able to hold the form, forcing them into an unnatural and contrived experience and restricting the available area to lay out multiple pages.

Eye-tracking glasses provide an alternative head-mounted option to paper form eye tracking. The glasses integrate a scene camera in the eyewear that continuously records the field of view. The eye-tracking technology is mounted into the glasses itself. The trade-off with the head-mounted option is extensive time during data aggregation. Because the scene camera is not in a fixed position, the field of view is continuously changing. As a result, the coordinates for the data points are constantly changing while the participant interacts with the form. This requires researchers to map individual fixations to a still image of the stimulus. Although this option requires more labor, it permits users to hold the form, creating a more natural and realistic experience.

4 Case Studies of the Application of Eye Tracking in Our Paper Form Usability Test Work

4.1 Usability Test of a Paper Diary

Introduction. The Nielsen television diary is a paper-based booklet that allows participants in Nielsen’s panel to record their television viewing habits. Members of the Nielsen panel receive these diaries in the mail and are asked to record their television watching behavior during a specified interval. Despite the increased availability of other modes of entering television behavior (i.e., Nielsen desktop websites and mobile applications), the paper diary is still a heavily relied upon source for data collection. Although desktop websites and mobile applications are more prevalent among higher-income households, mailed paper diaries are necessary for a representative sample, because lower-income households are less likely to have access to electronic resources. Nielsen, at the time, was in the process of redesigning their paper diary form. Full details of this study have been presented by our colleagues [15, 16].

Stimulus. The diary was printed on double-sided saddle-stitched 8½″ × 11″ sheets of paper. The cover displayed the title of the diary and the Nielsen name. Inside the diary, Step 1 asked participants to answer questions about the number of TV sets and people in their household. Step 2 asked participants to list the local channels they received as well as the method (i.e., cable, satellite, antenna) and list their local television stations. Step 3 asked participants to answer questions about who watches TV in their household by entering this information in specified slots. Step 4 asked participants to record their daily TV viewing. Step 5 instructed participants on how to seal and submit the diary.

Method. Seventy-four people (29 male, 45 female), with an average age of thirty seven, and diverse demographics participated in the study. Participants were randomly assigned to use one of three versions of the paper diary: an “Old” diary, a “New” diary, and a “Prototype” diary. The backing of the diary was mounted to a stand above a Tobii X2-60 eye tracker. In additional to the eye tracking, we collected conventional performance and self-report metrics. Participants were asked to complete Steps 1 through 5 of the diary. After completing the task, participants were asked to complete a satisfaction questionnaire. The moderator then conducted a debriefing interview with each participant.

Example Cover Findings. The original version of the paper diary had the words “Nielsen Television Viewing Diary” in the center of the front cover. The “New” and “Prototype” versions of the cover page contained motivational phrases in the center, such as “Your viewing matters…Tell us what YOU watch!” as well as faces of people from diverse backgrounds. For the redesigned versions, the Nielsen name and logo was moved to the bottom left corner of the cover. Participants fixated more on the center of the cover than on the other areas of the page for all three designs. This resulted in an issue with the two redesigned versions of the cover page because participants did not fixate on the name and logo. Self-reported data supported the eye tracking: Participants who received the original version of the diary made comments such as, “It [Nielsen logo] is the first thing I looked at.” Participants who received the redesigned versions made comments such as: “I noticed [the Nielsen logo] only when you pointed it out. It is not clear that it had any important information” (Fig. 1).

Fig. 1.
figure 1

Left to right: “Old” and “New” diaries. Mean fixation count heat maps show more fixations in the center of the page for both designs.

Full size image

This finding highlights the importance of designing for desirability and credibility of a form, as well as how eye tracking helped inform this finding. Without noticing the name and logo of the sender, respondents are unlikely to find mail material trustworthy, and they will often quickly discard it. And without a desirable cover, respondents are unlikely to open the diary to get started with the process. Here, eye tracking was able to inform designers about the elements on the cover that were most likely to be missed. As a result, we recommended moving the logo and name to the center of the cover, where it would be more visible.

Example Step 3 Findings. Step 3 asked participants to answer questions about who watches TV in their household by entering this information in specified slots at the top of the page. The “Old” diary and “Prototype” diary contained an example directly below the fillable slots, while the “New” diary contained an example beneath the fold. Unlike Steps 1 and 2, which progressed linearly from top to bottom and left to right, Step 3 forced participants to start at the bottom of the page and move to the top in order to progress in the correct sequence. The “Old” diary and “Prototype” diary, performed poorly in regard to noticeability. Eye tracking showed that 45 % of participants did not fixate on the example until after they had completed the fillable areas—a finding that exemplifies the importance of the “findable” facet of user experience. We recommended that the example be placed before the actual content, where users will be more likely to notice it before they are asked to input information.

Example Step 4 Findings. Eye tracking also provided relevant insights into how participants processed the fillable fields in Step 4. The “Old” diary and “New” diary had a column order that listed the station and channel number earlier in the sequence. The subsequent columns requested that participants enter the name of the program and the people in the household who watched it. Gaze plots indicated that participants were confused about this order, because they did not look in an orderly left-to-right pattern as one would expect. Gaze plots indicated that participants did not look in an orderly left-to-right pattern— a result that demonstrated the inefficiency with processing information in this layout. The “Prototype” diary had a different column order which had participants enter the name of the show before the other information. Gaze plots revealed a more F-shaped pattern [8] of participants looking down the page and then looking in a linear left-to-right pattern when entering information, — a result that demonstrated a marked improvement in processing efficiency. We recommended the ordering of the “Prototype” diary, with the television show entry field occurring earlier in the progression (Fig. 2).

Fig. 2.
figure 2

Three fixation gaze plots that exemplify eye movement-patterns when using the three different diaries. The “Prototype” diary grid (right) resulted in more linear gaze patterns than the “Old” (left) and “New” diary grids (middle).

Full size image

Practical Considerations. Our first challenge with conducting the study was selecting the placement of the eye tracker. At first, we tested the usability test setup with an older and larger eye tracker (Tobii X120) and mounted it above the form. We encountered issues with this setup when we found that participants’ eyelids were more likely to block the eye tracker from collecting data. Before data collection began, a newer and more compact eye tracker (Tobii X2-60) was launched. The size of this eye tracker allowed us to place it below the form. This placement resulted in a considerably higher rate of capture.

Other challenges with implementing eye tracking into this study related to the interactive nature of the form. Responding to the form resulted in head movement; consequently, during this interaction, the eye tracker was less likely to collect eye movement data. As a result, we focused on providing results from data collected while the participant was not moving his or her hand to respond. Participants also tended to block the eye tracker while they wrote on the form. To overcome this, we delineated an area below the form so participants had a cue as to where not to place their hands.

4.2 Usability Test of a Multi-page Government Form

Introduction. This multi-page form is completed to report discrepancies in personal finances to the Federal Government. The purpose of this study was to determine the issues associated with completing the form and provide recommendations to correct those issues. Full details of this study have been presented by our colleagues [12].

Stimulus. Before the form began, there were three pages of instructions with text and tables. The first part of the form was a series of “yes” and “no” questions to help respondents self-assess whether they should continue with completing the form. The second part of the form asked respondents to enter their personal information and the information for a family member. The third part of the form was a grid format that asked participants to account for different amounts of items associated with their income and expenses. The fourth and last part of the form asked for the respondent’s signature.

Method. Nine people (4 male and 5 Female), with an average of forty two, and diverse backgrounds from the Washington, DC, area participated in this study. Eight of the nine participants reported that they typically complete this form themselves by hand or with computer software; one participant reported to complete the form with the assistance of a professional. The form was mounted to a flat vertical surface for viewing and writing above a Tobii X2-60 eye tracker. In additional to the eye tracking, we collected conventional performance and self-report measures.

Before starting the task, participants completed a questionnaire that asked about their past experiences with this type of form. Participants were provided with scenario information so they did not enter their own personally identifiable information. Participants were also provided with supporting documents that helped them complete the form.

Example Findings. Eye-tracking gaze plots demonstrated that participants read most information on the first page of instructions, skimmed through the second and third pages of instructions, and then began working on the form. An analysis of the quantified fixation data demonstrated that, although most of the first page instructions were fixated, certain areas of the page, particularly the information that came later on the first page, tended to be skipped.

Most of the sections on the second page of instructions were not fixated. However, the center of the second page, which was a numbered section, had higher counts of fixations. This is consistent with research that has shown that users tend to read numbers and bulleted, bolded items [4, 9]. Participants had very minimal fixations on the third page of instructions.

Participants’ self-reported comments during the debriefing interview supported the eye-tracking and performance data about the use of instructions. One participant commented on the length of the instructions by saying, “I feel like the instructions were so long and perilous that I could really only retain about 10 %.” On the satisfaction questionnaire, participants, on average, responded that they read “some of the instructions.” In debriefing, we asked participants about the way they usually interact with tax form instructions. Responses indicated that, consistent with the session observations and eye-tracking data, participants tended to skim and skip instructions. For example, one participant said: “[I used them] the way I usually do. I read the first page and then I skip the rest. It’s typical of how I do it. I read about 30 %, and I know that the information is there and that I can go back. But I don’t ever finish the whole instruction booklet.” Another participant said: “I usually skim through it. Usually [these types of] forms are laid out the same way.”

The evidence suggested that the instructions are not being used as intended and highlights the importance of the “usable” facet of user experience. The instructions were presented in narrative format, which suggested that respondents should read through the full instructions before they start the form. However, our research suggested that most respondents will visually scan the information on these pages before they start. The later the information appears in the instructions, the less likely it will be read. Because the instructions are lengthy, they are more likely to be used as a reference when working on the form, and less likely to be read before getting started.

We recommended moving instructions to sections of the form where the specific information. For example, we recommended placing a condensed version of the instructions for completing the line items next to their respective fillable lines in the form. We also recommended that the remaining information in the instructions should be reformatted to facilitate an efficient scan pattern, such as reformatting text by chunking different pieces of information together into bulleted lists with bolded items.

Practical Considerations. We faced similar challenges to those in our first case study. In this study, we also used the more compact eye tracker (Tobii X2-60) and placed it below the form. In the first case study, the form was mounted at a 45° angle. In this study, we mounted the form to a vertical flat viewing service at a 90° angle. This resulted in a trade-off: we collected a higher rate of eye movement data samples, but it made the form more difficult to write on. In this study, the emphasis was on the use of the instructions, so we proceeded with the vertical setup. As in the first case study, we focused on providing results from data collected while the participant was not moving his or her hand to respond.

5 Conclusion

Despite playing an important role in organizations and customer relationships, forms and disclosures are often poorly organized and difficult to complete. Principles of usability testing commonly employed in more interactive and complex applications provide significant value to improving the design and organization of forms. In particular, the use of eye tracking provides an additional level of insight into users’ attentional allocation and progress through a form or disclosure. In this paper, we have summarized and synthesized relevant literature related to electronic forms and presented two case studies demonstrating the efficacy of usability testing paper forms and emphasized the value derived from including eye tracking in these tests. In addition, we have provided practical guidance for user experience researchers considering using eye tracking for paper forms. In summary, we have shown that forms regardless of medium can benefit from the same usability methods and measures implemented in more interactive environments. In the future, we hope businesses and organizations recognize the value of usability testing and the additional insight eye tracking delivers in creating useful, usable, accessible, credible, findable, and desirable forms.