1 Introduction

Informational and reference websites help readers accomplish their goals without the publishing organizations realizing any immediate financial benefits. While the cost of producing and supporting these sites is often easy to track, the value they provide can be much more difficult to measure. Schaupp et al. [1] recognized that measuring the success of such sites is difficult “because the definition of success changes depending on the perspective that the stakeholder adopts.” The stakeholders in an informational site include, at a minimum, the site’s readers and its sponsoring organization, who have different, yet ideally related, goals and definitions of success. As the definition of success varies with the stakeholder perspective, it becomes difficult to know what to measure, let alone how to measure it. The taxonomies we propose in this paper seek to clarify the goals of these stakeholders. As a result, strategies for measuring success towards meeting those goals can then be identified, and organizations can chose how to measure reader performance more accurately and obtain a better indication about their return on investment.

This paper reviews the theory and previous literature from which we propose two taxonomies that delineate reader and organization goals with informational websites. Within the framework of these taxonomies, we discuss how and when organizations should collect data about the readers’ experiences in order to track how well they relate to the readers’ and organizations’ goals, and help organizations author and design effective content to meet their readers’ goals.

2 Background and Literature Review

The lack of an expectation of generating revenue is the key factor that differentiates informational sites (e.g., government, medical, library, and reference sites) from commercial websites. This paper focuses on informational sites that do not have any immediate commercial goals. An example of such an informational site is the United States National Aeronautics and Space Administration (NASA), www.nasa.gov. An example of an informational/commercial site is the W3Schools site (www.w3schools.com) because the site contains advertisements, making its real motive commercial. While such commercial/informational sites might be able to apply some of the concepts in this paper, their hybrid goals are not the focus of this paper.

The following sections describe published work that helped us develop an improved taxonomy of stakeholder goals. We build on Rouet’s TRACE model [2] of document processing and then review usability properties and website measurement metrics in order to characterize readers’ experiences as readers interact with the content they encounter on the web. Finally, we describe reader and organization goals to explore appropriate measures for meeting differing stakeholder goals.

2.1 Document Research Task Models

Rouet [2] describes how people research information on the web in a 9-step Task-based Relevance Assessment and Content Extraction (TRACE) model that starts with readers formulating the information-seeking task and constructing a task model to use to accomplish the task. If external information is required, the TRACE model says that readers will iteratively evaluate documents to collect and process the information necessary to accomplish the task, until they feel that they need no additional information to complete the task. This model is consistent with Redish’s observation that readers will read until the think they have the answer [3].

2.2 Website Usability and Measurement Metrics

Website usability has been characterized in many ways and assessed with many metrics and tools. Watson [4] summarizes usability properties from ISO-9241 [5], Nielsen [6], and Quesenbery [7]. Nielsen and Loranger [8] reiterated the same usability properties that Nielsen [6] listed. Many have noted that an understanding of usability involves appreciating its purpose and use [9–11]. To assess the reader (user) experience, many have suggested the use of usability measures such as effectiveness, efficiency, satisfaction, memorability, engagement, ease of learning, error rates, and error tolerance [6–8].

Organizations use many metrics to assess website activity and effectiveness [1, 12–14]. Web servers provide activity logs; services such as Google Analytics provide rich data from instrumented websites [15]; and online and offline questionnaires can solicit feedback from web-site visitors [16]. However, because many of these metrics were originally designed to support the specific goals of revenue-producing sites, they can present challenges when they are used to monitor and assess the effectiveness of informational sites. If the tools’ goals align with the goals of an informational site, the tools can provide valuable information. However, when they misalign, the data provided can range from not useful to confusing. By characterizing the goals of informational sites more precisely than earlier taxonomies have afforded, measurement metrics and tools aligned with these goals can be identified and developed.

3 Goal-Oriented Taxonomies of Informational Sites

We propose two stakeholder taxonomies that identify goals for using and posting informational content on a website: one that identifies the readers’ goals for visiting informational websites and one that identifies the organizations’ goals for producing informational websites. Categorizing readers’ goals in this way should help organizations analyze where and how readers accomplish their goals and in turn identify suitable metrics to measure how well the site helps readers accomplish their goals.

3.1 Reader Goal Taxonomy

Redish [17] extended the reader’s goals tested by Sticht et al. [18] to describe three reading goals: reading to do, reading to learn, and reading to learn to do. These reading goals still apply to web readers today, but they do not help identify where the content fits within the reader’s task(s) to accomplish his or her goals. We therefore propose five goals that readers can have when they visit informational websites:

  • Reading to be reminded (Reading to do lite)

  • Reading to accomplish a task in a website (Reading to do here)

  • Reading to accomplish a task outside a website now (Reading to do now)

  • Reading to accomplish a task outside a website later (Reading to learn to do later)

  • Reading to learn (Reading to learn to use later or to apply with other information)

Reading to be Reminded. Reading to be reminded, or Reading to Do Lite, occurs when readers visit an informational site with the confidence that they know most of what they need to know about a topic and just need a refresher. Readers use the website as a form of offline information storage that they may use either online or elsewhere. Brandt et al. [19] noticed this pattern while observing software developers who “delegated their memory to the Web, spending tens of seconds to remind themselves of syntactic details of a concept they new [sic] well.” By knowing the information is available online, readers do not need to remember the details, just where they can find them. An example of such a website might be an online help website that reminds one how to get the ribbon to reappear in Microsoft Office.

Because readers will read “until they’ve met their need” [3], readers will spend as little time in the site as they need accessing the site shortly before or after they start a task. Once readers have been reminded of the information they need, they will finish their original task when convenient. Figure 1 illustrates an example of readers’ interactions with the content when reading to be reminded. In Fig. 1, the reader referred to the content once before starting the task and twice while performing the task to accomplish his or her goal.

Fig. 1.
figure 1

Task and content interaction with a reading to be reminded goal

Full size image

Topic design principles needed to serve this reader goal include making content easy to find, navigate, and read. Visible headings and short “bites” and “snacks” of information [3] are well suited to such a goal. Because such content access is typically very short and highly focused, the readers’ success in using a search engine is critical; however, it is often successful because readers already know the terms and context they are seeking and may have already bookmarked the page in their browser.

One success metric that can be measured in the background is the search term relevance to the topics sought by the reader. Asking about the reader’s satisfaction should be done soon after the interaction with a brief satisfaction questionnaire—perhaps just one question, such as “Did this topic help you?” Otherwise, the survey could impose on the readers’ overall task flow.

Those who assess a reader’s success with informational websites should be aware that traditional, commerce-oriented web metrics could allow for misinterpretation of the reader’s short interaction with the site. Google Analytics [20] describes Bounce Rate as “the percentage of sessions…in which the person left your site from the entrance page without interacting with the page.” In a commerce-oriented site, the goal is to attract customers, have them engage with the site, and then go through the purchase funnel. In that context, a high bounce rate is a bad outcome because it suggests that people are not interacting with the site and not finding what they are looking for. In a Reading to Be Reminded interaction, however, a high bounce rate could be a sign of success, with readers finding what they need quickly and leaving the website.

Short average time-on-page values [21] could also be misleading. If the page is well designed and needed information is found quickly, readers who come to the page for a reminder will leave quickly and spend little time reading the page. An understanding of the readers’ satisfaction is required in order to interpret whether a high bounce rate or a short average time-on-page is good or bad. Otherwise, these page visit values could easily be misinterpreted in the context of informational websites.

Reading to Accomplish a Task in the Website (Reading to Do Here ). Reading to accomplish a task in the website, or Reading to Do Here, is characterized by readers interacting with a page in a website to accomplish a specific task through the site. The readers’ goal is to complete the desired task, such as register for a library account, subscribe to an online newsletter, or renew a business license.

The readers’ interaction with the content begins shortly after they decide to accomplish the task and ends just after they complete it. Figure 2 illustrates such a task. Readers want to find the page to help them accomplish the task as quickly as possible and complete the task in the least number of steps and amount of time possible. They want to know if they have successfully completed the task before they leave the website. After they leave the website, they generally will not remember much about the experience unless it was especially negative or positive.

Fig. 2.
figure 2

Task and content interaction with a reading to do here goal

Full size image

Figure 2 shows a very common type of web interaction. Web usability guidelines describe the design implications that depend on the site, context, and audience in many texts [3, 22]. Because the task is performed almost entirely in the context of the web interaction, measuring the success of the interaction is easily accomplished without imposing on the reader. The web server can collect data concerning the time spent in the interaction; the rate of successful operations (e.g., registrations, applications, or whatever the interaction is designed to accomplish); and the path through the interaction (e.g., back tracks, sidetracks, and early exits). Requests for qualitative feedback should occur soon after the interaction so readers’ remember the interaction.

Reading to Accomplish a Task Outside the Website Now (Reading to Do Now ). Readers interacting with a website to accomplish a specific task outside the website now, or Reading to Do Now, seek to complete the desired task at the same time that the website provides the information necessary to achieve their goal. Examples of such websites include sites that describe how to repair a household appliance or how to cook a meal.

The readers’ interaction with the content begins after they decide to perform the task. Depending on the nature and duration of the task, the reader might return to the content several times during the task; however, the readers’ interaction with the content ends when they feel confident to complete the task without additional information—which may or may not coincide with task completion. Figure 3 shows the task and content interaction of a task in which the reader refers to the content for only the first half of the task and feels confident enough to continue without further reading the content.

Fig. 3.
figure 3

Task and content interaction with a reading to do now goal for accomplishing a task outside the website

Full size image

This interaction can influence several aspects of the design. For example, if readers would be likely to print the web content to save or take with them, it might be inconvenient to have the web content spread over several web pages. However, because readers might stop interacting with the content at any point, the content could be divided into individual pages of logical steps with natural breaks. Such a design could provide the sponsoring organization with information about the readers’ interactions without inconveniencing readers. Breaking a task into steps or subtasks could be modeled as a sequence of reading to do now tasks.

Because time can elapse between when readers stop interacting with the content and when they actually complete the task, asking for satisfaction or successful task completion might require creativity. For example, one could use social media to indicate the task was completed with a “post your finished project to Facebook” link, which could trigger a short satisfaction survey or a “rate this recipe” dialog.

Tracking progress, success, and satisfaction for this type of interaction requires coordination with the content design. The task and subtask flows must be modeled in the content’s design so that the instrumentation used to collect data about the interaction coordinates with the readers’ interaction. Because readers can leave the content without reading all of it and still complete their task successfully, metrics such as average time-on-page and path are less relevant to the readers’ experiences than satisfaction and success. It is impossible to know if it is bad or good that readers exit a procedure on the first step without knowing whether they are also dissatisfied or unsuccessful. Perhaps the instructions were confusing and they left with a negative experience. Alternatively, the instructions on the first page might have provided the information and the confidence the readers needed to proceed and be successful without further reading. In either case, without assessing the readers’ experience, most of the web metrics collected by the server and web analytics modules may provide only ambiguous data.

Reading to Accomplish a Task Outside the Website Later (Reading to Learn to Do Later ). When a reader is Reading to Accomplish a Task outside the Website (Reading to Learn to Do Later), the task and the content that provides the prerequisite learning to accomplish the task later are separated in time, to the point where they may become separate but related tasks. An example of a website that facilitates this type of reader’s goals—where the ultimate task is accomplished outside the website—is the United States Internal Revenue Service’s form page (www.irs.gov/Forms-&-Pubs). While the readers’ goal might be to file an annual tax return, they may need to obtain and download a form from the forms page, but they plan to fill out the tax return (a separate goal) later. Figure 4 illustrates an example of the relationship between the content interaction and the actual performance of the task studied.

Fig. 4.
figure 4

Task and content interaction with a reading to learn to do later goal for accomplishing a task outside the website

Full size image

The web usability goals for this reader goal include search-engine optimization and effectiveness, and term and vocabulary alignment. Depending on the nature of the content and the task, it might be necessary to assess the task and its performance as separate events. In this case, it would be necessary to include affordances to test the intermediate goal of learning the content before the task is attempted and consider using methods to collect and coordinate information about task completion. Learning, for example, could be measured through interactive quizzes in the content (such as through a game) but task completion would need to be measured in a way that is task appropriate. For example, a driver’s license test preparation site could quiz the reader on the content read to determine the readers’ learning and the content’s effectiveness, and even provide feedback if desired. The task of passing the written driver’s license test would occur later and be measured at the Department of Licensing. If possible, the two experiences could be related to evaluate the effectiveness of the test preparation and actual task completion. Asking the reader about satisfaction could also be done during and after the readers’ content interaction, as long as the questions did not overly disturb the learning task.

Reading to Learn (to use Later or to Apply with Other Information). Reading to Learn to Use Later, without a particular or immediate task in mind, is similar to what Sticht described as Reading to Learn [18]. The critical distinction is that the reading task in the web is a subtask of the reader’s ultimate goal of learning a new concept or skill. An example of a website that facilitates this type of reader goal, where the goals are accomplished after using the website or in conjunction with using other information, would be reading websites and books about design principles so as to use the information later when designing a website.

In a Reading to Learn to Use Later goal, as shown in Fig. 5, the reader reads information from many different locations, of which the content on the sponsoring organization’s website might be only one information source. Unlike Reading to Be Reminded, the interaction with the content with this goal is more involved because the information they are reading is new and the reader might consult multiple sources to accumulate the information required to reach the learning goal. It is difficult to measure how well readers are accomplishing their ultimate learning goal when their interaction with the website may be one step of many and they might not use the information until much later.

Fig. 5.
figure 5

Task and content interaction with a reading to learn to use later goal

Full size image

The design of the web page could be modified to encourage or require readers to interact with the page so as to collect information about the readers’ experience. For example, the content could include quizzes, and links or other affordances such as prompts to share the content with a social network.

3.2 Organization Goal Taxonomy

The previous section described five readers’ goals, but organizations’ goals for informational websites are also important and not always as clear or easy to measure. Halvorson and Rach [23] state that “content is more or less worthless unless it supports a key business objective and fulfills your users’ needs.”

To help organizations clarify their goals, we propose three goals for organizations that produce informational websites:

  • Brand awareness

  • Brand loyalty

  • Cost reduction

We drew these goals from marketing literature and kept their “marketing” names to make them easier to relate to existing literature and metrics. Therefore, they will need some adaptation by organizations that do not have specific revenue-producing goals.

Measuring the influence of web content on the organization’s goals requires a clear understanding of how the content contributes to the organizations’ goals. Ittner and Larcker [24] note that 77 % of the companies they surveyed did not link their metrics to business performance. For the metrics discussed here to be meaningful, they must be linked to the organization’s goals.

Brand Awareness. McGovern [25] says, “There is one word that describes all great Web brands: useful.” This would seem to apply to commercial and non-commercial websites alike. Keller [26] describes two approaches to measuring customer-based brand equity:

  1. 1.

    “The ‘indirect’ approach attempts to assess potential sources of customer-based brand equity by measuring brand knowledge (i.e., brand awareness and brand image).

  2. 2.

    “The ‘direct’ approach attempts to measure customer-based brand equity more directly by assessing the impact of brand knowledge on consumer response to different elements of the firm’s marketing program.”

The methods to accomplish such measurements are typically survey based and take place outside the web experience. It is impossible to assess such measures accurately by asking or observing only the part of the market that comes to your website.

Measuring market-based goals requires some adaptation for products and brands that are not commerce-oriented. For example, the “market” for a government-sponsored website could be defined as those citizens who live within or interact with the government’s jurisdiction. If there is no pre-defined market then, for an informational website, the market would consist of all the people who could benefit from the information. For the NASA website mentioned earlier, the market would include at least United States citizens; however, it is likely much larger than that. If, for example, a student in Germany wanted to research the NASA site, would that make him or her part of the market? If the market is defined as only U.S. citizens, then, the German student would not be included in the market. However, that does not mean he or she does not have something in common with the target market—in this case, an interest in NASA.

Tempting proxies for the market’s awareness of a site are often measured by comparatively easy-to-collect website traffic and interactions, such as by counting page-views, click-throughs, engagement, interaction with content, and social media posts. Such metrics are important elements of an awareness metric, but they need to be considered in the context of the market as a whole. It is possible that a change in such metrics could simply reflect market shifts and not relate to the website or its content.

Brand Loyalty. Once the market is aware of the site, organizations might want to keep track of how likely first-time visitors are to return to the site and whether market awareness is being sustained. Some common metrics used to measure awareness are the Net Promoter Score and the Willingness to Search [27].

Reichfield [28] presented the Net Promoter Score as a way to determine brand loyalty by asking customers, “How likely is it that you would recommend our company to a friend colleague?” Reichfield [28] suggests this strongly indicates loyalty “because when customers recommend you, they’re putting their reputations on the line.” While the Net Promoter Score is a temptingly simple metric, Sauro [29] cautions that if it is used, it should be used with other metrics to provide the information required to explain it.

The Willingness to Search metric measures “the likelihood that customers will settle for a second choice product if their first choice is not available” [27]. In his commentary on the Net Promoter Score, Sauro [29] suggests Freed’s word-of-mouth index [30] as a way to collect positive and negative aspects of the underlying sentiment. In any case, as Ittner and Larcker [24] observed, metrics should be carefully linked back to the financial goals of the business.

One non-market aspect of governmental websites is that there is usually only one per jurisdiction, which means there is often no alternative for the information provided by the site. However, a loyalty measure is still important, because visitors have the option to ignore an agency’s website and also tell their friends to ignore it if it is not useful. The word-of-mouth index can alert the stakeholders of an informational website hosted by a government agency to the negative sentiment in their target market—even when there is no other source for the information.

Cost Reduction. Online content is an increasingly popular self-service technology and a way that organizations can reduce customer service costs [31]. When implemented well, online content can provide readers with information needed to accomplish tasks that would otherwise be more costly (e.g., call centers, live customer-service agents).

Tracking the cost savings offered by self-service content cannot, however, be done by monitoring content alone. To track the savings, the cost and usage of the alternatives to the content (e.g., call centers, customer support interactions) must be tracked against a baseline measured before introducing the self-service content. An exception to this, however, is in the reading to do here case where the self-service operations can be compared to the operations they replace.

4 Conclusion

Informational websites often have many stakeholders—readers and sponsoring organizations—with potentially different goals. That readers and organizations have different goals does not mean they must have competing goals. Identifying the relationship between the readers’ tasks and the website’s content can help guide the organization in authoring and designing the website content to fit the interaction model and can help organizations select which metrics will provide the most useful data. The taxonomies presented in this paper can help simplify and focus the discussion about these relationships within an organization as they plan, design, and author web content for informational websites in order to attract and retain the desired readers.

These taxonomies provide the authors of informational content that they will publish on informational websites with a means to model how readers will use their websites to accomplish their goals. They also help the stakeholders within an organization identify the goals that can be measured and tracked. These taxonomies are intended to be used in the context of existing audience and task-analysis studies—not to provide a substitute for them. These taxonomies provide a framework within which these studies and analyses can be applied to simplify and focus the discussion of the data they produce.