Background

Clinicians require up-to-date and reliable resources to keep up their knowledge. Clinical practice guidelines (CPGs) are an important knowledge translation tool that can address this need, but uptake of CPGs can be inconsistent [1, 2]. Previous research has identified that aside from content itself, effective communication is also required to improve guideline implementation [2, 3]. Format of CPGs, for example, is identified as one of 6 key domains of implementability by Kastner and colleagues [3]. However, less than half of available guideline assessment tools address evaluating the format of guidelines [4]. Notably, even in tools that do evaluate format such as GLIA and CAN-IMPLEMENT, the focus is on the formatting of content within a document e.g. using bold text for recommendations or including visual algorithms [5, 6]. These recommendations do not address the full scope of formatting considerations, since many CPGs are more versatile now than a single document.

Increasingly, guideline organizations are creating multiple versions of their CPGs, which has been thought to improve guideline dissemination [7]. Short summary versions of guidelines and interactive tools based on guideline content may be utilized more easily at the bedside than the full text version. The website modality is particularly suited for keeping these different versions of content in one accessible location. As the internet is already starting to replace traditional sources of information like books and colleagues, CPG websites will only become more common [8, 9]. However, to our knowledge, none of the current guideline assessment tools make recommendations specific to CPG website creation.

We performed a mixed methods usability study of the Diabetes Canada (DC) CPG website, a comprehensive CPG website with a wide reach to both clinicians and the general public [10,11,12,13,14]. Usability testing has been widely used to assess and improve health technologies including websites [12]. Our aim was to assess the effectiveness of the content and format of DC Guidelines website, using our results to inform recommendations for this and other CPG websites.

Methods

We used mixed methods consisting of (a) questionnaire, (b) usability testing, and (c) semi-structured interviews to assess the live version of the Diabetes Canada Clinical Practice Guidelines website, updated April 10, 2018 [10].

Website

The Diabetes Canada CPG website has a wide reach with health care providers (HCPs) as well as the general population [11]. The content of the website can be divided into 4 categories. Its contents and main components are described in Tables 1 and 2.

Table 1 Content of the DC Guidelines Website
Full size table
Table 2 DC Guideline website components and descriptions
Full size table

Participant and setting

We recruited clinicians from four professions frequently involved in diabetes care: family physicians, nurses, pharmacists, and dieticians. Investigators performed convenience sampling by emailing their own personal and professional network of health care providers from across Canada. From there, snowball sampling was used to elicit further contacts [15]. In order to ensure an adequate number of dieticians were represented, the publicly available Dieticians of Canada registry was utilized to cold email potential participants [16]. All participants were assigned to a randomly generated numeric code for data collection.

Outcomes

We assessed the following outcomes for the usability tasks:

Task success rate

Successful completion was defined based on whether the response given by the user matched an acceptable answer from the investigators’ answer key.

Task completion time

The time from the initiation of first movement on the screen after instructions have been given until either a) a complete answer is given or b) the user asks to move on to the next task. If the user made further attempts for exploration purposes after completing the task initially, the extra time was not counted.

Clicks per task

The total number of clicks users made until the task was completed or aborted.

Resource used to complete task

The type of resource on the CPG website (full guidelines, a static tool, or interactive tool) participants used to complete a task successfully or unsuccessfully. If an attempt was aborted, the last page the participant landed on was considered to be the resource accessed.

Path

The links and pages accessed by the user, in the order in which they accessed them, from the beginning of a task until the task was completed or aborted.

Start of path

The section of the website (quick buttons on the home page, the links of the navigation menu, or the search function) that lead the users to the resource used to complete task. As users could explore multiple resources before arriving at the final answer, we defined the start of the path as being the most proximal to the final answer. For example, if a user followed the path [home page button ➔ page A ➔ back ➔ navigation menu link ➔ page B], the start of the path for the task would be navigation menu, as it is most proximal to page B, the resource used to complete the task.

Search attempts and success rate

The number of times users tried to use the website’s built-in search function and whether they succeeded in arriving at the correct resource.

Error type

Scheme was developed in data analysis (described below) to categorize unsuccessful task attempts into distinct error types.

Data collection

Demographic and practice characteristics questionnaire

Participants completed a sixteen-item electronic questionnaire prior to usability testing sessions via email. This included questions about patient’s age, practice type and setting, and prior familiarity with the Diabetes Canada CPG website (Additional File 1).

Usability testing

Usability testing was conducted remotely over the internet using proprietary video conferencing software (GoToMeeting 2019 version) [17]. The software allowed participants to share their computer screen and communicate by audio with the investigator (WW). Sessions were approximately 1 hour in length. The sessions were recorded in video format with corresponding audio.

Each participant completed eight usability testing tasks in random order. Instructions for the tasks was provided in writing using the text chat feature of the conferencing software.

Participants vocalized their thought process aloud using the “Think Aloud” methodology [18, 19].

Audio was transcribed to text in non-strict verbatim format with filled pauses (i.e. “um”, repeated words, and stutters) removed. Video was used to transcribe the pathways participants took from the start to completion of each task into text format.

Tasks

The eight usability tasks (Table 3 and Additional File 2) were developed to reflect commonly encountered clinical situations in the diagnosis and treatment of diabetes. Some of these scenarios were used in the usability testing of the previous iteration of the Diabetes Canada CPG website. We included scenarios that could be answered using five of the six most accessed tools on the previous iteration of the Diabetes Canada website, in order to ensure the most popular tools were being assessed [11]. Five other clinicians uninvolved in the study were asked for input to further refine the tasks.

Table 3 Usability testing tasks. T2DM = type 2 diabetes mellitus
Full size table

The usability tasks were divided into two types: information retrieval and resource retrieval. A previous study of the 2013 version of the website found that users likely entered the website to use a specific resource or to answer a specific clinical question [11].

The five information retrieval tasks asked participants to find the answer to a clinical question in a hypothetical clinical scenario (Table 3). For example, a task would describe a patient with diabetes and ask the clinician how often they should monitor their blood glucose. Resource retrieval tasks asked participants to locate a tool on the website, such as a patient handout. An answer key was developed with predetermined acceptable correct answers. All tasks could be completed using at least two resources on the website (e.g. full guideline chapter and quick reference guide).

Semi-structured interviews

Immediately after completing usability tasks, participants were interviewed to answer thirteen open-ended questions pertaining to the strengths and weaknesses of the website. The questions specifically asked for feedback on the website content as well as the format (see Additional File 3 for interview guide).

Data analysis

Demographic and practice characteristics questionnaire

Results were entered into a spreadsheet and descriptive statistics performed.

Usability testing

Transcripts and path texts were analyzed to extract the outcomes defined above for the usability tasks. Descriptive statistics was performed.

One investigator (WW) reviewed the recordings and think-aloud transcripts of all unsuccessful attempts and developed a preliminary coding scheme for type of error made (Table 4). This was then applied by a second investigator (DC) independently to recode the errors. All disagreements were reviewed and settled by discussion.

Table 4 Type of errors
Full size table

Semi-structured interviews

We performed qualitative content analysis of the transcripts of the semi-structured interviews using grounded theory, similar to previous studies in usability [20]. Two investigators (WW, DC) independently performed open coding of the transcripts to derive the initial themes, sorted under the categories of positive comments, negative comments, and suggestions for improvement [15]. We then performed axial coding of these initial themes to identify strengths and weaknesses of the website in terms of content and format.

Results

User characteristics

Fourteen clinicians participated in the study (Table 5). Most of the participants were young (age 20–39) and female. All participants had heard of the 2018 DC CPG website and 12 of 14 (85.7%) had used the website previously.

Table 5 User characteristics
Full size table

Success rate, completion time, clicks taken

Overall, users had a 79% success rate across all tasks and used 144 (± 152) seconds and 4.6 (± 3.9) clicks per task (Table 6). Resource retrieval tasks had a higher success rate than information retrieval tasks.

Table 6 Success rate, time taken, and clicks for usability tasks
Full size table

Task 7 had the lowest completion rate of 43%. Users also spent the longest time and most clicks on this task. Task 1 had the second lowest completion rate of 57% but was the shortest task at 86 (± 41) seconds.

Resource used and path taken

Interactive tools were more frequently used to complete tasks successfully than the full guidelines and static tools (Table 7). In unsuccessful attempts, the three types of resources were used about equally. The Quick Reference Guide was the most frequently used among static tools.

Table 7 Resource accessed when completing information retrieval tasks
Full size table

The homepage quick buttons leading directly to interactive tools were almost as frequently the start of final paths as links on the navigation bar (Table 8).

Table 8 Start of path when completing information retrieval tasks (n = 70)
Full size table

Search function

Although the search function was not used to complete information retrieval tasks (Table 8), some users did use search unsuccessfully before changing their strategy, while others used it to complete resource retrieval tasks. In total, the website’s built-in search was accessed 13 times and successfully found an answer or resource in 2 of the 13 attempts (15.4%).

Errors

Interpretation error accounted for 48% of total errors (Table 9). Task 1 alone accounted for over half of these, with all six unsuccessful attempts being interpretation errors. They were made by three of four users that accessed the quick reference guide, containing an algorithm, and three of nine users that accessed the screening interactive tool. Users gave up most frequently (3 of 8 instances) in task 7 on pregnancy and diabetes.

Table 9 Types of errors committed by task
Full size table

(Task 2 and Task 8 had perfect completion and are not shown on this table.)

Content-related themes

The top strengths and weaknesses pertaining to website content are shown in Table 10.

Table 10 Top strengths and weaknesses of website content identified through content analysis and their representative quotes
Full size table

Users showed a preference for interactive tools, static tools, and quick reference guides, while the full guidelines text was perceived as less useful. Users liked that the interactive tools were “quick” in generating answers.

Several themes addressed the presentation of information within the content. Users wanted to see more visual representations of content, such as tables and flowcharts. Some users pointed out there was too much medical jargon on the website, which could make it less useful for patients.

Users had mixed opinions about the breadth of content. Several users praised the website for being comprehensive, but similar number of users thought content was sparse in certain clinical areas.

Format of the website

The top strengths and weaknesses pertaining to website format are shown in Table 11.

Table 11 Top strengths and weaknesses of website format identified through content analysis and their representative quotes
Full size table

Navigation elements were the most highly rated aspects of the website design. Most users found the left-hand navigation bar useful. Almost half of users found the links to the tools on the main page helpful. Being able to access the same information in multiple ways was also seen as a positive.

Many users still found it difficult to find information. Links that were formatted as lists, rather than as buttons, were challenging to use for most users. About a third of users thought the search function did not work well.

Discussion

Overall usability

Within the context of our chosen tasks, the Diabetes Canada CPG website had good usability with a task success rate for both information and resource retrieval of 79%. Users brought up many positive qualities of the website in the interview, including praise for the individual content on the website as well as design elements. A third of users praised the website for being intuitive.

Effective content

The different versions of guideline content on the Diabetes Canada CPG website facilitated task completion. For the information-retrieval tasks, the majority of successful attempts (90%) were completed using the interactive tools, static tools, or quick reference guide, rather than the full guideline chapters. Users also considered the alternate versions to be strengths of the website, while full guideline chapters were perceived to be less useful.

The semi-structured interview elicited “Comprehensiveness” as a strength and “Not enough content” as a weakness of our website, which may seem contrary to the barriers of time and information overload. One explanation is that CPGs can have serve other purposes, such as for dedicated learning outside of clinical care, in which there are less time constraints. Guideline use can be improved by having alternate versions for different purposes [7]. Clinicians have significant point-of-care information needs, raising 0.18 to 1.5 questions on the average patient, most commonly regarding treatment and diagnosis [21]. However, time and information overload are important barriers to seeking information [21,22,23]. Even at the point-of-care, clinicians can have different preferences regarding volume of information, with “competing demands of brevity and comprehensiveness” [20, 24]. While full text guideline documents are more comprehensive, alternate versions contain targeted information which can be utilized more quickly at point-of-care. Thus, creating websites with multiple versions of the guidelines in one place gives users the choice of how and how much CPG information they want to use, and may ultimately improve uptake.

Users also wanted to see more visual content such as tables and algorithms on the website. Visual presentation can enhance information delivery and has been recommended by multiple authors [3, 25, 26]. Interestingly, we found that while visual presentation may speed up information delivery, it does not guarantee accuracy: task 1 took users the least time to complete, yet it was the task with the second most errors. Usability testing should be considered for elements like algorithms, to ensure accurate information delivery.

Effective format

Existing recommendations for CPG format include using boxes to highlight recommendations, algorithms and tables, different versions, and layering [5, 6, 26, 27]. However, little recommendations exist in the CPG literature pertaining to web design.

Most users brought up the fixed navigation bar on the website as a strength, as it allowed them to jump around different sections without using the back button. Many users also liked the quick buttons on the home page that linked to the interactive tools. In contrast, organization of links in other parts of the website were rated poorly as was the search function, which performed poorly, with only 15% of attempts generating the desired result. A major weakness of the search function we identified was the default search mode of “exact phrase” for search terms. This could lead to omission of desired results if the phrase is worded differently on the webpage or if the user makes a spelling mistake. The algorithm may also be flawed, as we sometimes observed less relevant webpages containing the search terms prioritized over the more relevant webpage in the results. The World Wide Web Consortium’s guidelines on web accessibility recommends implementing search functions that can generate results that account for spelling errors, different endings on search terms (stemming), and use of synonyms [28]. It also recommends using techniques like meta tags to optimize results. Our next steps for improving the website includes re-designing the search function to reflect these recommendations.

In our study, the average amount of time taken to complete information retrieval tasks was only 160 seconds. This is comparable to the amount of time actual users spent per session on the previous iteration of the Diabetes Canada CPG website (180 seconds) and less than the amount of time internal medicine physicians have been observed searching online sources while working (252 seconds) [11, 29]. When a task took much longer to complete (task 7 at 352 seconds), it resulted in the lowest success rate (43%), with some users giving up after not finding information quickly, highlighting the importance of intuitive navigation.

The strengths and weaknesses identified in our study of a CPG website are echoed in existing website design guidelines not specific to clinician use. For instance, the use of navigation menus (left-hand positioning specifically for primary menus) is recommended in the Research-Based Web Design & Usability Guidelines published by the U.S. Department of Health and Human Services (HHS) [30]. Both the HHS and International Organization for Standardization (ISO) guidelines recommend allowing access to important options on the home page and providing an effective search function [30, 31].

Recommendations for CPG website design

Based on our findings and literature review, we have synthesized a list of recommendations for guideline developers and disseminators (Table 12) that can be adapted or added to existing guideline implementation planning checklists and tools [32]. Incorporating web design recommendations in the assessment of CPG websites could lead to improvements in their usability and uptake.

Table 12 Recommendations for the design of Clinical Practice Guideline websites based on our findings and literature review
Full size table

Study limitations & strengths

Limitations include convenience sampling of participants, lack of iterative testing, and the inability of usability tasks to completely replicate real-world use of the website. Our study demographic overrepresented younger, female users (64% younger than age 40, 64% female) and experienced users. This may have affected some of the study results, although we know from real-world data of our website that the actual end users also skew towards younger, female users (64% younger than age 44, 70% female). Younger users are more likely to be comfortable with using technology and may be more adept at completing tasks. Sex has not been found to influence user experience of websites in other studies [33]. Previous experience may have led to better completion rate and speed, but studies show that experience affects these metrics minimally when web designs are highly effective [34]. Strengths of the study include the variety of usability tasks chosen, inclusion of participants with differing clinical backgrounds, mixed-methods design, and and independent coders.

Conclusion

Multiple versions of CPGs (e.g. interactive tools, static tools, summaries) can be used to answer clinical questions more quickly. Usability testing can be used to identify previously unknown issues with CPG content. Effective web design should be assessed in the creation of CPG website.