1 Introduction

In recent years, digital newspapers are outdoing the traditional newspaper [1] and digital media has become the most important arena for general information sharing and public debate. In the emerging e-society, this trend will likely continue, with even more of the media content and public debate taking place online. Thus, media plays a key role in ensuring freedom of expression, an essential foundation for democracy [2]. Consequently, ensuring accessibility for all to these digital arenas is essential for equal participation in society.

In Norway, citizens’ rights to access Internet services are stated in the Discrimination and Accessibility Act (DAA) [3], the regulation for section 14 in DAA [4] and in governmental Information and Communication Technology (ICT) policy. The objective is equal access to the e-society for all citizens. The Norwegian government has allocated large funds to increase broadband coverage and improve the level of digital knowledge in the population. In 2013, 99.9 % of Norwegian households had possibilities for broadband access. Only less than 2000 households lack this opportunity [5]. In primary schools as well as lower and upper secondary schools ICT skills is defined as one of five skills which form the basic conditions for learning and development in school, work and society [6]. However, despite of the fast growing e-society, little attention has thus far been paid to accessibility concerns in the arena of digital media and public debate.

NRK.no is the second biggest website in Norway [7] and the official website for the state-owned Norwegian Broadcasting Corporation (NRK) which is the largest media house in Norway. The website contains several sections, including a news section, an online TV player, radio player, and dedicated content for children. Our research focuses on the digital news section of the NRK.no. It has been conducted in cooperation with NRK New Media, the department responsible for publishing and rendering content at the NRK.no website.

We have previously published, in [8], a summary of findings from end user group interviews and heuristic evaluation of the accessibility of the content creation and management (CMS) tool, Polopoly, in NRK. The evaluation mainly used Authoring Tool Accessibility Guidelines (ATAG) 2.0, focusing on accessibility of the CMS tool for journalists, as well as to what extent it supports the authors create accessible content.

In the current paper, we present a heuristic evaluation of selected web pages on NRK.no using Web Content Accessibility Guidelines (WCAG) 2.0, i.e., from the end users’ perspective, as well as elaborate and discuss our earlier findings in relation to this. In addition, we look into important considerations that need to be addressed when evaluating large and complex websites from large organizations.

The following sections are organized as follows. In Sect. 2 we describe the workflow for publishing content on NRK.no as well as the environment and applications for the publishing process. In Sect. 3, we provide a short summary of our previous research on Polopoly, approaching the publishing process from the journalists’ perspective. In Sect. 4, we take the end users’ perspective, presenting results from focus group interviews and heuristic evaluations of NRK.no. In Sect. 5 we discuss our findings and provide recommendations for improving the accessibility of the web sites. Finally, in Sect. 6, we conclude the paper and discuss future work.

2 NRK.no Applications and Workflow

NRK is a large organization and the website is complex, offering a wide selection of multimedia content. Several applications are part of the publishing process in addition to the CMS and rendering tools, and the process is part of a large environment, both technological and organizational. When evaluating the accessibility of NRK.no website, we need to take into consideration the framework of software tools involved in content creation and publishing, as well as the publishing process itself.

In this section, we give an overview of the workflow for publishing web content on NRK.no and the relevant software tools involved in this workflow.

2.1 Relevant NRK Applications

Two systems are used together to publish content on NRK.no: Polopoly and Panorama. Polopoly, used for content creation and content management, is an external system from AtexFootnote 1. The users of Polopoly are mainly journalists who write articles for NRK, about 400 in total, according to NRK, producing over 250 articles daily. The Panorama rendering system, developed in-house at NRK, is used for rendering the web pages for NRK.no. Figure 1 shows an overview of the relevant applications.

Fig. 1.
figure 1

Overview of environment and applications

Full size image

Polopoly stores metadata and content (articles, media objects) in a database, while Panorama retrieves metadata and content from this database and renders the content on the web page. Thus, Panorama handles presentation and appearance of content at NRK.no, while Polopoly handles editing and managing content. In consequence, to be able to conduct a thorough evaluation using ATAG B.1, which largely concerns WCAG 2.0, we need to include Panorama in the evaluation.

In addition to Polopoly and Panorama, there are other applications involved in publishing content on NRK.no. These are TansaFootnote 2 for spell checking and DisqusFootnote 3 for discussions/comments. Tansa is used as a plug-in or extension to the web browser, and does not work directly with Polopoly. Tansa is available to NRK journalists and mainly used as a spell checker. Disqus is a widely used, free community tool for handling comments. End users log in to Disqus when posting comments to articles open for discussion on NRK.no. Both Tansa and Disqus are third party tools that are used by NRK. Consequently, NRK is not responsible for the accessibility issues related to these tools. To be able to evaluate NRK.no satisfactorily, we will have to take these other applications and tools into consideration, preferably also conducting heuristic evaluation and user testing on these.

2.2 Publication Work Flow

When a journalist creates a new article item, s/he first logs on to the Polopoly web interface and creates a new article in the category where the new article is to appear, for example “Distrikt (Regional pages)” or “Viten (Knowledge)”. Panorama will use this information to render the article using the corresponding template for the given category.

In addition to the category of the article, journalists normally provide the following elements: a headline as the main title, an alternative title to be displayed in article lists, a lead paragraph, and body text of the article. Polopoly provides a text editor for adding lead paragraph and body text. During this process of creating and writing an article, Tansa monitors the text typed into Polopoly, and discovers misspellings.

A variety of article elements may be added to an article such as audio/video objects, fact boxes, citations, images, links, and maps. When adding images, journalists may add a description of the image, which Polopoly will use to generate a caption and alternative text. The journalists also typically add their name(s). There are several other optional settings available to the journalist, such as running head, language, and geo-positioning. Once the article is ready for publishing, the journalist can choose to promote an article on the front page.

The activities of journalists in Polopoly concerns mainly adding and editing content, as well as setting options that will add metadata to the article. The page layout on NRK.no, for instance, the number of columns for a given type of article on NRK.no, is defined and decided by specialists, not journalists. Panorama renders web pages before they are sent to the browser. The pages are rendered according to predefined layout templates created for different page categories, e.g., regional pages, “Viten (Knowledge)”, and “Ytring (Opinions)”. This includes the placing of content as well as choice of colours, font sizes, etc. The design is responsive, thus images, for instance, are scaled on the fly.

3 Summary of Previous Research

The main objective of the evaluation reported in [8] was to discover to what extent the content management system for publishing content on NRK.no, Polopoly, is compliant to Part B of the Authoring Tool Accessibility Guidelines (ATAG 2.0). Although the rendering tool Panorama was taken into consideration where necessary, it was not fully included in the evaluation.

Our evaluation of Polopoly included the criteria in B.2, B.3, and B.4. In addition, we evaluated B.1.2.2 (Copy-Paste Inside Authoring Tool) and B.1.2.4 (Text Alternatives for Non-Text Content are Preserved) which we considered relevant. In total, 26 success criteria were evaluated. Our findings show that Polopoly is not compliant to most of the criteria. Furthermore, we found that Polopoly does not provide enough support for journalists to create accessible content. For example, Polopoly does not have an option to add alternative text to pictures. Instead, Panorama combines caption and title of the picture, name of photographer and bureau as the alternative text for the picture. This creates an unnecessarily long and seldom relevant description.

Experiences and results from this evaluation show that in order to further investigate the accessibility of NRK.no it is necessary to conduct heuristic evaluation and user testing on the web pages.

4 Evaluating the Accessibility of NRK.no

For this evaluation we have carefully selected a set of web pages at NRK.no. These pages are “Ytring (Opinions)”, “Distrikt (Regional pages)”, “Nyheter Beta (News Beta)”, “Kultur (Culture)”, and “Viten (Knowledge)”. The content for these pages is edited in Polopoly and rendered by Panorama as described earlier. The front page has not been included in our evaluations, as NRK uses a different tool for editing this page.

In the following, we first present the focus group interviews, a brief summary published in [8], and then we describe the heuristic evaluation of the selected pages based on WCAG 2.0.

4.1 Focus Group Interviews

Participants and Procedure. In order to ensure that the participants represented the diversity of potential users, we recruited participants through various non-governmental and non-profit organisations. The participants sampled a broad variety in age, education level and work experience. The groups of participants included: (1) people with visual impairment, including blind and partially sighted, (2) people with hearing impairment, including hard of hearing and sign language users, (3) people with cognitive impairment, (4) people with foreign language background, (5) elderly/seniors, and (6) volunteers in an organisation for underprivileged people. In total 19 participants took part in the interviews. The participants were given a list of links to selected web pages on NRK.no in advance so they could prepare for the group interviews.

Five focus group interviews were conducted in March 2014. Two researchers from the project were present: one facilitating the focus group interview and the other observing and taking notes. The interview sessions were audio recorded in agreement with NSDFootnote 4 requirements.

The interview sessions were organised into two main sessions. The first session focused on the participants’ general use of Internet and NRK.no, personal preferences and use of devices. The second focused specifically on their experiences with different functions and design of NRK.no, including general design, structure, layout, navigation, text content, language, multimedia, and debate. For each topic, the interview facilitator opened the discussion with appropriate questions and/or showed examples from the website.

Findings. From the interviews, it is clear that users experience a variety of challenges when using NRK.no. Some identified issues were common across several groups of participants. Regarding structure, layout and design, all groups of participants expressed they experienced navigation elements to be generally adequate.

Despite of the general positive experience, there were several challenges reported in relation to navigation, particularly when using a screen reader or navigating by keyboard. The main issues include menu items not accessible to the screen reader, menu items that do not expand, submenus that cannot be reached using tabulator, unlabelled buttons, and buttons named “…”. Other navigation issues not related to screen readers or keyboard navigation include inconsistent use of names in menus and page title, and inconsistent use of county names in the pages for regional news. The main challenges experienced in relation to videos include the lack of subtitles, and play-buttons not being accessible to screen readers or not visible when using inverted colours/high contrast.

Regarding the use of colours and colour contrast, the challenges concern the use of soft colours, i.e., colours of low saturation, which results in poor contrast and makes it difficult to perceive some elements or areas on the page. This was particularly evident for participants that need to use settings for high contrast or inverted colours. Some participants reported that setting options in the Operative System (OS) for high contrast or inverted colours might even aggravate this effect, so that some elements cannot be seen at all unless moving the mouse over it. Examples of such elements are play-arrows, menu-arrows, and logos. Apart from these issues, participants reported that using high contrast settings and inverted colours (OS settings) works relatively well with NRK.no.

Personal settings allowing users to specify their preferences were communicated as an important feature by several groups of participants. Some of the visually impaired participants reported they were dependent on such settings.

Another comment, common among the participants, is that the debate opportunities were not widely used by the groups. Most people we interviewed had never posted any comments, although some had read posts from other commenters. Participants using screen readers reported that comments posted in discussions were not accessible to screen readers.

Individual groups of users also reported other important issues. For example, participants with visual impairments reported missing keyboard navigation support for menu items and videos, where the play button was not accessible through keyboard. Their comments also included occasions of the image text (caption or alternative text) being needlessly long and sometimes even irrelevant to the image content or containing links. The group of hearing impaired participants commented on the importance of showing whether a video has captions or not, which would save users the frustration of downloading and open a video just to find out it does not have captions. Other participants reported inconsistent use of font or text style for the same text element in some articles, e.g., using italics or quotation marks (“ ”) for direct quotation. The elderly users found that some icons were not easily recognizable and that for some menus, the element names and their corresponding page titles were not consistent with each other.

Although the identified issues did not completely prevent individual participants from using NRK.no, they resulted in low efficiency and sometimes created confusion, frustration and irritation. Addressing these issues will likely improve the overall user experience of the NRK.no webpages.

4.2 Heuristic Evaluation Using WCAG 2.0

The purpose of WCAG 2.0 is to ensure accessible web content. It consists of four principles, each organised into a set of guidelines with success criteria that has to be fulfilled for conformance.

The evaluation of selected pages at NRK.no took place in May/June 2014. The pages included in the evaluation were the same as in the focus interviews. The evaluation was conducted using an HP Laptop (Intel Core i5 vPro) or a PC (Intel i3), both with Windows 7 operative system, Firefox Version 27/28 browser, ZoomText 10 and Infovox screen reader. ZoomText and Infovox are used regularly by one of the researchers who conducted the evaluation.

All 61 success criteria in WCAG 2.0 have been evaluated. Nine criteria were met, 43 criteria were found not met, and the remaining nine were not applicable since NRK.no does not cover all the features in WCAG 2.0. Table 1 shows the results from the evaluation.

Table 1. Results from heuristic evaluation with WCAG2.0
Full size table

In particular, news headings generally appear after images and captions, which makes the content difficult to follow, and time-consuming to find what one is looking for. In addition, the top image of articles is so huge that only a part of it is shown in the browser window. This is particularly confusing for orientation and navigation (WCAG 3.2.3).

The identified navigation issues related to Guideline 2.4 may be regarded as severe. Only two out of the 10 success criteria are met: one in four at level A and one in three at level AA. Issues such as lack of bypass blocks, links with uninformative names, inconsistent use of headings and heading levels, almost invisible focus indicators, and lack of breadcrumb tails make the pages in NRK.no difficult for users to navigate, find content, and determine where they are.

Another issue that needs to be highlighted is Readability (Guideline 3.1). NRK.no does not have a “Plain Language” text alternative. Nor does it have mechanisms for explaining unusual words and abbreviations or possibilities for pronunciation of word with ambiguous meanings.

As Table 1 shows, the selected pages in NRK.no do not fully comply with WCAG 2.0. Since the selected pages are representative in NRK.no, we can assume that the results are valid for the whole NRK.no website. The findings clearly indicate that much work needs to be done in order for NRK.no to be fully accessible. It is, however, important to note that due to NRK’s continuous effort in improving their websites, some of the issues identified in the heuristic evaluation have already been rectified. For example, dropdown menus can now be accessed by keyboard (Guideline 2.1), page sections are adjustable to screen size when zooming with bowser (Guideline 1.4), and an informative label has been added to the search field (Guideline 3.3).

5 Discussion and Recommendations

From the focus group interviews and the heuristic evaluation using WCAG 2.0, we can see that issues related to image captions and alternative text are experienced by users as sometimes long and even irrelevant to the image content. The selected pages in NRK.no are not compliant to WCAG 2.0 Guideline 1.1 (Text alternatives).

When considering how image captions and alternative text are created using Polopoly and rendered through the rendering tool Panorama, and relating this to our evaluation of the CMS Polopoly and the NRK publishing process, we may find a possible explanation. Our heuristic evaluation using ATAG 2.0 show that Polopoly is not compliant to criteria B.1.2.4 (Text Alternatives for Non-Text Content are Preserved) and B.2.2.1 (Accessible Option Prominence). Although Polopoly does allow journalists to add a description for an image, the CMS generates both an alternative text and an image caption from this description. It does not allow journalists to distinguish between an alternative text and an image caption. Nor does it allow them to add a dedicated alternative text. Furthermore, when rendered, Panorama creates another yet different alternative text by combining the generated image caption, photographer’s name, and the name of the photo bureau into an “associated text”, ignoring the original alternative text altogether. Consequently, the combination of Polopoly and Panorama does not give journalists any real opportunity to distinguish between captions and alternative text for images. Nor do they allow them to create separate image caption and alternative text.

Most participants in the focus group interviews commended the quality of article content in NRK.no and there were no comments regarding the use of foreign words or particular challenges regarding textual content. According to the 2013 OECD PIAAC survey of adult skills, Norway scores significantly higher than OECD average in literacy in adults aged 16-65, and the mean score is high both among immigrants and across different educational levels in the population [9]. Further, only 9.3 % of the adult population in Norway has a literacy score of Level 1 in literacy proficiency. Level 1 literacy proficiency equals to being able to read relatively short digital or printed texts to locate a single piece of information that is identical or synonymous with information given in a question or directive [10]. Nevertheless, many people in the general population will benefit from content, especially text, designed to be easy to understand and read, particularly people with cognitive disabilities, people with low language skills, and people with auditory disabilities that may impact reading and perception of written language [11]. Considering possibilities in Tansa for customising dictionaries and looking at text in context, we presume that using these functions in Tansa may increase the accessibility and usability of published textual content.

Participants who are screen reader users in the focus group interviews commented that participating in discussions on NRK.no is extremely difficult, if not impossible, when one cannot see the screen visually. As mentioned in Sect. 2, NRK.no uses Disqus as a plug-in discussion tool. Disqus is written in JavaScript and works on platforms that can handle JavaScript. However, a known drawback when using JavaScript is that it may introduce accessibility problems unless measures are taken to ensure that generated content and functionality are accessible to assistive technologies.

Subtitles on NRK videos is on the wishlist of many participants, including the deaf, hard of hearing, elderly, dyslectics, immigrants, and “language trainers”, regardless of noisy or quiet environments. As many of the posted videos are produced for TV and have captions when shown on TV, we assume that the subtitles issues for the videos on NRK.no can be easily solved technically. It is therefore difficult to understand why the videos do not have subtitles when posted at NRK.no. Today, more and more information is conveyed through videos. Therefore giving all users and potential users of NRK.no possible alternatives for supplementary information is of great importance. The deaf and hard of hearing participants suggsted that marking subtitled videos with a dedicated symbol would save them from having to download the video in order to find out whether it has subtitle.

Participants in the focus group interviews, particularly seniors, commented that they found inconsistent use of names across menu elements and page titles both annoying and confusing. One such example is the menu element named “Oslo” which takes the user to a regional page titled “Østlandssendingen”. Consistency, which relates to WCAG 2.0 Guidelines 2.4 and 3.2, is an important principle for ensuring that the user knows where s/he is and where to find content. We therefore recommend that NRK ensure consistency in their use of names across menu elements and page titles.

As we have shown, Polopoly and Panorama in combination can cause some of the issues identified in NRK.no, and the use of third party application such as Disqus may influence users’ possibility to take part in discussions. Thus it becomes important to take the context, including the environment and applications, as well as the publishing process itself into consideration when evaluating large and complex websites in large organizations, as accessibility issues may be introduced during the publishing process itself, and third party applications may influence the accessibility of the content presented to end users.

6 Conclusion and Future Work

In this research we have conducted heuristic evaluations of the content creation and management (CMS) tool, Polopoly and selected web pages on NRK.no, as well as focus group interviews with diverse user groups. Results from our evaluations show that Polopoly is not compliant with ATAG 2.0 Part B, and that the evaluated pages on NRK.no not fully compliant with WCAG 2.0. We have learned that evaluating a web site containing complex multimedia requires taking the larger environment of applications into consideration, including the publishing process workflow, as these may influence accessibility, usability, and user experience. Further, results from conducted focus group interviews have shown that users in general have positive attitudes towards NRK.no, despite several groups of users experiencing challenges when operating and using content on the website.

NRK has a long history of accessibility awareness and focus on offering accessible content, and they are continually working on improving accessibility of their website and media services. Most recently NRK TV has implemented visual interpretation (audio description) of films and audio caption (captions read by synthetic speech). The technology and solutions could be transferred to NRK.no, thus improve accessibility of NRK.no.

Findings from our research have provided recommendations for improving the accessibility. The results have also indicated that many challenges remain and further research is necessary. In order to gain deeper insight into how end users experience the accessibility of NRK.no, we are currently performing user testing of the same selected pages with the same groups of people who participated in the focus group interviews. We have also recruited people with mild intellectual disability, as well as elderly (age 70 +) who are at the beginning stage of dementia.

To learn more about how NRK can better support the production of universally designed web content, future research needs to include the following aspects:

Journalists’ Perspective: we need to understand how the content creation tool can support the creation of accessible content without disturbing the journalists’ workflow. We also need to understand how increased awareness among journalists about how to create accessible content can be achieved through training and increased knowledge.

The Rendering Process: A closer study of how Polopoly and Panorama works together, particularly the automatic changes made by the rendering tool, Panorama, as some of the issues resulting in lack of compliance to WCAG 2.0 success criteria seem to be introduced in rendering process.

Other Applications: We need to study how the use of Tansa may affect the published content, as well as how NRK can utilise Tansa better for improving accessibility and usability of the web content. The tool used for comments, Disqus, strongly influences users’ opportunity to participate in any discussions at NRK.no, and therefore needs to be evaluated for accessibility issues.