Keywords

1 Introduction

MOOC (Massive Open Online Course), as a type of E-learning and distance education, provides unique learning opportunities for underprivileged people, including people with disabilities, elderly and people who live in countries with low infrastructure. Different MOOC platforms such as Coursera, edX, Desire2Learn, Canvas, and FutureLearn host hundreds of courses with millions of participants. For example, by August 2015, Coursera offered more than 1100 courses from 121 partner universities and 15 million users [13].

In Norway, MOOC has drawn much attention in both research and education recently. MOOC.noFootnote 1 is a Norwegian MOOC platform built upon Canvas that hosts courses offered by Norwegian educational institutions. This platform is provided by BIBSYSFootnote 2, an agency under the Norwegian Ministry of Education and Research. Most recently, our college together with seven other European universities received funding from the EU Erasmus + programme for the MOOCAP project (MOOC Accessibility Partnership) in order to develop a series of MOOCs in accessibility and universal design of ICT. The Norwegian government has appointed a MOOC-committee to investigate how the Norwegian educational authorities and institutions should meet the challenges posed by the rapid growth of higher education delivered over the Internet. In their report [6], the committee stated, “MOOCs have the potential to strengthen both the access to and quality of higher education for persons with reduced functional ability, thus making it possible for them to choose higher education”. According to the committee, high-quality technology and contents, and adherence to the principles of universal design are a prerequisite for the ability of MOOCs to make education more accessible to everybody.

Literature has shown that although MOOC courses have been studied from data analysis, student activities, pedagogy, and accessibility perspectives [4, 7, 10], very little attention has been paid to the accessibility of the platforms from instructors’ perspectives [2, 8, 9]. Although some platforms provide accessibility guidelines to enable instructors to create accessible course contents, it is also important that the MOOC platforms themselves are accessible for instructors. By instructors, we mean the people that use MOOC platforms to create courses, including people with disabilities. We argue that a systematic study focusing on the accessibility of the MOOC platforms from instructors’ perspective is necessary to ensure universal and equal access to the MOOC platforms.

In this paper, we present results from a heuristic evaluation of Canvas conducted as part of an ongoing project where we examine the accessibility of a carefully selected group of MOOC platforms and their support to instructors in order to create accessible course contents. The goal of this project is to investigate the accessibility of MOOC platforms from instructors’ perspective, identify accessibility problems, and propose recommendations for addressing the problems. Our overall aim is to contribute to the global efforts in ensuring universal design of MOOCs and equal access to education.

2 Background and Related Work

With the increasing popularity of MOOCs, research communities on accessibility have started to pay attention to the accessibility of MOOC platforms and courses. Such research is still in its early stage. In this section, we will present the current research on MOOC and accessibility.

The majority of research published so far focus on the accessibility of courses. For example, Al-Mouh et al. [1] tested the accessibility of Coursera courses from users and experts perspectives. For users, they tested a set of essential tasks using screen readers. For experts, heuristic evaluations were conducted with 10 courses. They found that the courses failed to comply with WCAG 2.0 guidelines. Sanchez-Gordon and Luján-Mora [8] selected five Coursera courses and conducted heuristic testing to identify the potential accessibility problems for elderly students. They found that all the courses have accessibility issues. Calle-Jimenez et al. [3] used three automated tools to evaluate a Geo-MOOC course. They found that although one of the tools reported more accessibility problems that the others, there are still issues that none of the tools could detect. They further argued that user testing should be conducted in order to complement results from expert testing and testing with automated tools.

Another category of research focuses on the platforms and providers. For example, Bohnsack and Puhl [2] evaluated MOOC platforms including Udacity, Coursera, edX, OpenCourseWorld and Iversity. They used protocol observation when conducting user testing with blind users, rather than W3C guidelines. edX was found to be the only platform that is accessible with their test configurations for blind users. Iniesto et al. [5] analysed the degree of accessibility of two MOOC providers (UNED COMA and UAb iMOOC). Their findings showed that none of the platforms have achieved a level that could indicate that they are accessible and understandable to the users.

3 Evaluating Canvas

3.1 Canvas

We have chosen CanvasFootnote 3 as our case for the evaluations. Canvas is an open source Learning Management System (LMS) from InstructureFootnote 4 used as a MOOC platform, e.g., in the Canvas NetworkFootnote 5.

LMS platforms aim to serve educational institutions having a managed number of users (administrative staff, teachers, and students), making scalability predictable, and offering a relatively advanced portfolio of services for course creation, course and class management and administration, as well as tools for creating and editing learning materials. MOOC platforms aim to offer a number of courses to a less predictable number of users, thus entailing much higher requirements for scalability. MOOC platforms usually offer only a subset of the tools that an LMS offer.

Canvas, from the instructor’s perspective, is organised as a dashboard for navigation, courses, grades area, and a calendar. It contains a number of tools for planning and managing course content, learning materials, and students. The Dashboard is the page displayed after login. It presents the user with a list of recent activities, a list of upcoming tasks, and link to a calendar view. Through the Dashboard, users navigate between activities, tasks and the calendar. In addition, instructors can add a new course here, if s/he has permission to add courses. The Modules area is used for organising a course, and displays as the course home page to students. The course home page is the first page students see when they open a course. In Modules, the instructors can create and edit modules in a course, manage the content elements in a course module, and reorder modules and elements to organise the course flow. Functionality offered for managing course modules and content elements include add, edit, delete, move, and rename. In addition, the modules and content elements in a module can be individually published or unpublished.

There are several kinds of content elements that can be added to a module; pages, files, discussions, assignments, quizzes, announcements, and other learning materials. Each offers necessary functionality to add/edit content and to set element specific options. Pages is for adding content and learning materials to a course or a class wiki in the form of text, images, video, as well as links to other pages and files/resources. Specific user access can be set for each page. The page history is available to the instructor. Files allows the instructor upload learning materials and other resources such as documents, images, and media in different formats. Resources may be course specific. Discussions is for graded or non-graded class discussions of course related topics. Assignments provides opportunities for assessing students. It may include quizzes and discussions that may be graded. Quizzes allows creating and administering online quizzes and surveys, and can be used to conduct exams and assessments. Announcements offers functionality for communicating with students and post course-related topics. In addition, Canvas offers a number of other tools, including tools to manage grades, syllabi, evaluate course components and student performance, as well as to conduct real-time lectures or conferences.

The text editor is utilised in most of these content elements. The WYSIWYG text editor in Canvas is from TinyMCEFootnote 6, which is a text editor available in several similar platforms, for example as a plug-in for Moodle. TinyMCE offers both an html editor and a rich content editor and supports embedding video content, math formulas, and other rich media. Instructors can use both editors when adding content to courses.

3.2 Methods

We have evaluated the accessibility of the Canvas platform from instructors’ perspective through heuristic evaluation based on the Authoring Tool Accessibility Guidelines (ATAG) 2.0 by W3C Web Accessibility Initiative. Three researchers performed the evaluation, two of the researchers have several years of teaching experience in higher education and uses LMS tools on a daily basis. All three researchers have a background in computer science; two have many years’ experience with research in the area of digital accessibility.

ATAG 2.0 is a set of principles organised into guidelines and criteria for accessible authoring tools. It is divided into two parts, one targeting the accessibility of the authoring tool interface for authors/users (Part A), and the other how these tools support the creation of accessible content (Part B). ATAG 2.0 shares the conformance model in Web Content Accessibility Guidelines (WCAG) 2.0 and has three levels of conformance: A (lowest), AA (middle), and AAA (highest). Each higher level builds on and incorporates the lower level(s), thus conformance to level AAA implies conformance to levels AA and A, and conformance to level AA implies conformance to level A. It is necessary to conform to all success criteria on a certain level to be regarded compliant with ATAG 2.0 on that level. Our aim is to investigate the accessibility of the platform as experienced by instructors creating and managing MOOC courses, therefore we have used ATAG 2.0 Part A in our heuristic evaluations. Based on an examination of the principles and success criteria in ATAG 2.0 Part A, we selected 28 success criteria relevant for our evaluation.

The open source version (not production version) of Canvas used for the evaluation was installed on a server running Ubuntu 14.04 in early December 2015. The main bulk of evaluations took place in December 2015 and January 2016, and were conducted on laptops with operating systems Ubuntu 14.04, Windows 7 Enterprise, Mac OS X El Capitan v.10.11.3, and browsers Firefox 39.0 and 44.0, Chrome 47.0.2526.111 m, and Safari v. 9.0.3. Assistive technology used in the evaluations include screen readers NVDA v. 2015.3 and VoiceOver v. 7.0, in addition to built-in magnifiers for Windows and Mac OS X (no magnifiers were used with Ubuntu).

As we are looking into accessibility from instructors’ perspective, we first identified the main tasks instructors may have to handle in relation to creating, running, and maintaining a MOOC course. These tasks were grouped into three categories; (1) Course access management; (2) Creating and organizing course content; and (3) Grading and feedback.

Course access management involves tasks such as enrolling students and allowing or removing access to the course for course instructors and others. As administrative staff commonly handles these tasks, we have concentrated our evaluation on the two other categories that contain tasks typically handled by course instructors.

Creating and organizing course content covers the main tasks that a course instructor will perform as part of preparing and running a MOOC. These include creating and editing a course page and a course plan, modules, syllabus list, assignments, announcements, and quizzes. In addition, this task includes preparing and/or uploading learning materials such as lectures and relevant resources.

Grading and feedback includes grading assignments and giving feedback to students. As participating in discussions with other students is an important part of a MOOC for many students, we have included creating and editing discussions as one of the tasks in this category.

To evaluate the accessibility of Canvas for instructors, we selected a set of Canvas elements relevant for a chosen set of the above tasks. The Canvas elements used in the evaluation includes Dashboard, Modules, Files, Pages, Quizzes, Assignments, Announcements, and Discussions. The researchers first conducted individual evaluations. After that, the individual results were collected and discussed together by all the researchers.

4 Evaluation Results

The results from our evaluation show that Canvas does not fully comply with ATAG 2.0, Part A. From the 28 criteria used in the evaluation, we found that Canvas complies fully with 11 criteria, complies partially with eight, and does not comply with one criterion. Eight criteria were not applicable or not available. An overview of our evaluation results can be seen in Table 1. Please note that partial compliance in some cases indicate variations in results among the browsers included in the evaluation. In other words, if Canvas complies to one criterion in one browser, but not in another, it is considered as partial compliance.

Table 1. Overview of Canvas compliance with ATAG 2.0. C = compliant, not C = not compliant, partial = partially compliant, n/a = not applicable or not available.
Full size table

Looking at levels of conformance (A, A, AAA), we can see that from the 14 evaluated level A criteria, Canvas was found to comply fully with 6, partially with 5, none were not compliant, and 3 were not applicable or not available. For the 7 level AA criteria, Canvas complies fully with 3, partially with 2, none were found not compliant, and 2 were not applicable or not available. For the 7 level AAA criteria, Canvas fully complies with 3, none partially, 1 not compliant, and 3 not applicable or not available. An overview of Canvas according to level of compliance is shown in Table 2.

Table 2. Overview according to level of compliance (A, AA, AAA)
Full size table

During the evaluation, we identified the following accessibility issues for the partial and not compliant cases. The relevant guideline(s) are shown in parenthesis:

  • Not all non-text icons show text alternatives when mouse-over. (2.1)

  • Spelling errors are not always indicated visually or read by screen readers. (2.2)

  • The screen reader does not read alternative text for all buttons in the text editor toolbar in all browsers. (2.1, 2.2, 3.1)

  • No customization for keyboard shortcuts. (3.1)

  • No keyboard shortcut to save editing contents. (3.1)

  • Animated gifs cannot be stopped. (3.3)

  • Search results do not include alternative text for embedded images in the editing view. (3.5)

  • Number of matches after text search is not indicated by the screen reader when using some screen readers and browsers. (3.5)

  • Only option for Undo/Redo in editing view is by using keyboard shortcuts. (4.1)

  • No undo/redo option or confirmation when uploading files. (4.1)

Please note that all the accessibility issues involving the editing view do not belong to the Canvas platform itself, but is related to the fact that there is no search tool in the text editor plug-in used in the version of Canvas that we evaluated. Consequently, the searches in this evaluation were performed using the search functionality available in the browsers.

In the following, we present some examples of the above accessibility issues. In the Modules tool, some non-text icons do not show text alternatives while mouse over, but the screen reader reads the alternative text (2.1.1). Misspellings are indicated visually (red underline) in the text editor, but there is no indication of spelling errors when adding title to a course element, such as an announcement, discussion, page etc., with the exception of Safari with VoiceOver. Screen readers indicate spelling errors when using Safari and Firefox, but not in Chrome. In IE, the NVDA screen reader reads “button” for most buttons in the text editor toolbar, except a few (font colour, background colour, paragraph, font size) (2.2.1).

Although navigation by keyboard works relatively well in the evaluated tools, the keyboard focus often ends up in the address field (URL-field) of the browser after closing a dialog box, which may be experienced as troublesome by the user, adding a number of unnecessary keystrokes trying to find back to where s/he was on the page (3.1.3).

Search results in the editing view are only visually indicated, the screen reader does not indicate if there are any matches, with the exception of Safari with VoiceOver. Search results do not include matches in alternative text for embedded image (3.5.1).

In Files, there is no Undo/Redo option, but there are Cancel & Open buttons. Once the user has chosen the Open button, the file upload starts without any further confirmation from the user to proceed. This may present a problem in cases where the user accidentally clicked the button, or the file is very large (4.1.1).

Other accessibility issues that instructors may find challenging include the screen reader not reading all pop-up messages, for example in the Files, where the warning informing users how to improve accessibility when moving files is not read by the screen reader, and the poor contrast of non-visited elements in the left hand side menu, as shown in Fig. 1.

Fig. 1.
figure 1

Poor contrast in left hand side menu

Full size image

Canvas provides hints to users about how they can improve accessibility by giving context-relevant warnings in some tools, which is a very useful feature. For example in Modules, where a warning message pops up when the user hits the module reordering button by mouse-over or by keyboard navigation. Another similar example is the Calendar, where users are advised to use the Agenda view for improved accessibility. However, for users not able to perceive the screen visually, navigate by keyboard and use assistive technology, some of these messages are difficult to find unless already known, and may therefore be missed by the very users they are intended for. Figure 2 shows a warning message for improved accessibility in Modules.

Fig. 2.
figure 2

Warning message hint for improved accessibility in Modules

Full size image

5 Discussion and Recommendations

The role of the instructor varies among different MOOC platforms. In some platforms, instructors do not have direct access to the platform; they instead hand their course material over to someone who will take care of uploading the course material and manage the platform. After uploading, the instructor can view the course material, but has no direct access for making amendments. Consequently, instructors do not have any relations at all with the MOOC platform itself in these cases. In Canvas, the instructor has direct access to the platform for managing course modules and editing course contents, which from an instructor’s point of view may be an advantage. However, this also poses a greater challenge regarding the accessibility of the platform with a potentially greater diversity of people accessing it.

The evaluation results for Canvas accessibility are relatively positive in comparison with findings in our previous research where we evaluated LMS platforms [11, 12], including Moodle, which is another LMS commonly used for MOOCs. There are, however, other issues relating to the Canvas platform that may present additional obstacles for instructors. In the following, we explore some of these potential challenges.

Users are informed through the online documentationFootnote 7 that Canvas supports the two latest versions of any browser, and recommend users to update to the newest browser and most up-to-date Flash plug-in available. In addition, Canvas presents the user with a list of supported browser versions. There is however no information about backward compatibility to earlier versions of the listed browsers. Assistive technology, such as screen readers, rarely manage to keep up with the rapid update of versions that is common in software such as browsers. In addition, many users use older equipment, computers, operating systems and other software due to a variety of reasons, including their familiarity, availability, and infrastructure, as well as economic or other reasons. This makes backward compatibility with older versions an important issue to ensure equal access for a diversity of instructors using a wide range of equipment.

In the online documentation, Canvas also informs users which screen readers are supported for certain browsers. For example, for PCs, JAWS is supported for Internet explorer versions 10 and 11, and NVDA for the latest version of Firefox. Users are also informed that there is no screen reader support for Canvas in Chrome.Footnote 8 This places limitations on what browsers users of assistive technology can use with Canvas, and considering that Chrome, according to the browser statistics in 2015Footnote 9,Footnote 10 is by far the most used browser worldwide, this may affect many instructors that are screen reader users.

In a previous project [11, 12], we have investigated three LMSs from teachers´ perspective: Moodle, Fronter, and SAKAI. The results from the Canvas evaluation shows fewer accessibility issues than we found in our investigation of Moodle and the other two LMSs, but there were some common issues in all platforms. In the following, we compare the results from the Canvas evaluation with results from our previous research [11, 12], specifically the evaluation of Moodle.

Both platforms have only static input components (3.2.3), and previews show the course content as it will be presented to students (3.7.1). Neither of the two platforms provide options for customising keyboard shortcuts (3.1.5). Canvas complies fully with more criteria than Moodle; there are nine criteria only Canvas complies fully with (2.2.2, 3.1.4, 3.1.6, 3.2.1, 3.2.2, 3.6.3, 4.1.3, 4.2.1, 4.2.2), while Moodle complies fully with one criteria that Canvas complies with only partially (3.3.1). Both platforms partially comply with five criteria (2.1.1, 2.2.1, 3.3.1, 3.5.1, 4.1.1).

Considering possible reasons for these differences, particularly as Moodle and Canvas both use the same text editor plug-in, one reason may be the time span of almost three years between the evaluations of Moodle and Canvas and the focus on accessibility has increased the past few years. We hope that this is an indication of improved accessibility in general in these and similar platforms.

Based on the evaluation of the selected Canvas course elements according to ATAG 2.0 criteria, we recommend the following for increasing accessibility in Canvas.

Improve Support for Efficient Keyboard Navigation.

Ensure keyboard focus comes back to the place it was before opening a dialog box. Add keyboard shortcut to save editing contents (in text editor).

Improve Support for Screen Readers.

Ensure that screen readers always indicate spelling errors and the number of matches in search results. Ensure screen reader support for all commonly used browsers, e.g., Chrome, and in particular free screen readers such as NVDA.

Improve Options for Avoiding or Correcting Mistakes.

Ensure confirmation from user before starting upload of large files. Provide Undo/Redo options, including keyboard shortcuts, when setting properties for elements, e.g., when creating a quiz. Provide Undo/Redo options as visible choices in the text editor, for example as buttons, to avoid that this feature is available only as shortcuts.

Other recommendations include adding options for stopping or not automatically starting animated gifs (in text editor), include alternative text in search results, and ensure backward compatibility with older browsers.

6 Conclusion and Future Work

Focusing on Canvas’ accessibility for instructors creating course contents in MOOCs, we have presented our findings from a heuristic evaluation of the Canvas platform based on a selected set of 28 success criteria from ATAG 2.0, Part A. We have also compared the results with results from our previous evaluation of Moodle. Our results show that Canvas complies better with the selected criteria when compared with Moodle. Our research shows that Canvas provides much support for instructors to create accessible MOOCs, but it does not comply fully with ATAG 2.0.

We have not included ATAG 2.0 part B in our evaluations so far. To investigate how well Canvas supports instructors in creating accessible course content, we will need to conduct heuristic evaluations of ATAG 2.0 part B. In addition, user testing on Canvas with a diverse group of participants will be necessary to discover further possible challenges that cannot be revealed by heuristic evaluations alone.

To continue our systematic investigation of accessibility of MOOC platforms from instructors’ perspective, we are planning evaluation of other MOOC platforms. Future work includes user testing on the evaluated platforms with instructors in order to validate the results from the heuristic evaluations and identify any further accessibility problems not revealed in the heuristic evaluations.