Keywords

1 Introduction

The Erasmus + European project “MOOCs for Accessibility Partnership” (MOOCAP) [1] set out to provide education on accessible design in Information, Communication and Technology (ICT) covering a wide range of topics that would be of interest to all those involved in the delivery and consumption of digital products, publications and services. Over the course of three years, the project team delivered a series of Massive Open Online Courses (MOOCs) and online courses, all available as Open Educational Resources (OERs) with a Creative Commons Attribution 4.0 International Licence. The introductory course aimed to help learners understand how “those with sensory, physical and cognitive impairments may be disabled by barriers encountered when using digital technologies.” [2] The MOOC did not require any previous knowledge about the topics covered, but set out a series of accessibility skills that could be acquired if a learner completed the course over five weeks. 9642 learners registered on the three runs that were held in 2016–2017 and of those who signed up 5,604 (58%) stayed to at least view one step at any time in any course week. According to FutureLearn [3] “the average number of learners on a course is 50% of joiners”.

Research has shown there is a wide variation in the reasons for dropping out of courses [4] due to internal and external factors [5, 6]. Examples include:

  • “No real intention to complete,

  • Lack of time,

  • Course difficulty and lack of support,

  • Lack of digital skills or learning skills,

  • Bad experiences,

  • Expectations,

  • Starting late and

  • Peer Review” [7].

When developing the Digital Accessibility MOOC, the project team was aware of the high dropout rates and wanted to ensure the course platform was accessible, easy to use and encouraged interaction with tutors and mentors. Ease of use checks with assistive technologies showed that FutureLearn fulfilled the accessibility criterion and offered captions and transcriptions for all the videos or audio files uploaded to the system. There was the added bonus of an embedded discussion forum so that learners could easily comment on the content. Researchers have suggested that FutureLearn is a social learning platform because comments appear alongside each step or activity, rather than requiring the learner to work in an unrelated discussion area on the website. The former tends to encourage increased interaction between learners and this was seen as being important, where those participating on each course run outnumbered the teachers and mentors [8].

Those creating the MOOC (eight university departments in seven different countries) were aware that their contributions varied in content style and topic from discussing different types of disability with personas to complex accessibility standards related to web page design and development. The authors of this paper, coming from a partner university, were interested to see how many learners completed the various activities presented on the Digital Accessibility MOOC, as well as how many actively engaged with content and their peers. By analyzing the comments from the various steps and activities alongside completion levels it was felt there might be the possibility of seeing which topics encouraged engagement and which should be the focus of future courses and in particular suitable for repurposing into an alternative format and offered as additional Open Educational Resources.

2 Method

The initial task was to discover how many learners completed each step. Questioning whether there were any anomalies or was there a general level of dropout over the course of the five weeks? A step (as seen in a FutureLearn MOOC) is made up of a single topic page that may contain a short video or audio clip with text, possibly images and usually a question to engender discussion. Steps are grouped as activities, which are rather like a section in a chapter of a book. The chapter becomes the week of activities in the online course. There followed an analysis of the number of comments in each step compared to the number of learners. The learner’s unique ID was used due to data protection and where steps required no commenting these were omitted. It was felt this comparison might illustrate different levels of engagement to the types of content or topic. Any replies to comments were omitted from the analysis as these tended to show appreciation or question the previous comment rather than the content within the course. The results were aligned with the number of ‘likes’ on each original comment about the content to illustrate more active engagement with the topic being discussed.

Finally, an analysis of the comments found in the reflection steps, offered at the end of each week, was undertaken to find any discussion that might be related to particular topics. It was felt that this task would not only offer an insight into the learning outcomes, but also whether adaptations should be made to the way a topic was presented.

3 Results

Of the 5,604 joiners on the three runs of the Digital Accessibility MOOC only 4185 were active learners. According to FutureLearn these learners are “those (of any role) who have completed at least one step at any time in any course week, including those who go on to leave the course.” Recording the completion of a step is dependent on the learner manually selecting the ‘mark as complete’ button. When analysing the 98 steps over the five weeks it was found that the rate of completion dropped from 2465 learners per step to 839 learners. This was not unexpected as stated at the beginning of the paper and it was found that the rate of attrition was relatively uniform after the first week (see Fig. 1).

Fig. 1.
figure 1

The average number of participants who completed a step per week

Full size image

Moreover, despite encouragement from mentors on the Digital Accessibility MOOC to engage with the questions posed at the end of many steps or to be reflective at the end of each week and to discuss topics independently of the teaching staff, it was found that the majority of learners still failed to regularly comment. Swinneton et al. [8] discovered in their research, on average 47% of those using the discussion area only make one or two comments. On the Digital Accessibility MOOC the average number of comments per learner was 6–7 throughout the three runs.

Further inspection of the data showed that the average percentage of learners who completed an activity and commented on the topics in the reflection step provided each week, varied very little: 21–27% over the five weeks. However, when linking this to the percentage of ‘likes’ for particular comments it became clear that there was a distinct difference marking out week 4. It had the second lowest number of comments relating to the reflection step (214) with the second highest number of likes for the comments about the content (415) (Table 1).

Table 1. Number of comments and likes compared to the number of learners per week
Full size table

On inspection of week four’s content, spread over 20 steps, there were fewer questions compared to the other weeks. Only nine steps had questions, and there were three quizzes to break up the amount of content and allow learners time to self-assess their knowledge building. Questions in the activity steps asked for answers that related directly to a web accessibility activity such as the one asked after Step 4.7 “What barriers can a web form have for people with limited dexterity?” The reflection steps at the end of each week always had the same three questions namely:

  1. 1.

    What have you found to be good, useful or interesting this week?

  2. 2.

    What actions will you now take to learn more about these topics and take the lessons forward in your life and work?

  3. 3.

    What questions, if any, have arisen for you?

So the question asked, was why this particular reflection step caused an increased amount of engagement? There were nearly twice the number of likes per comment, compared to other reflection steps (34% to 66%). It was decided that a thematic analysis of the comments might show what had made this particular reflection step different from the others. An initial familiarisation with the content of the 214 comments showed that learners had sometimes answered all three of the questions in one comment. Data within the comments was coded and a thematic analysis undertaken [9]. Four themes appeared, an appreciation of the week’s content in terms of helpfulness and raising awareness, mentions of complexity, the need for further materials and the wish to revisit materials with other comments relating to particular web accessibility questions. Table 2 shows the worked-out themes from the comments (with percentages) found in the week four reflection step.

Table 2. Themes for classifying comments provided in the week four reflection step.
Full size table

4 Discussion

By examining the content of activities for week four, it became clear that this was not only a very full week, but also the activities might be considered quite complex, because it was about web content accessibility standards and the barriers to ease of access for those with disabilities. It included Web Content Accessibility Guidelines (WCAG), conformance and user testing often undertaken by developers or those involved with online content development. However, when posing questions for reflection, with guidance to help learners think about general aspects of the week’s activities, the most popular answers showed appreciation to those who provided the content. This gratitude was welcoming, but made the authors of this paper wonder whether the actual reflection questions needed to change in order to provide a better understanding of the learner outcomes.

However, research has shown that when learners participate actively with comments such as replying or liking, it shows more active engagement with the course [10] which could be said to be true of week four as there were comments that revealed a deep interest in the subject. Those who managed the activities often said how much they appreciated the amount of information and external links. The minority who mentioned the complexity of it all (9%) or were new to the subject often praised the way it had been broken down into steps, but still wanted an equivalent of a ‘Dummies guide’ with key points and less technical language.

There were other comments about the amount of links and resources provided and how they would need to be revisited or shared with colleagues at a later date. These comments highlighted the fact that time may have been an issue, with a sense of overload in terms of the amount of reading material provided. The idea of offering slides with key points and even smaller mini chunks of content with diagrams for the crucial elements might have helped learners cope better in this type of situation. Videos and audio recordings were purposely kept below three minutes and in the reflections, no one said they were too long. Transcripts and captions were offered and the former could be downloaded and kept for reference.

There was an awareness that the materials would only be available for 14 days after the course unless further access was purchased. On the other hand, it was pointed out that, for this particular course, all the materials had been made available and could be repurposed in different formats, once gathered from the MOOCAP project website.

When examining the comments regarding particular activities (29%) it was clear that practical advice, such as the use of web accessibility testing tools, was also highly valued. Several learners on the course shared their expertise in their comments. One of the limitations of a MOOC is the difficulty of providing hands on experience from a distance. The repurposing of this aspect of the course with more interactive online activities, illustrating how the tools worked, where they were useful and where they failed to pick up accessibility errors, would have provided improved support. This could have helped those learners who were educationalists rather than developers.

Finally, the problem of having to leave the MOOC platform to read or interact with other websites could be considered a weakness of all online learning where there are hyperlinks out to similar areas of interest. Not knowing the skills and interests of the learners who join a MOOC can make it difficult for content providers. A judgement has to be made as to the level and complexity of knowledge provided and in the case of the Digital Accessibility MOOC there was a tacit understanding that the content would be offered at several levels with the external links providing basic and/or advanced support materials. All the resources had clear explanations, but these could have been presented with indications as to the level of knowledge required, perhaps on a sliding scale from basic to advanced or college level to post graduate.

5 Conclusion

Many researchers have commented on the way learners fail to complete MOOCs based on course completion figures, the gaining of a certificate or the passing of tests. It could be said that this way of measuring the success of a MOOC fails to address the diversity of those participating and what they may wish to achieve. The Digital Accessibility MOOC had two main groups of participants namely educators and those in Information Technology professions and when reviewing the comments in the week four reflection step, it was clear that members of both groups had engaged with the content. However, the reasons for engagement included the wish to pass information on to others and use of the knowledge gained in daily work, as well as showing a genuine desire to participate with particular topics despite their complexity. Some learners were willing to share their knowledge when others included questions in amongst their comments and topic specific threads developed. This process of discussion, commenting and liking comments has resulted in researchers highlighting the potential for social learning via MOOC platforms. However, as Tubman et al. [11] point out “MOOC environments have unique challenges for pedagogy which are not present in other socioconstructivist learning environments: the scale and diversity of participation”. Nevertheless, these peer on peer interactions can be very valuable as a way of raising awareness about learners’ positive and negative feelings and when problems are arising.

Even with a very small number of comments from one reflective step taken from one week in the Digital Accessibility MOOC it was possible to prioritise the type of activity that should be repurposed into alternative formats in order to better support some learners. For example, the highlighting of crucial elements of the content into a series of online interactive slides, such as offered by SlideWiki [12], that can be made available as open education resources (using the Creative Commons licence CC-BY 4.0), would offer support to those wishing to share the information as well as collaborate on the topic. The addition of metadata to provide information to learners about the level of each activity may also be useful to prevent concerns about complexity, but a true appreciation on the part of authors as to the time it takes to absorb materials is hard, due to the diversity of those participating as mentioned before.

Most MOOCs offer content in a variety of layouts, with use of text and multimedia, as a whole course and rarely offer their content in ways that can be easily repurposed. As part of the development of the content for the Digital Accessibility MOOC as well as to facilitate the collaboration of experts within different countries, institutions and fields, a “stepwise” process for authoring was organized. Each learning object was carefully documented, with accessibility metadata and grouped in small chunks of information making up the various activities [13]. These individual artefacts such as single images, audio files to videos with their transcripts and caption files plus text based content in separate word processed documents can be adapted to suit all levels of education. The concept of offering multiple ways of learning has been recognised as an effective way of encouraging inclusive teaching and learning practices with “multi-modal opportunities in terms of representation, engagement and expression” [14].