Keywords

1 Introduction

In 2012, when the use of Massive Online Open Courses (MOOCs) exploded, many academics were looking at them as a new avenue with great potential for transforming and improving education. The use of advanced technology made it possible to scale up and reach massive amounts of users, potentially bringing (free) education within arm’s reach for anyone with an Internet connection. However, almost seven years later, MOOCs have only partially fulfilled their potential, as they manage to draw in large numbers of users, but also see the majority of them dropping out [1]. Based on an analysis of “565 course iterations from 261 different courses, with a combined 12.67 million course registrations from 5.63 million learners” [1, p. 130], it appears that the majority of people who enrol in a MOOC never enter the course (52% of the study sample), and those who do join, are mainly active during the first two weeks, after which their level of activities drops sharply [1]. We argue that a drop in activity levels may be partly due to a lack of user engagement, and MOOCs may benefit from a gamified intervention targeted at increasing engagement.

Inspired by social presence theory, we propose to stimulate engagement by making MOOC users aware of the presence of fellow students, thus, emphasizing the ‘social factor’ in MOOCs, in contrast with the rather individual-oriented approach typically followed so far. Our assumption is that by enabling users to perceive the (social) presence of their fellow students, a sense of community (SoC) can be generated, which may positively impact levels of engagement and learning performance. According to social presence theory [2], engagement can be enhanced in online learning environments by creating a sense of community and belonging among users [3]. However, as this theory postulates, to develop a SoC it is important that users perceive others as 'real persons’ in the shared online learning environment (i.e., the MOOC). Perceiving the presence of others online is not an inherent characteristic of MOOC platforms, where usually the only ‘social’ feature, the only ‘social affordanceFootnote 1’, available is the discussion forum [4]. Several studies have investigated how forums should be designed to foster user engagement [4, 5], but only few have considered to include and design solutions, for generating a SoC among MOOC users in order to raise engagement levels [6].

Taking inspiration from games and social presence theory, we have designed, developed, and implemented several game elements (which are described in Sect. 3 - experimental design) to address the issue of user engagement in MOOCs. In designing gamification, we have taken into consideration the characteristics of the scenario of application, the problem found in this scenario that we aim to address, and the target audience (in accordance with our previous work [7]).

  • Scenario of application: The MOOC platform for the current study, Open edX, lacks features that enable users to perceive their fellow students. The only space where people can interact within this platform is the discussion forum, and the perception of which and how many people are online is not immediate. Most participants are not aware of how many other users are following the same course.

    Implication for the gamification design: to facilitate a SoC, the shared online learning environment should facilitate sociability, which is the degree in which the online environment supports social affordances[8, p. 284]. In the shared social space, it is important to perceive the other as real, close. Proximity has been shown to play an important role in the development of a SoC [9]. One way to create “virtual proximity” in online learning scenarios, is to provide awareness information about group members, in our case MOOC fellow students. Furthermore as [10] reports, different types of interactions (student-instructor and student- student) are important, because together these “strengthen students’ sense of membership” [7, p. 153]. Therefore MOOC users will not only need to be aware of the other members, but also to interact with them.

  • Target audience: MOOC learners can be as heterogeneous as the general public. Interactive technology has to consider user characteristics, and more specifically their perceptions and predispositions. While some people easily perceive others as being present, others may require more explicit input to perceive the same level of presence.

    Implication for the gamification design: considering the heterogeneity of our target audience, the game elements need to be developed in such a way that each user determines her/his own level of social involvement, respecting and taking into account the individual needs and characteristics.

    • Problem to solve: we aim to address the lack of engagement and retention of users within an online open course.

      Implication for the gamification design is to address the lack of social features (affordances) in the Open edX platform, with the purpose of enhancing its sociability. By doing so, we aim to generate a feeling of others (social presence, in term of awareness and proximity) among MOOC users, which will lead to the generation of a SoC. In turn, this is expected to increase levels of participant engagement in the gamified (experimental) condition compared to the non-gamified (control) condition.

To present our gamification design and its effects, we have organised the remainder of this paper as follows: section two introduces our theoretical framework, based on social presence theory and SoC. The third section presents related works, mainly with respect to how engagement has so far been investigated in MOOCs. Section four details our research questions and hypothesis, then the experimental procedure is described, followed by the results, discussion and conclusions.

2 Theoretical Background

The term and theory of social presence has evolved over the recent years. It has been coined in the field of telecommunications by [2]. Initially social presence was defined as the “degree of salience of the other person in the interaction” [2, p. 65], using a communication medium. [2] conceptualised social presence as “a quality of a communication medium that can determine the way people interact and communicate” [8, p. 117]. According to [2] the degree of social presence can vary in relation to the medium used (i.e. videos have a higher level of social presence than audio). [12] shifted the attention from the medium to the person, defining social presence as “the degree to which a person is perceived as a real person in mediated communication” [9, p. 151]. Lastly with the Community of Inquiry (CoI) framework [13] the focus has passed from the person to the community.

To understand how social presence, sense of community (SoC) and engagement are linked, we can refer to the study of [3]. Results from this study suggest that social presence correlated with SoC and students with a stronger perception of SoC also felt more engaged [3]. Furthermore, “Online researchers emphasise social presence as a key factor in student engagement” [11, p. 3] and relate it positively to “students learning […] and student satisfaction” [11, p. 3]. Engagement in general is an abstract and multidimensional concept [14], and students’ engagement in particular, has been investigated and conceptualised in many ways across several disciplines [15,16,17]. From a technology enhanced learning perspective, the type of engagement we aim to study and foster is generated in online environments and for which social presence and SoC is needed. The latter is seen as an element of the social space that exists among participants and described in previous work [8, 18]. Therefore, in the framework of this study, engagement is studied as the degree in which the learners are involved in online activities and interact, communicate, with others (mediators and peers). Engagement, as such, is generated and influenced by the experienced presence of the others and social presence. In other words: “Engagement is composed of individual attitudes, thoughts, and behaviours as well as communication with others. Student engagement is about students putting time, energy, thought, effort, and, to some extent, feelings into their learning” [11, p. 147].

3 Related Work: Engagement in MOOCs

Engagement is a popular topic in the literature on MOOCs. Authors have described the construct via literature review [19], through theoretical frameworks, such as self-determination theory [20], learning analytics [21], and machine learning algorithms [22]. The studies conducted have identified, among other results, the type of users that engage in MOOCs [4]. Furthermore, it has been shown that teacher/instructor styles play an important role in engaging students [23], as well as videos [24], formative assessment, feedback practices [25], and time management solutions that support learners [26].

Gamification has been suggested as a potential strategy to stimulate user engagement in MOOCs [27, 28]. For designing a gamified solution we have investigated the ‘games’ literature, with the aim of understanding which factors retain millions of players within their online communities. Based on [29], the two factors that seem to retain players in a specific group or guild are: (1) SoC (membership, sense of belonging, group identity), which positively impacts retention and ‘relation switching cost.’ Such a cost in online games can be described as “the losses accompanied with the breaking of the bonds that have been formed with other gamers” [29]. A high SoC is accompanied by high relation switching costs, yielding players more likely to stay in the same group (or guild). (2) The second factor that retains players in a specific group is interdependence. “Interdependence is the degree to which members in a community rely on each other to make decisions and take actions [30]” [29]. In order to create a SoC and interdependence within MOOCs, it is vital to make the users aware of the others and generate social presence.

Although the importance of social presence in online learning settings has been well documented [7, 31,32,33], most research in the area of social presence is situated within the formal education context [6]. Only a few studies examine learner perceptions of social presence in MOOCs [6]. What we propose in this study, is not only a conceptual elaboration but also a technological solution that embraces the concept of social presence and SoC to enhance engagement of users within a MOOC learning environment. Lastly, in our previous work [33], we also theorize on the correlation between social presence and SoC, hypothesizing their impact on engagement and learning performance. This work is also an attempt to empirical verify this connection.

4 Research Question and Hypotheses

The main research question underlying our study is: Can a gamified solution help to increase MOOC user engagement and learning performance through mechanisms of social presence and SoC? We hypothesize that by enabling users to perceive the (social) presence of their fellow students through gamified solutions, a SoC can be generated, which positively impacts levels of engagement and learning performance, see Fig. 1. From this assumption, the following research hypotheses (H) are derived with regard to our experiment (detailed below):

Fig. 1.
figure 1

Representation of the connection among variables

Full size image
Fig. 2.
figure 2

Avatar selection interface (Exp view)

Full size image
Fig. 3.
figure 3

Clan name definition (Exp view)

Full size image
Fig. 4.
figure 4

Online Status (Exp view)

Full size image

H1: Our gamification design contributes to the feeling of social presence among MOOC users: perceived social presence will be higher for users in the experimental (gamified) vs. control (non-gamified) condition;

H2: Our gamification design contributes to the SoC among MOOC users: SoC will be higher for users in the experimental (gamified) vs. control (non-gamified) condition;

H3: Social presence and SoC are positively associated, i.e., higher levels of social presence are associated with higher levels of SoC;

H4: Our gamification design contributes to MOOC user engagement: engagement will be higher for users in the experimental (gamified) vs. control (non-gamified) condition;

H5: SoC is positively associated with user engagement, i.e., higher levels of SoC are associated with higher levels of engagement.

H6: Our gamification design contributes to MOOC user learning performance: test performance will be better for users in the experimental (gamified) vs. control (non-gamified) condition;

H7: Our gamification design contributes to MOOC user retention: retention will be higher, c.q. dropout will be lower and later for users in the experimental (gamified) vs. control (non-gamified) condition;

5 Method

Participants and Procedure

A total of 255 people enrolled in the MOOC, of which 154 were active. Participation in the MOOC as well as in the experiment was voluntary, and information about it was provided in the introductory video and via additional information. At the beginning of the course, participants’ background information and the consent to use their data were collected. 155 participants provided background data and informed consent. The average age of the participants was 43.4 (SD = 13.89), n = 53 had a master’s degree; n = 47 a bachelor’s degree, and n = 25 a high school diploma (the rest declared ‘other’). The majority (n = 108) of the participants were from NL, but also BE, IN, ES, PK, GB, FR and AU were represented.

Log data (H4, H6–7) were registered during the MOOC for 154 online users. In addition, one week after the MOOC had started, participants were asked to fill in a survey containing the social presence scale (SPS) and the SoC measure (H1–3, H5). The SPS was filled in by 45 (of 98, 45.9%) users in the experimental condition and by 19 (of 56, 33.9%) in the control, whereas for SoC, data were complete for 47 (48.0%) users in the experimental condition, and 23 (41.1%) in the control condition.

The Study Site

The MOOC under investigation was titled “How Cryptography keeps the Internet Secure”, at its first edition. It ran between January 2019 and February 2019, spanning four weeks in total. Each week had dedicated content and included a knowledge test. Video lectures, video scripts and lecture notes were available for all participants, and were released weekly. The MOOC was provided by the Open University of The Netherlands on Open edX platform.

Experimental Design

To test the above mentioned hypotheses, MOOC users were randomly assigned to an experimental (gamified) or control (non-gamified) study condition. The gamification design consisted of the game elements listed below (see Table 1), manipulated between groups with the purpose of targeting MOOC users’ feelings of social presence and SoC, which were expected to positively affect levels of engagement. Within the experimental group, users were further assigned randomly to one of two clans (Fig. 5).

Table 1. Overview of included game elements and experimental manipulations thereof.
Full size table
Fig. 5.
figure 5

Chat tool (Exp view)

Full size image

As mentioned in Table 1, the experimental group was asked to perform a number of challenges:

  • first week: (1) group identity challenge, participants used the group voting system needed to agree on their clan name, aim, logo, rules, and roles, (see Fig. 2, representing the group voting system used to define the clan name); (2) knowledge challenge (test), (in competition with the other clan);

  • second week: the (1) crypto-challenge, which consisted of sending an encrypted message to the other clan and decrypt the response back, and the (2) knowledge challenge (test);

  • third week: was a very information heavy week and we asked only for the knowledge challenge (test);

  • fourth week: (1) discussion challenge, where all groups (clans and control) were asked to challenge a peer or the teacher in a discussion and (2) the knowledge challenge (test).

Measures

Social presence was measured using a shorter version of [35]’s scale. The scale assesses social presence across the two dimensions of ‘awareness of the others’ and ‘proximity to the others’, with 4 and 12 items respectively, all rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree)Footnote 2 SoC was measured using the instrument of [32] consisting out of 10 items using 5-point Likert scales (1 = strongly disagree to 5 = strongly agree)Footnote 3 Engagement was assessed based on the log-data collected throughout the course, and included the amount of content page views, info page views, test page views, test submittals, and chat activities, assessed at week and total course level. Only those activities that were measured in both the experimental and control group were included for group comparisons. Higher levels of activity were assumed to reflect higher levels of engagement. Learning performance was operationalized as the percentage of correct answers on the weekly knowledge tests. A higher proportion of correct answers reflected better performance. Retention, cumulative dropout was assessed at week level in both groups, with users being defined as dropouts if they neither had any registered content page views, info page views, test page views, test submittals, or chat activities within that week, nor thereafter. In addition, the number of days to dropout were defined as the last day at which an online activity was registered for a certain user, calculated from course start (higher number of days reflecting later dropout).

Analyses

In order to test hypothesis H1 (social presence), and H2 (SoC) independent samples t-tests were performed. To test H3 (association between SoC and social presence), a regression analysis was performed with social presence as independent (predictor) variable, and SoC as dependent (outcome), as well as a Pearson correlation. Regression was also performed to test H5 (association between SoC and engagement).

H4 (engagement) was assessed by using independent samples t-tests to test whether the amount of content page views, info page views, test page views, test submittals, and total amount of online activities were significantly higher in the experimental versus control group, and χ2 to test whether the proportion of participants using chat functionalities was significantly higher in the experimental vs. control condition. H6 (learning performance) was assessed using χ2 to test whether the proportion of correct answers on the knowledge tests was significantly higher in the experimental vs. control condition. Lastly, χ2 and independent samples t-tests were performed to test, respectively, whether the proportion of participants dropping out was lower in the experimental vs. control condition, and whether the average number of days to dropout were higher in the experimental vs. control condition (H7; retention/dropout).

6 Results

H1: Perceived social presence will be higher for users in the experimental (gamified) vs. control (non-gamified) condition;

As Table 2 reports, the levels of social presence were significantly higher for users in the experimental group, compared to those in the control group thus confirming the hypothesis. Furthermore, the dimension of social presence in which the two groups differ significantly is ‘proximity with others’, which is higher in the experimental group compared to the control one.

Table 2. Comparison social presence and sense of community between control and experimental groups on the base of questionnaire data.
Full size table

H2: Perceived SoC will be higher for users in the experimental (gamified) vs. control (non-gamified) condition, as Table 2 shows, the level of SoC were significantly higher for users in the experimental group, compared to those in the control group, thus confirming H2.

H3: Higher levels of social presence are associated with higher levels of SoC: a regression analysis has been performed with SPS as independent (predictor) variable, and SoC as dependent (outcome) variable, the association is significant (F(2,61) = 44.79; β = .707, p < .001; R2 = .595), regardless of treatment or control conditions, thus social presence and SoC are significantly associated in both groups, H3 is confirmed. Furthermore the Pearson correlation performed with Rasch measures, reported in Table 3, shows a positive correlation between social presence and SoC only in the experimental group, see the first column of Table 3, supporting therefore H3.

Table 3. Correlation sense of community and social presence- experimental and control groups (Rasch measures)
Full size table

H4: Engagement will be higher for users in the experimental (gamified) vs. control (non-gamified) condition: see Table 4. Users in the experimental and control group did not significantly differ with regard to the total amount of registered content page views, test page views, test submittals, and total number of online activities at course level.

Table 4. Comparison control and experimental groups on the base of log-data
Full size table

However, more info page views were registered for users in the experimental group, and a significantly larger proportion of users in the experimental condition used chat functionalities compared to the control condition. Also, the data suggest that activity for users in the control group dropped more steeply after course start, compared to that of users in the experimental group, with a trend-significant difference in the last week of the course (t(152) = −1.573, p = .069). The two clans in the experimental group did not significantly differ on any of the engagement measures. Taken together, H4 was only partially confirmed.

H5: Higher levels of SoC are associated with higher levels of engagement: no association was found between SoC scores and online activities (F(1,68) = .382; β = .075, p = .539; R2 = .006; adjusted for treatment condition: F(2,67) = .749; β = .032, p = .805; R2 = .022). H5 was not confirmed.

H6: Test performance will be better for users in the experimental (gamified) vs. control (non-gamified) condition: as Table 4 reports, the proportion of correct answers on the knowledge tests was significantly higher in the experimental group compared to the control group, thus confirming H6. The two clans in the experimental group performed equally well on the knowledge tests (clan 1: 77.3% correct answers; clan 2: 78.7% correct answers; χ2 (1) = .237, p = .627).

H7: Dropout will be lower and later for users in the experimental (gamified) vs. control (non-gamified) condition: see Table 4. Although the data suggested differences in rate and speed of dropout in favour of the experimental condition, these differences were not statistically significant, thus not supporting H7.

7 Discussion and Conclusion

This study investigated the effects of a gamified intervention targeting user engagement in MOOCs through mechanisms of social presence and SoC. To this end, seven game elements were designed and implemented in the MOOC embedded in the platform Open edX. Using an experimental design, users were randomly assigned to a gamified or non-gamified condition (experimental vs. control group).

The data showed that the game elements did trigger social presence and SoC among MOOC users, however, the expected association between these measures and user engagement was not confirmed. More specifically, seven hypotheses were tested, of which four were confirmed (H1-3, and H6), one was only partially confirmed (H4), and the remaining two (H5 and H7) were not supported by our data. From this we conclude that our gamification intervention had an effect on (i) users’ learning performance (H6): MOOC participants accomplished significantly better results compared to participants in the control condition; (ii) the development of feelings of social presence (H1), in particular for the proximity dimension; and (iii) development of a SoC (H2). Furthermore, social presence and SoC were associated (H3), particularly in the experimental group, however, we did not observe a direct association between these feelings and user engagement (H5). As far as engagement is concerned, users in the experimental condition showed to be significantly more engaged compared to the control group in the usage of the chat tool and in the view of the info pages. This enables us to only partially confirm H4. Moreover, users in the experimental group seemed to have a higher level of retention compared to their colleagues in the control group, showing a less pronounced decline in activities as a function of course duration compared to what was typically reported in previous work [1]. However, possibly due to the scarce number of participants involved, this apparent difference was not statistically significant, therefore H7 was not confirmed. Results from this study should be viewed in the light of several limitations. First of all, the sample size was limited, thus the possibility of null findings representing ‘false negatives’ cannot be excluded. Secondly, although log-data were collected for the complete sample, only a selection of users provided questionnaire data, thereby potentially biasing results. Lastly, we cannot ascertain that the implementation of our game elements in the online environment was without technical problems for all users. Further studies are therefore warranted, in which these issues are addressed.

Despite these limitations, this study represents a step forward for the field of gamification of MOOCs. It enriches the gamification field by introducing a technological solution that embraces theories known in the field but never applied to gamification before. Furthermore, it shows a more technologically advanced way towards designing and implementing gamification within MOOCs, taking into consideration the application scenario, the target audience and what is actually done in the game world. Also, this study enhances MOOCs: MOOC platforms, in general, do not allow to seize upon this potential, the “social” aspect seems totally left aside. In Open edX, in particular, it is hard to understand that there are other users online in that same course: if a user is keen to be involved in some kind of social interaction, s/he has to hunt for the discussion forum. Our solution instead enables MOOC users to be aware of the others without the need of taking any action, by simply being online.

In conclusion our data show that the game elements designed to produce social presence and SoC among MOOC users were proven to successfully fulfill their purpose.