Keywords

1 Introduction

In the last decade, there is a growing body of literature [1,2,3] that seeks to develop a descriptive framework to capture teaching and learning activities so that teaching ideas can be shared and reused from one educator to another, so called Learning Design (LD) [4]. While the early work in LD has focused on transferring the design for learning from implicit to explicit, the relationship between LD and the actual learner response has been not fully understood. As the majority of feedback takes forms of assessments, and course’s evaluations, which typically takes place after the learning process has finished (except for formative assessments), it prevents teachers from making in-time interventions. Recently, the advancement in technology has allowed us to capture the digital footprints of learning activities from Virtual Learning Environment (VLE). This rich and fine-grained data about the actual learners’ behaviors offer educators potentially valuable insights on how students react to different LDs.

Learning analytics (LA) has the potential to empower teachers and students by identifying patterns and trends from a wide variety of learners’ data. Substantial progress has been made both in conceptual development [5, 6] as well as how to design appropriate predictive learning analytics to support students [7, 8]. Nonetheless, in line with [7, 9], findings from LA research in the past have been rather limited to delivering actionable feedback, while ignoring the context in which the learning data is situated. Thus, there is an increasing interest to align LA with LD, as the former facilitates the transfer of tacit educational practice to an explicit rendition, while the latter provides educators with pedagogical context for interpreting and translating LA findings to direct interventions [10,11,12,13,14]. While there are abundant discussions on the value and impact of integrating LD into LA to improve teacher inquiry [13, 14], only a few studies have empirically examined how teachers actually design their courses [15, 16] and whether LD influences satisfaction, VLE behavior, and retention [9, 17,18,19].

This study builds on previous work by [17, 19, 20] by dynamically investigating the use of learning design in 24 modules over 30 weeks at one of the largest distance higher education institutions in Europe using a combination of data visualizations and social network analysis. Our work contributes to the existing literature by capturing: (1) how learning activities interact with each other across modules, and (2) how teachers configure their course at activity level.

2 Learning Design at the Open University

2.1 Aligning Learning Analytics and Learning Design

In the last five years, LA has attracted a lot of attention from practitioners, management, and researchers in education by shedding light on a massive amount of (potentially) valuable data in education, as well as providing means to explicitly test existing pedagogical theories. Scholars in the field of LA have exploited various sources of data, such as activity logs of students [21], learning dispositions [22,23,24], or discussion forum [25]. While these studies provide important markers on the potential of LA in education, critics have indicated a gap between pedagogy and LA [26, 27]. Interesting patterns can be identified from student activities, such as number of clicks, discussion posts, or essays. However, these patterns alone are not sufficient to offer feedback that teachers can put into actions [8, 24]. Without a pedagogically sound approach to data, LA researchers may struggle with deciding which variables to attend to, how to generalize the results to other contexts, and how to translate their findings to actions [27]. Hence, LD can equip researchers with a narrative behind their numbers, and convert trends of data into meaningful understandings and opportunities to make sensible interventions.

Since the beginning of the 21st century, the term learning design has emerged as a “methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies” [1]. For more discussion on the origins of ‘learning design’ and ‘instructional design’, we refer readers to Persico, Pozzi [12]. Several approaches for designing learning have been proposed, yet, one common stage in almost every approach was the evaluation of the LD [12]. Persico, Pozzi [12] argued that the learning process should not only depend on experience, or best practices of colleagues but also pre-existing aggregated data on students’ engagement, progression, and achievement. In a similar manner, Mor et al. [13] suggested that LA could facilitate teacher inquiry by transforming knowledge from tacit to explicit, and perceive students and teachers as participants of a reflective practice. For instance, in a study of 148 learning designs by Toetenel, Rienties [28], the introduction of a systematic LD initiative consisting of visualization of initial LDs and workshops helped educators to focus on the development of a range of skills and more balanced LDs. Feeding information on how students are engaged in a certain LD during or post-implementation can provide a more holistic perspective of the impact of learning activities [10].

Several conceptual frameworks aiming at connecting LA with LD have been proposed. For example, Persico, Pozzi [12] discussed three dimensions of LD that can be informed by LA: representations, tools, and approaches. Lockyer et al. [10] introduced two categories of analytics applications: checkpoint analytics to determine whether students have met the prerequisites for learning by assessing relevant learning resources, and process analytics to capture how learners are carrying out their tasks. In the recent LAK conference 2016, Bakharia et al. [14] proposed four types of analytics (temporal, tool specific, cohort, and comparative), and contingency and intervention support tools with the teacher playing a central role.

In this paper, we will use the conceptual framework developed by Conole [1] and further employed by Rienties, Toetenel [17]. Both conceptual and empirical research has found that the Open University Learning Design Initiative (OULDI) can accurately and reliably determine how teachers design courses, and how students are subsequently using these LDs [17, 19].

While there were numerous discussions in aligning LA with LD, the amount of empirical studies on the subject has been rather limited. For example, Gašević et al. [8] examined the extent to which instructional conditions influence the prediction of academic success in nine undergraduate courses offered in a blended learning model. The results suggested that it is imperative for LA to taking into account instructional conditions across disciplines and courses to avoid over-estimation or underestimation of the effect of LMS behavior on academic success. From our observation, most of the empirical studies attempting to connect LA and LD are derived from students activities [10], or differences in discipline [8], rather than how teachers actually design their course [29].

In our previous work, we have highlighted explicitly the role of LD in explaining LMS behavior, student satisfaction, retention, and differences in prediction of academic success [8, 9, 17,18,19]. For example, in our first study linking 40 LDs with VLE behavior and retention, we found that strongly assimilative designs (i.e., lots of passive reading and watching of materials) were negatively correlated with retention [18]. In a large-scale follow-up study using a larger sample of 151 modules and multiple regression analyses of 111,256 students at the Open University, UK, Rienties, Toetenel [17] revealed relations between LD activities and VLE behavior, student satisfaction, and retention. The findings showed that taking the context of LD into account could increase the predictive power by 10–20%. Furthermore, from a practitioner’s perspective, the combination of a collaborative, networked approach at the initial design stage, augmented with visualizations, changed the way educators design their courses [28].

While these three studies at the Open University UK (OU) highlighted the potential affordances of marrying LD with LA on a large scale, two obvious limitations of these studies were the aggregation of learning activities in predicting behavior and performance (i.e., rather than their interaction), as well as the static rather than longitudinal perspective of LD. In these studies [9, 18], aggregate learning design data across the 40 weeks of each module were used, while in many instances teachers use different combinations of learning activities throughout the module [29]. To address this, in our recent study [20], we have dynamically investigated longitudinal learning design of 38 modules over 30 weeks and found that learning design could explain up to 60% of the students’ time spent on VLE. While learning design at weekly level has revealed promising results, the design of individual learning tasks has not been examined due to the lack of data. Therefore, this study takes a further step by looking at the learning designs and the inter-relationships between learning activities at individual task level.

2.2 Research Questions

Our previous works have shown a diverse of learning designs across different disciplines over time. In this study, we take a further step by looking at the learning design at activity level.

  • RQ1: How are different types of learning activities connected, both within the module and between modules?

  • RQ2: What media were used to deliver the individual learning activities?

3 Methodology

3.1 Study Context

This study took place at the Open University UK, which is the largest distance education provider in Europe. Data in this study was generated from the OU Learning design initiative, which helps teams in defining their pedagogic approach, choosing and integrating an effective range of media and technologies, and enable sharing of good practice across the university [30]. When using data to compare module design across disciplines and modules, according to our previous work [17, 19] it is important to classify learning activities in an objective and consistent manner. In particular, each module goes through a mapping process by a module team which consists of a LD specialist, a LD manager, and faculty members. This process typically takes between 1 and 3 days for a single module, depending on the number of credits, structure, and quantity of learning resources. First, the learning outcomes specified by the module team were captured by a LD specialist. Each learning activity within the module’s weeks, topics, or blocks was categorized under the LD taxonomy and stored in an ‘activity planner’ – a planning and design tool supporting the development, analysis, and sharing of learning designs. Next, the LD team manager reviews the resulting module map before the findings are forwarded to the faculty. This provides academics with an opportunity to give feedback on the data before the status of the design was finalized. To sum up, the mapping process is reviewed by at least three people to ensure the reliability and robustness of the data relating to a learning design. Even so, coding learning activities remains a subjective undertaking and efforts to increase the validity in coding instruments have to date resulted in lack of data which can provide context to analysis (Table 1).

3.2 Measurement of Learning Design

Seven categories of learning activities were measured in terms of workload, which is the number of hours that was allocated for each type of learning activities. Time spent on learning activities was restricted based on the size of the module, such as 30 credits equated to 300 h of learning, and 60 credits equated to 600 h of learning. However, the actual LD depends on individual teacher. Descriptive statistics of the seven types of learning activities can be found in Table 2, Appendix.

In addition, assimilative activities of five modules were decomposed into sub-categories such as: Words, Figures, Photos, Tables, Equations, Audios, Videos, and Others (Fig. 1). These represent different channels that students absorbed information. Due to limited space, we chose to report the results of an exemplar module in Social sciences. In this exemplar module, there were in total 267 individual learning activities that were decomposed, descriptive statistics can be found in Table 3, Appendix.

Fig. 1.
figure 1

Workload tool at individual task level of an exemplar module in the Social sciences

Full size image
Table 1. Learning design taxonomy
Full size table

3.3 Data Analysis

Prior studies of Social Network Analysis (SNA) in e-learning, particularly in the improvement of LD have concentrated on examining patterns of learner communication and collaboration in various situations, such as when discussing, blogging and e-mailing [31]. Within the last three years in LA, SNA has been shown to be an effective tool to explore the relationships of learners in online discussion forum [29], as well as in face-to-face interactions, tracked for instance in eye tracking movements [32]. However, none has looked at the LD from a social network perspective, identifying connections between learning activities using ‘big data’. Hora, Ferrare [29] suggested that teaching practice should be best viewed as situated in and distributed among features of particular settings. According to the systems-of-practice theory by Halverson [33], local practices are informed, constrained, and constituted by the dynamic interplay of artifact and tasks. Thus, in order to understand how teachers design their course, it is necessary to consider the inter-relationships among different learning activities, which is why we have employed a social networking analysis approach.

We used Tableau 10.1 to visualize the LD of 24 modules over 30 weeks of study time, and social network analysis (UCINET 6.627) to visualize the inter-relationships among learning activities. The LD dataset was a weighted two-mode network as it consisted of different learning activities across several weeks. Since we are primarily interested in the relationships among learning activities, the dataset was transformed to a one-mode network. We refer readers to our previous work [20] for more details of the data transformation process.

4 Results and Discussion

4.1 Learning Design Within and Between Modules

The average time allocated for different types of learning activities per week of 24 modules were illustrated in Fig. 2. At a glance, there were a lot of fluctuations in the time allocated for each type of learning activities over time, which implies a dynamic usage of LD from educators. This is an interesting finding in itself, as it demonstrates that the format of learning changes on a weekly basis, rather than following an identical format week after week.

Fig. 2.
figure 2

Learning design of 24 modules over 32 weeks

Full size image

In line with our previous findings [17, 19, 20], assimilative activities accounted for the majority of study time (M = 3.49, SD = 3.29), followed by assessment activities (M = 1.64, SD = 2.80), and productive activities (M = 1.16, SD = 1.49). Other types of learning activities such as communication, experiential, finding information, and interactive activities were underused on average. Assimilative activities and assessment activities seemed to follow opposite path, suggesting that where educators provide a lot of content, they do not provide assessment tasks and vice versa. In the beginning of a module, more assimilative activities were used to disseminate information to students whereas more assessment activities were used in the end of the module. The initial peak in assessment activities at the beginning of the module (week 5) suggests that many educators include an early assessment task to identify students that require additional support. Further correlational analysis (not included) suggested that assessment activities were negatively correlated with assimilative, information, communication, and productive activities. It was also interesting to see that many educators also included substantive time for productive tasks throughout their module, but the time allocation stops when students are due to prepare for their final assessment. Figure 2 shows, in line with our previous work [20], that educators had the tendency to reduce other learning activities when introducing assessments, in order to remain a balanced workload.

After capturing the dynamic picture of LD over time, we took a further step to examine how different learning activities are configured within each module. Due to the limited space, we only reported the LD of an exemplar module (60 credits) in Social sciences throughout the rest of the analysis. This module was selected based on the availability of LD data at individual task level, e.g. the time educators expected learners to spend on activities was mapped by minute on a weekly basis. A close look at the LD within modules (Fig. 3) revealed a combination of assimilative, assessment, productive, and finding information activities that were used. Similar to the overall trends shown in Fig. 2, this exemplar module allocated the majority of study time for assimilative activities (M = 3.29, SD = 2.38) with six formative assessments during the learning process and a final exam at the end.

Fig. 3.
figure 3

Learning design and learning engagement of an exemplar module in Social sciences

Full size image

Our social network analysis demonstrated the inter-relationships between different types of learning activities used in the exemplar module (Fig. 4). The network density was 14.3%, with 6 ties in total, and the average distance between pairs was 1. Productive and assimilative activities were strongly connected, implying that in this module educators combined productive and assimilative in the weekly LD. Assimilative activities had strong influences on both productive and finding information activities with the weight of 71.0 and 13.5 respectively. On the other hand, there was no connection between assessment activities and others, despite of its high frequency in the overall LD. This suggests that educators excluded other learning activities when introducing assessments, allowing learners to focus on their assessment task.

Fig. 4.
figure 4

Inter-relationships between learning activities of an exemplar module in Social sciences

Full size image

To sum up, our analysis at module level indicated a wide variety of learning design between and within modules. SNA analysis of the exemplar module indicated the strong influence of assimilative activities in both workload and in relations with other learning activities. In the next step, we will consider the media types that are used in assimilative activities, which provides a rich picture of the media mix used in a particular Learning Design. This is important as it is likely that not only the activity type, for instance assimilative in this case, bears a relation on satisfaction and engagement of students, but also the way in which the activity is delivered. We decompose assimilative activities of this exemplar module at individual task level to unravel how the LD was configured within each learning activity.

4.2 Learning Design at Individual Task Level

When coding learning activities, media assets are indicated at a high level, in order to compare the overall amount of time spent on video, words, photos and figures for instance. We accept that this high level notation does not indicate whether a module includes one video of half an hour or six videos of five minutes, as the total time spent per item is recorded. The decomposition of assimilative activities of the exemplar module was illustrated in Fig. 5. On average, the majority of assimilative activities took forms of words (M = 3.32, SD = 1.92). This suggests that educators were more likely to use reading materials to convey information, but most weeks also included another media element. Figure 5 also shows that figures and videos were also used overtime, but in less frequency compared to words.

Fig. 5.
figure 5

Assimilative activities of an exemplar module in Social sciences.

Full size image

Further SNA analysis demonstrates the inter-relationships between different types of assimilative activities and other learning activities. There were in total 40 ties in the network, with the density of 22% and the average distance between a pair of ties of 2.036. Firstly, there were strong connections between the use of words with photos, tables, and figures. These forms of assimilative activities often appeared together in reading materials. In line with the multi-media principle of Mayer [34], this module employed an integrated representation of graphics and words. Given the nature of this module, most of the graphics were representational (visuals that illustrate the appearance of an object), organizational (visuals that show qualitative relationships among content), and interpretive (visuals that make intangible phenomena visible and concrete) [34]. The use of words had a strong influence on photos, figures, and tables with the weight of 38.9, 16.4, 38.4 respectively (out-degree centrality = 118.541) (Fig. 6).

Fig. 6.
figure 6

Inter-relationships between assimilative activities and other activities of an exemplar module in Social sciences. Note: Blue nodes represent assimilative activities, red nodes represent other activities (Color figure online)

Full size image

Secondly, videos were often used in combination with finding information activities and productive activities. For example, students were asked to watch a short video, and answer some questions using the information from the video. Alternatively, students were asked to interpret and draw conclusion using the information from the video.

The structure of the network also revealed interesting findings. There are two local networks in which the first one (right hand side) consists of mainly assimilative activities, whereas the second one (left hand side) consists of some assimilative activities (i.e. videos, others), finding information, and productive activities. The connection between words and productive activities acted as a bridge between these two local networks. The betweenness centrality of the edge productive-words was 28, which means there were 28 flows between all pairs of nodes which were carried using this edge.

5 Conclusion

This study examined the learning design of 24 distance learning modules over 30 weeks at the Open University UK using a combination of visualizations and social network analysis. Our first finding at module level suggested that LD as employed at the OU is dynamic and varies on a weekly basis for the modules investigated. This may be surprising as modules are often follow the same high level pattern, but the individual learning activities show a different visualization. In line with our previous findings [17, 19, 20], assimilative activities accounted for the majority of study time, followed by assessment activities, and productive activities. Assimilative activities and assessment activities seemed to follow opposite paths, suggesting that educators strategically reduced the time allocated for other learning activities when introducing assessments. Given the majority of OU’s students were having either a full-time, or part-time job, ensuring a balance learning design is vital for students who are sensitive to sudden changes in the workload.

Our second finding from the analysis on an example module in social sciences revealed interesting pattern of learning activities. While assimilative, productive, finding information, and assessment activities were used frequently throughout the module, there were no communication, interactive, or experiential activities. Our SNA indicated strong connections between assimilative, productive, and finding information activities. The data exposure provides educators with explicit feedback on their learning design, allowing them to reflect on current practices and predict potential problems. For instance, educators can consider introducing more communication activities, which have been shown in our previous work [20] to increase students’ engagement.

Thirdly, our analysis on 268 individual learning activities demonstrated the usage and connections of media in assimilative activities. In general, most assimilative activities took forms of words. This suggests that educators were more likely to use reading materials to convey information, but also included another media elements. Further SNA analysis revealed strong ties between words, figures, photos, and tables. This implies that educators employed an integrated representations of words and graphics, which has been shown to be effective in helping learners absorb information [34].

By capturing the pedagogical context, researchers in LA can go beyond the traditional process (trace data) – output (performance) model by incorporating the input (learning design). This will not only strengthen the predictive power but also empower educators to better translate LA findings into direct interventions.