Introduction

There is much interest in e-learning in higher education. Often virtual learning environments (VLE) such as Edmodo, Google Class, Moodle or BlackBoard are used to share learning materials with students and to provide access to a variety of resources and tasks for collaborative learning; some systems also have Apps that support mobile learning. The technologies in principle make learning possible that is framed by current theories about how people learn (NRC 2000). For example, computer-based learning tools can be used to elicit students’ prior knowledge, enable students to develop conceptual frameworks that render content meaningful, and support reflection; notions of computers as cognitive and metacognitive tools—i.e., tools that extend students’ cognitive abilities—have been noted in the literature (Azevedo 2005; Kim and Reeves 2007; Lajoie 2000). Collaborative interactions in the e-learning environments also support learning, and the widespread use of these has given rise to reconsideration of psychological theories that have previously been the domain of individual learners, for example from self-regulated learning to more social forms of regulation (Hadwin and Oshige 2011).

However, many studies (Bates 2012; Rodriguez 2013) and informal observation have shown that in practice instructional models focusing on individual learners and the delivery of learning content remain very common, rather than models that emphasize student interaction with content and peers and knowledge construction to develop insight into what is to be learned. One aspect of the problem is epistemological—many lecturers continue to believe that instruction is the most effective way to teach, and that educational technology is a non-essential add-on to the learning environment. But a lack of sustained professional development by which lecturers develop new practices that take advantage of current theories of learning and the affordances of the technologies for greater interaction also is a contributing factor. Research shows that in many courses the use of the VLE is limited to sharing instructional materials such as presentation files, reading material, and links to websites and videos for use by individual students (Mahdizadeh et al. 2008; Christie and Jurado 2009). In an exploratory case study of four e-courses randomly selected from 43 taught at a university in Tanzania we found a similar state of affairs (Msonde and Van Aalst 2014). For example, all four instructors made teaching notes and presentation files available to the e-learning platform, and their training advised them to upload lecture notes and PowerPoint slides, which are non-interactive learning resources. However, the students expected more social and constructivist learning from the use of the technology. Such expectations are in line with how educational technologists view educational technology as supporting social and constructivist models of learning.

These problems are internationally widespread, but the context we focus on is undergraduate chemistry education in Tanzania. As other African countries have, Tanzania has experienced macro-economic growth (higher investment and increases in productivity) for many years (Robinson et al. 2011), but it also has large population growth and physical infrastructure development in higher education (HE) institutions that have not kept pace (Luhanga 2010). Developing expertise in science, technology and innovation (STI) is seen as critical for enhancing economic development, but the existing physical infrastructure does not enable HE institutions to build their STI capacities and skills base. Consequently, there has been large investment in e-learning in Tanzania and other East-African countries to improve access to higher education (Nkembo et al. 2011; Simonson et al. 2009; Swarts and Wachira 2010), and for developing students scientific knowledge and expertise. However, the learning effects so far have been disappointing (Pejovic et al. 2012). It is important in this context to develop e-learning approaches that promote metacognition, which is important for enhancing learning (NRC 2000). In contemporary learning metacognition can be supported by technology (Azevedo 2005). For example, students may communicate using social learning tools to share their thoughts and ideas deeply during the learning process. There is a need to develop the student metacognitive learning in higher education to orchestrate their reflective ability and active monitoring of one’s own thought.

This study had two goals. The first goal was to investigate the higher order thinking (cognitive and metacognitive) effects of three types of e-learning designs in Moodle, a VLE used extensity in Tanzania and around the world. As will be described in more detail later, the three designs varied in the degree to which they supported peer interaction. The first design can be understood as fitting instructionist theories of learning and did not support peer interaction acting as a control in the study. The second design added support for peer interaction via asynchronous discussion forums, and the third via podcasts and asynchronous discussion forums; both of these designs can be understood as fitting social constructivist theories of learning. The three designs are simple in terms of bandwidth requirements compared with the range of possibilities that VLEs offer; this choice was made with scalability in mind, especially in a geographical region with limited bandwidth and in which university lecturers have limited experience with teaching that is informed by social constructivist teaching. The second goal was to investigate the effects of iteration and pedagogical improvement in each design. In this respect, the study is a novel example of design research, which typically examines the development of only one design.

Background

State and effectiveness of e-learning

The dominant mode of course delivery in higher education science courses is the lecture-based course, in which the lecturer explains key concepts, works through sample problems, and so on. At its worst, this method is underpinned by a simplistic knowledge sharing epistemology, according to which students can add new information to their existing knowledge system; at its best, it is underpinned by key experimental findings from cognitive science. However, much research has concluded that lecturing results in limited engagement and cognitive benefit (Hake 1998; McDermott 1991; Taylor et al. 2002). There is widespread consensus that what matters most for learning is what students do themselves, and not what the lecturer does (Nilson 2013). In terms of the key research findings on how people learn (NRC 2000), lecturing may help students organize facts to some extent, but it often fails to elicit and build on prior knowledge and fails to develop metacognition and self-regulation. Various problems with lecturing have been cited as reasons why undergraduate students leave the sciences (Seymour and Hewitt 1997; Tobias 1990). The recently developed ICAP framework proposes a cognitive engagement hierarchy from passive engagement to engagement that involves the construction of representations that go beyond the content that is presented and peer interactions about these representations (Chi and Wylie 2014).

Unfortunately, many e-learning courses remain in the instructivist paradigm. Lecturers use the VLE to share reading materials, presentation files, and even links to websites and videos that are used during lectures, without requiring deep processing of the content beyond what is accomplished during the lectures. The crux is lecturers’ limited understanding of learning processes and their competence in carrying out other methods. Perhaps not surprisingly, meta-analyses of research and a recent comparative study of computer use in OECD countries found only small effects of ICT on learning (OECD 2015).

The major alternatives to knowledge sharing emphasize knowledge construction and developing insight into one’s learning process and achievements—i.e., constructivism and social constructivism. These perspectives are known well by educational technologists and do not need to be elaborated here. Suffice it to say that they require learners to be cognitively more active in evaluating their current knowledge, revising it by taking new experiences and information into account, and taking metacognitive actions such as planning, monitoring and reflection on their learning. Collaboration can support these processes. Educational technology provides a wide variety of tools that can support knowledge construction, either solo or collaboratively.

In this study, we were interested in modest departures from the lecture-based course that would be feasible in large-enrollment courses in Tanzania, so we focused on online discussions as opposed to other collaborative activities such as collaborative concept-mapping or developing wikis or other artifacts that represent co-constructed knowledge. We believe this is the right place to start for improving e-learning in our research context. We implemented three designs: Non-Interactive Learning (NIL), which involved making learning resources available to students; medium interactive learning (MIL), which added asynchronous discussions; and High Interactive Learning (HIL), which also added podcasts to further stimulate the online discussions. The term “interactive learning” may be interpreted differently by people in different contexts. In this study, interactive learning implied by the MIL and HIL designs refers to learning which involves multidirectional communications among individuals and content. As McMillan (2002) contended, interactive learning is associated with communications between students and their instructors as well as interactions with contents. We hypothesized that NIL < MIL < HIL in terms of the development of cognitive and metacognitive effects. Therefore, this study investigated the following research questions:

  1. 1.

    To what extent do the three designed asynchronous learning environments and pedagogical approaches enhance student interaction and learning engagement?

  2. 2.

    Do the designs of the asynchronous learning environments and pedagogical approaches improve learning and higher-order thinking?

The effectiveness of the three designed asynchronous learning environments was analyzed to determine whether they promoted student learning, as measured by a chemistry test relevant to the course objectives.

Conceptual framework

Ideas about learning have been shifting from lecturing and individual construction of knowledge toward social and collaborative learning (Bereiter and Scardamalia 2002). The theoretical and pedagogical assumptions of collaborative approaches rely on the concept that student thinking reaches saturation in a social learning context (Mercer 1994). Thus, social learning is a communicative process through which knowledge is constructed in an inquiry stance and shared among members in a learning community. Institutions of higher education have traditionally been viewed as communities of scholars, with inquiry being a crucial method for students to explore and develop ideas. Learners within a community of inquiry have the opportunity to articulate and consolidate a repertoire of ideas into coherent meaning (Garrison and Arbaugh 2007; Linn and Eylon 2011).

The literature articulates that the cognitive processes necessary for students to develop deep learning occur in dialogue (Woo and Reeves 2007). The emergence of social learning tools has enabled instructors to create social learning space essential for these dialogues to occur. They afford instructor to sustain social discourses in which learners have the opportunity to explore and exchange ideas on a deeper level. Tools such as discussion forums and podcasts have the capability to foster knowledge sharing and collaboration (Rodrigues et al. 2011), and provide deeper learning. For example, Anderson (2008) demonstrated that enhanced online discussions creates social presence, which enables students to support or challenge each other and engages them in active social negotiation of meaning. Thus, online discussion forums are potent learning tools that promote social interaction and foster the construction of new knowledge and critical thinking.

Podcasts are one of many Web 2.0 technologies that have potential benefits for education. In this study, the use of podcasts was not only intended to stimulate (Fernandez et al. 2009) and motivate students to learn (Hill and Nelson 2011), but also to promote student thinking (Traphagan et al. 2010). The repetitive and controlled playback of complex segments of information, and the actions of students to bring what they learn from podcasts to the learning community may enhance their thinking.

Although there is general agreement in the literature that appropriate use of technology in teaching is likely to have positive effects on student learning (Tamim et al. 2011), there is no clear evidence of their direct effect on student achievement. Most previous research (e.g. Bassili 2008; Hew 2009; Ng’ambi and Lombe 2012) lacks empirical evidence to verify whether the specific use of technology has a significant effect on student thinking and achievement. Pressing questions thus remain whether the use of technology and associated pedagogical approaches create meaningful student interaction and higher-order thinking, particularly in countries with low-bandwidth, where resources are also limited. We define the term higher order thinking as all intellectual tasks that challenge learners to analyze, synthesize, interpret, and manipulate information beyond memorization. In other words, higher-order thinking involves learners own thought in exploring and generating ideas, monitoring learning progress, and making inferences beyond what is explicitly presented. Thus, given the physical distances and limited studying conditions, having high-quality e-learning designs is very important in providing students with meaningful learning experiences.

Pedagogical designs

We designed three asynchronous learning approaches that evolved via an iterative design. They were later implemented and evaluated in an instructional unit using three cycles of educational design research (McKenney and Reeves 2012). The overall design of the study is illustrated in Fig. 1. The most typical design found by Msonde and Van Aalst (2014) was used as a control, with the additional two designs aimed at promoting student interaction and engagement with course content. The control condition (NIL) provided students with opportunities to gain access to and learn from a range of non-interactive resources (Clarke and Fournillier 2012) such as slide presentations, notes, and links to Web pages performed at their own pace. The pedagogical orientation used was traditionally based on an instructor-centred approach to teaching. However, such a pedagogical approach did not harness the learning capabilities of the Internet and interactive learning tools. Our key aim was to examine whether an instructivist mode of teaching would have a positive impact on student learning.

Fig. 1
figure 1

Source Adapted from McKenney and Reeves (2012)

Overall research design.

Full size image

In the first intervention, an MIL design, we added discussion forums to the resources and methods used in the NIL in order to create a student-centred learning environment that would involve them actively in learning. There is much evidence suggesting that engagement with discussion forums is not high, that students do not necessarily respond to the ideas of others, and that these discussions end prematurely (Hewitt 2005; Wise et al. 2012), but pedagogical strategies are also available for boosting participation (Hew and Cheung 2008). Therefore, the researchers asked students to discuss a given topic, read the views of others and give comments on ideas written, or compose thoughtful questions and or answers in order to improve ideas. The use of online discussion forums was intended to put social constructivist methodology into practice by deepening the level of students’ inquiry through critical questioning.

In the second intervention, involving highly interactive learning (HIL), we also used podcasts. These can be viewed repeatedly, so students can replay sections they have not heard clearly or understood the first time (Kay and Kletskin 2012; Griffin et al. 2009); as a result, students may have more questions and ideas that they bring to online discussions. We selected 17 suitable podcasts with high-quality content, and organized them into relevant episodes. The length of each podcast ranged from 7 to 12 min, an optimal design based on social cognitive theory, which emphasizes using instructional strategies such as podcasts in chunks, to reduce the extraneous cognitive load (Kay and Kletskin 2012). The researchers and the instructor asked students to watch the video podcast or listen to the audio podcast carefully. They also required students to deduce major concepts being discussed in the podcasts and write a brief summary to express their understanding, and share those ideas to online discussions for further improvement. These pedagogical strategies focused on encouraging students to take ownership of their learning. Instructors working in developing countries, where bandwidth and the opportunity to learn advanced pedagogical methods for e-learning are both limited, would find it useful to know whether coupling online discussions and podcasts can enhance the learning of their students.

Methods

Setting and participants

The study was conducted at a major university selected from eight public universities in Tanzania. This university had a well-developed technological infrastructure with a local area network connected to student residence halls and teaching areas, and every academic department had at least one computer laboratory. Three instructors were asked to participate in the study, and one, who taught freshman organic chemistry, accepted; the curriculum resembled that of similar courses in higher education in the UK. All 102 students in the B.Sc. with Education program who were registered for this course agreed to participate in the study. The students were assigned randomly to three tutorial sections. The use of discussion forums and podcasts were not common, so it was necessary for the students and the instructor to develop some expertise in using them effectively.

Procedures

All three tutorial sections used the NIL design for the first topic to lay a baseline and evaluate the equivalence of the three sections on the most relevant variables. For the three subsequent topics, each tutorial section used one design (i.e., NIL, MIL, or HIL). We considered other research designs, in which students in all sections would experience more than one design, but it was realized that the instructor and students needed more time (more than one topic) to learn and develop their practices with a specific design. It was seen that the first iteration alone would not be sufficient to evaluate the effectiveness of the designs, therefore we settled on the present research design.

Data collection

Data were obtained from interviews, subject tests, and the content of online discussion forums. Three students from each section were selected for interviews, covering the range of performance on the initial pre-test (low, medium, and high scores). Each of these students were interviewed four times before the intervention and at the end of each unit. Each interview was audio-recorded and lasted for approximately 45 min. The audio recordings were transcribed verbatim and respondents were invited to review the transcript for their interviews.

The researchers and the instructor developed 60 questions covering the three units expected to be taught. The questions were reviewed by three subject specialists to determine whether the three learning domains were appropriately covered. An experienced chemistry instructor was also invited to review the questions. The questions were pilot-tested with 30 students who were not participants in the study. Following modification, three sets of tests were then developed from the 60 questions. Each test comprised 20 questions covering each of the three units. Students in different sections were tested before and at the end of each unit. The pre-test and post-test was the same for a given unit.

Online discussions were also used in every intervention cycle of the two experimental conditions. Five discussion forums were conducted for each condition with each discussion topic lasting 3 weeks. In both treatments, the students discussed the same topics in the discussion forums. The online discussions were not assessed, since not all sections had them. The students’ messages and graphics that contributed to the online discussion forums were a crucial source of data for determining the level of student interaction and thinking.

Data analysis

All interviews were transcribed verbatim and entered into ATLAS.ti software for coding and analysis. The coding of interview transcripts helped to acquire a deeper understanding of the students’ views about learning before and during the three iterations. The researchers coded all of the interview transcripts and generated a set of codes. A number of recurring codes were clustered into dimensions (main codes) and sub-codes. To establish coding reliability, the sub-code descriptions and associated examples were given to an independent coder for checking. The researchers and the independent coder agreed on 30 of the 35 different types of sub-codes. Inter-coder agreement was 0.819 (Cohen’s kappa), suggesting excellent inter-coder agreement (Gwet 2012). Discrepancies were resolved through negotiation. The researchers refined the sub-codes, and re-coded all of the data using the refined coding scheme, which comprised 4 main codes: teaching presence, interactivity, engagement, and academic success.

The students were tested more than once on the same dependent variable to measure mastery of content. Thus, we used repeated-measures ANOVA to analyze the test scores. NetMiner 4 social network analysis (SNA) software was used to analyze the online discussion forum log files, which generated patterns for post-writing efforts, network densities, and interactions cliques (Scott 2012).

Content analysis was applied to the online discussion log files to analyze advances in the students’ thinking. To understand the conversational trends, we read the messages in the whole of the sampled Moodle discussion forums. We identified various themes across the three iterations and organized these into inquiry threads, which were entered into ATLAS.ti for coding. We defined inquiry thread as an emergent unit of ideas representing a conceptual line of argument in online discourse (Zhang et al. 2007); it is composed of a chronological sequence of built on ideas to address a certain focal problem. We developed a coding scheme based on the Practical Inquiry Model (Garrison et al. 2001) to analyze the discourses within the inquiry threads. The four major cognitive dimensions in Garrison’s (2001) study (triggering event, exploration, integration, and resolution) were adapted and further developed theoretically. The first author developed most of the codes using data from discussion forum 1 of the two treatments. He improved the code descriptions consistently, and organized the codes into main and sub-codes.

To establish the reliability of the coding scheme, the sub-code descriptions were given to the second author to check. The researchers were in agreement on 22 of the 29 sub-codes. Statistical analysis of the ratings of the different coders resulted in an inter-coder agreement of 0.72 (Cohen’s kappa). We resolved our discrepancies through negotiation, then the first author revised the coding scheme based on the agreement reached, and re-coded all of the data.

Results

The results are presented in several subsections. First, we present the students’ views on their learning before the intervention, followed by their views on it during the intervention. Next, we present the students’ learning processes in terms of social interaction, academic achievement, and advances in student thinking.

Students’ views on learning before the intervention

The main themes that emerged from data in this phase were interactivity and teaching presence. The term teaching presence refers to the acts of instructors to connect and facilitate online learning community (discourse) and create activities required to promote the students cognitive and social learning. The students revealed that before the intervention, the e-learning designs had not provided them with opportunities for collaborative learning. Most students were learning in isolation, as their instructors had not guided them in exploratory inquiry. As such, the students were uncomfortable with the learning environment, and the absence of social contact:

Just try to put yourself in a situation where you get only notes and you are left without any assistance. Would you be in a position to get vibrant learning? I think not. There is little communication between us and the instructors, which leaves us with a lot of learning misconceptions. (P86:53; 43:43)

Views such as this showed that the students were dissatisfied with the way in which the educational experience was structured. They wanted more interaction with each other and with the teaching presence, which involves designing and facilitating online discourses. They relied on isolative learning, which led them to assimilate chunks of unconnected information. These results were not surprising and consistent with both our previous findings (Msonde and Van Aalst 2014) and the literature generally. What is interesting here, though, is that the students envisaged more interactive learning in an e-learning course. These views were found in all three tutorial sections.

Students’ views on learning during the intervention

To obtain a clear picture of how the students perceived their learning and how this learning developed during the three cycles, we analyzed each course design and each cycle (iteration) separately.

Non-interactive learning (NIL) environment

The main themes that emerged from the three iterations of the NIL environment was student interaction and academic success. Results showed that the students perceived the designs of the three iterations of the NIL approach to be unexciting, restricting them from learning interactively. The instructional strategies used were not facilitative. As a result, students appeared to embrace surface approaches to learning:

If e-learning provides just lectures and reference materials, what do you expect? I think we will get low knowledge because we don’t get different ideas, and there is no chance for discussions. I just learn myself sometimes parroting difficult concepts. (P76:68; 39:39)

Views such as this one suggested that some of the students were uncomfortable with the lack of social learning because they wanted to learn collaboratively but had no chance to do so. Although the learning culture was influenced by the traditional teaching approaches, the students’ preference was for social learning supported by the learning tool. They expected that using these tools would provide them with opportunities to explore ideas further and engage them in collaborative construction of knowledge. The findings revealed that the lack of social learning forced students to rely on a traditional learning approach. Generally, e-learning practices that are less interactive are perceived as being poor in quality, and require instructors to change their traditional teaching culture and integrate participatory pedagogical strategies.

Moderately interactive learning (MIL) environment

In the MIL environment, teaching presence, students’ interactivity, and engagement with contents were the main themes realized throughout the three iterations. The results showed that the introduction of online discussion forums enabled the students to develop knowledge-sharing habits. However, during the first iteration the teaching presence did not enable students to participate actively in their learning. The situation was different during the second iteration, during which the instructor engaged the students more in social learning where the students connected what they learned beyond the given content:

The instructor provided a lot of posts during discussion. He tried to advise us to extend the discussion beyond the lesson we were studying. It was helpful to receive critiques from the instructor as it gave us insights on how to better our ideas. (P85:45; 24:24)

This excerpt shows that instructor facilitation enabled students’ deeper learning. The actions of the instructor in encouraging the students to deepen their level of inquiry, and improve their ideas resulted in higher-order thinking. After the third MIL iteration, the students reported a high degree of collaborative learning. They perceived the instructional strategies that were used encouraged them to extensively read educational material to obtain evidence to support their ideas during online discourse. Since the students had similar views, we selected one excerpt which conveys their views:

The change in guidelines made by our instructor enabled us to think deeply. This is because before asking any questions of each other or giving a comment you need to read extensively so as to get supporting evidence on what you want to argue. (P85:120; 22:22)

In this excerpt the student was talking about the effect of the changes in pedagogical strategy, which resulted in the students’ extensive engagement in reading educational material that gave insight into the lessons. They were also highly involved in social interactivity and collaborative learning. These pedagogical strategies were more likely to change the students’ learning culture and help them to achieve higher-order thinking.

Highly interactive learning (HIL) environment

The themes found in the three iterations of the HIL environment include teaching presence, student interactivity, and effortful engagement with content. For example, in the first iteration of the HIL approach, the students perceived that the audio podcasts provided them with additional learning possibilities. However, the instructor had not provided students with immediate and specific feedback, instead providing feedback to the whole tutorial group after a long time. As such, the students became inactive in contributing to the ongoing online discussions. Presumably, many were still not confident enough to cope immediately with a new learning culture and delay in instructor feedback, thus, contributing greatly to their inactivity. The inclusion of video and audio podcasts in the second and third iterations attracted them toward active learning. We asked students to explain what new learning experiences they had gained. The majority (67%) of interviewed students perceived effortful engagement for repeatedly replaying of podcasts material enhanced their learning; others perceived the social interaction to be a new experience; and a few students felt that carrying out reflection was a new learning experience:

The experiences gained are many. For example, the instructor tirelessly led the online discussions and I was reading ideas of others and writing down my own views. I was also carefully watching the podcasts so as to come up with the main ideas raised. The newest experience was writing reflections of what I learned. (P72:15; 23:23)

This student was explaining the importance of teaching presence in online learning and the manner in which they undertook learning. The key element highlighted in this excerpt is the development of social interactivity during learning. The students were eager to read and improve on the ideas written about by others. As a result, they were likely to acquire a wider knowledge base. The students also reported an attraction to learning brought about by the presence of podcasts. The students’ effortful engagement of replaying these podcasts enabled them to grasp the concepts addressed therein and to reflect on what they had achieved, suggesting the attainment of higher-order thinking.

Students’ learning processes during the interventions

Students’ learning process was analyzed in terms of student interaction patterns to answer the first research question.

Level of social interactivity in online discourse

Social network analysis (SNA) was used to determine the students’ interaction patterns; their efforts in post writing were measured for each iteration.

Student post-writing effort

Results of repeated-measures ANOVA, given in Table 1, show that the main effect for time for the students’ writing efforts was statistically significant, F (4, 272) = 27.646, p < 0.0005, partial η2 = 0.289 (large effect). Similarly, there was a significant main effect between the groups (conditions), F (1, 68) = 25.136, p < 0.0005, partial η2 = 0.270 (large). Fritz, Morris and Richler (2012) categorized effect sizes as small for η2 < 0.06, medium for η2 between 0.06 and 0.14, and large for η2 ≥ 0.14.

Table 1 Descriptive statistics (M & SD) of post writing
Full size table

The mean post-writing effort per student in forums 3, 4 and 5 were significantly different between the groups, F (1,69) = 7.773, p = 0.007, partial η2 = 0.102 (medium); F (1,69) = 7.901, p = 0.006, partial η2 = 0.104 (medium); and F (1,69) = 15.316, p < 0.0005, partial η2 = 0.184 (large). However, in forums 1 and 2 the results were not statistically different, F (1,69) = 3.273, p = 0.075, partial η2 = 0.046 (small), and F (1,69) = 2.529, p = 0.116, partial η2 = 0.036 (small).

Network density for post linking

Three of five discussion forums, one from each iteration, were randomly selected to demonstrate linkages between student posts: Forums 1, 3, and 5 were those randomly chosen for this analysis. The written messages were coded to determine the extent to which the students were referring to or building on the ideas of others to produce coherent meaning; the results are shown in Table 2. A repeated-measures ANOVA revealed a main effect for linking of student ideas, F (2, 136) = 5.093, p = 0.007, partial η2 = 0.070 (medium), and significant differences between groups for between-subject effects, F (1, 68) = 6.178, p = 0.015, partial η2 = 0.083 (medium).

Table 2 Density of post (Note) writing and linking in three iterations
Full size table

Following from these results, a one-way ANOVA showed that the densities for idea linking in forum 5 were significantly different between groups, F (1, 69) = 4.867, p = 0.031, partial η2 = 0.067 (medium). There were improvements in idea linking between groups, such that student idea linking developed somewhat more highly over time in the HIL design. However, idea linking in forums 3 and 1 was not statistically different, F (1, 69) = 3.851, p = 0.054, partial η2 = 0.053 (small), and F (1, 69) = 0.008, p = 0.928, partial η2 = 0.001 (small).

Clique analysis

The social structure existing within a learning community may be described as a clique (Scott 2012). Zhang et al. (2009) defined a clique as “a sub-set of a network in which the actors are more closely and intensively tied to one another than they are to other members of the network” (p. 26). The analysis results given in Table 3 show that there were few cliques in either of the treatment conditions in the first iteration.

Table 3 Clique analysis for the note (idea) linking networks
Full size table

During the first iteration the cohesion index values for both groups were relatively high, suggesting that the students were demonstrating high levels of interaction within a clique. The cohesion index is a measure of how close individuals within a clique are (Scott 2012). In the second iteration the number of cliques increased moderately, while in the third iteration there was a dramatic increase in clique numbers. These increases in clique numbers imply that students experienced more interaction and collaboration. The lower cohesion index implies that closeness between students within cliques reduced with an increase in community-wide interaction.

Academic performance

Before the analysis, the researchers tested the data in order to determine whether the assumptions for ANOVA with repeated measures were met. A Shapiro–Wilk’s test showed test scores were approximately normally distributed (p > 0.05). A Levene’s test revealed no significant violation of the homogeneity of variance assumption across the groups (p = 0.315). Repeated-measures ANOVA of the test results indicated a significant main effect for treatment, F (3, 297) = 64.474, p < 0.0005, partial η2 = 0.394 (large), and a significant main effect for groups, F (2, 99) = 15.130, p < 0.0005, partial η2 = 0.234 (large). Table 4 presents the mean scores and standard deviations for each condition.

Table 4 Mean differences between the three learning environments
Full size table

Follow-up tests (Tukey’s HSD) found which statistically significant differences in the means of students in the HIL and the MIL environments (p = 0.004), and in the HIL environment and the control group (p < 0.0005). Further analysis showed that there were statistically significant interaction effects, F (6, 297) = 14.284, p < 0.0005, partial η2 = 0.224 (large), which were analyzed using tests for simple effects (Abrami et al. 2001). Figure 2 shows the estimated marginal means of the groups at different time points. A one-way ANOVA was run for the first iteration (time 2). Table 5 presents a group comparison of simple effects, and results revealed no differences in the mean of the groups, F (2, 99) = 0.173, p = 0.841, partial η2 = 0.003 (small).

Fig. 2
figure 2

Changes in means for the three e-learning designs

Full size image

However, for the second iteration (time 3), there were significant differences in the means of the groups, F (2, 99) = 8.227, p < 0.0005, partial η2 = 0.143 (large). The Tukey’s post hoc test gave a statistically significant difference in means between the NIL and HIL environments, p < 0.0005.

Table 5 Simple effect at the 2nd and 3rd iteration
Full size table

A one-way ANOVA for the third iteration (time 4) indicated significant differences between the groups, F (2, 99) = 42.049, p < 0.0005, partial η2 = 0.459 (large). Tukey’s HSD post hoc results shown in Table 5 reveal a statistically significant difference in means between the NIL and MIL environments, p = 0.001. There were also significant differences in means between the NIL and HIL environments, p < 0.0005, and between the MIL and the HIL environments, p < 0.0005.

Characterizing student thinking

An inquiry-thread analysis was conducted on three of the five discussion forums (1, 3, and 5) used to analyze the interaction patterns. To analyze student thinking, the social exchanges between actors in question–answer or idea–comment exchanges were examined first. Table 6 presents the descriptive statistics for the inquiry threads and the distribution of messages in the sampled online discourses. The data showed that the instructor initiated 11 and 7 inquiry threads, and wrote 51 and 72 messages, in the MIL and HIL environments, respectively.

Table 6 Thread and message distribution in the two experimental conditions
Full size table

The instructor messages were questions that asked students (a) to come up with insightful ideas relating to the problem, and (b) to deepen their previously stated ideas. Zhang et al. (2009) categorized these two types as questions for ideas and questions on ideas, respectively. The essence of these questions was to determine the kind of feedback the instructor provided and how they deepened the students’ inquiry as well as their thinking. Table 7 presents the proportions of the two categories of question asked by the instructor.

Table 7 Categories of instructor questions asked
Full size table

The data shows that questions on ideas increased across the three iterations for both treatments. The instructor asked a few questions for ideas but more questions on ideas, which were numerous in the third iteration of the HIL environment. The increase in questions on ideas not only indicated the level of commitment of the instructor, but also deepened the level of student inquiry.

Advances in student thinking

We also analyzed discussion forums 1, 3, and 5 to determine advances in student knowledge during the online discourses. Table 8 reports the codes related to knowledge exploration which entails the generation of questions seeking explanation and clarification (Zhang et al. 2007).

Table 8 Codes for exploration and idea improvement
Full size table

Table 8 shows that students made factual statements (60 coded instances), provided conjectures on problems (76 coded instances), gave explanations (78 coded instances) and elaborations (72 coded instances) of what they were discussing. They also asked questions seeking explanation (151 coded instances) and clarification (119 coded instances), using “what,” “how,” “why,” and “what if” statements. These types of questions extended the students’ thinking and led them to clarify earlier statements.

Chi square tests were used to investigate whether distributions of categorical variables that determine advances in student thinking differ from one another. The tests revealed that during the first iteration there were no differences in knowledge advancement between students in the MIL and HIL environments, χ 2 (df = 5, N = 68) = 11.663, p > 0.05. However, during the second and third iterations there were significant differences in knowledge exploration between the students in the two treatments, χ 2 (df = 5, N = 68) = 12.369, p = 0.03, and χ 2 (df = 5, N = 68) = 11.663, p = 0.04 respectively.

Knowledge (idea) integration and resolution

We defined idea integration as “those activities that involved students bringing together a repertoire of ideas into coherent meaning.” Table 9 categorizes integration as (a) building on others’ ideas (101 coded instances), where students expand on these ideas; (b) bridging the repertoire of ideas using authoritative sources (70 codes instances); or (c) making reflective statements (75 codes instances) that consolidate ideas into a coherent account (Linn and Eylon 2011).

Table 9 Codes for knowledge contribution and integration
Full size table

Chi square tests revealed no differences in knowledge integration during the first iteration, χ 2 (df = 2, N = 68) = 4.798, p > 0.05. However, there were significant differences in knowledge integration during the second and third iterations, χ 2 (df = 2, N = 68) = 27.055, p < 0.001, and χ 2 (df = 2, N = 68) = 4.798, p = 0.027, respectively. The data showed that the students’ knowledge exploration, integration and resolution were gradual and progressive, albeit much higher in the HIL environment and especially in the second and third iterations.

Discussion

Most previous design studies have not conducted experimental comparisons of different designs that involve a control group. Rather, they have tended to implement the same design in different classes, and evaluate its effectiveness. This study developed three pedagogical designs that are underpinned by different theoretical perspectives—from instructivist to social-constructivist—and tested their effects against each other on student interaction, learning and higher-order thinking. The study not only compared the three designs, but also let each evolve through three iterations, so it is one of the first studies we know that examines the development of the three pedagogical approaches and obtained powerful findings.

We discuss the findings in terms of the effect of the course designs on (1) student academic achievement, (2) interaction and engagement, and (3) advances in student thinking.

Student academic achievement

Several studies have reported the effects of technology on student achievement (e.g. Bassili 2008; Guerra and Grimon 2009; Ng’ambi and Lombe 2012). However, many of these studies lack empirical evidence for verifying whether discussion forums and podcasts have significant effects on student achievement. Van Stumm et al. (2011) demonstrated that students’ academic achievement can be associated with their cognitive ability, effortful engagement and intellectual curiosity. The findings of this study corroborate those of Van Stumm et al. (2011) in that student academic achievement was found to increase with an increase in intellectual curiosity, which helped the students’ to inquire logically of each other and to generate a repertoire of knowledge that contributed to their academic success. For example, the introduction of a discussion forum in the MIL design, created a social learning space, which empowered students to negotiate ideas through asking questions of and critiquing one another based on evidence. Such actions promoted the student interactions, and stimulated their intellectual curiosity that are crucial for promoting meaningful learning (Woo and Reeves 2007). It was quite surprising that, although the students were new to online learning, having a discussion forum led to progressive development of academic achievement. This result was quite strong compared with the literature findings in general, as strong learning effects are not often observed (Guerra and Grimon 2009; Maddrell 2011). For example, Guerra and Grimon’s (2009) found no evidence for student academic achievement when a Moodle discussion forum was used. The findings from this study, however, provide evidence that meaningful interactions can lead to significant academic achievement.

The goal of podcasts usage in the HIL design was to boost the online discussions and to determine whether the pedagogical advantage of promoting social networks between students during learning is realized. The theoretical assumption here is that students develop more thinking (Kay and Kletskin 2012), and improve academic performance (Traphagan et al. 2010) when podcasts are used. Findings from this design reveal that the students’ academic progress was even better when podcasts were incorporated in the HIL design. Previous studies have reported similar results, in which students performed better in tests when podcasts were used (Griffin et al. 2009; Traphagan et al. 2010). This study corroborates these previous studies in demonstrating that the use of podcasts together with discussion forums is a powerful way of promoting academic achievement. The repeated replaying of a complex segment gave students’ opportunities to practice and increased expertise in knowledge systems, which account for changes in one’s thinking. These findings indicate the potential benefits of repeating and sharing ideas to improve learning. Moreover, these findings provide evidence that addresses the concerns of some previous researchers (Bassili 2008; Hew 2009), who expressed skepticism whether the use of podcasts could lead to significant learning outcomes. The empirical evidence of this study suggests that it is possible for institutions of higher education to design high-quality distance-learning courses that would maximize the educational returns using the currently available resources. However, the use of podcasts to boost the online discussions is not a low-bandwidth addition. Therefore, there is a compelling reason to implement the MIL design in higher education in Tanzania as it is technically not very demanding and there is good pedagogical advice for making online discussions effective. For the HIL design, which requires more bandwidth and instructor additional effort, it may be worthwhile to distribute podcasts materials offline for easier accessibility and powerful contribution on student learning. Having access to high-quality e-learning in higher education increases the likelihood that students will develop higher-order thinking and thus be capable of contributing positively to the welfare of their community.

Student interactivity and engagement

Interaction and engagement are essential aspects of learning. Although the instructor in this study tried to develop and sustain online dialogue within each environment, this had little effect on students in the NIL design. Given the limited opportunity for collaboration, the students passively retrieved and memorized course content posted by the instructor. Numerous studies demonstrate that the cognitive processes necessary for students to develop deeper learning occur in dialogue (Woo and Reeves 2007). Thus, the MIL design created a moderate social learning space essential for these dialogues to occur. For example, the students used the discussion forums to explore ideas, ask their peers productive questions, and critique the ideas of others. These findings are encouraging in that using forums can have a worthwhile effect over just posting instructional materials. Some of the factors that contributed to the improved interaction were the instructional strategies, which led the students to actively participate in exploration and exchange of ideas. Other factors were the instructor presence, and empowering the students with an agency for solving inquiry problems. The instructor worked diligently to support student learning by asking them to give ideas on their understanding of certain concepts, and wrote follow-up questions to students’ written ideas on various inquiry problems.

While students in the MIL environment demonstrated significant social interaction, those in the HIL environment exhibited extensive and interesting social and cognitive dynamics. In addition to being able to learn socially using discussion forums, they had the opportunity to listen to or watch podcasts repeatedly, and think deeply about the concepts (Griffin et al. 2009). These students were also more willing to bring the concepts learned from the podcasts to the learning community for further idea improvement. These elements represent some elements of social constructivist theory, which emphasizes students’ interactions in the process of constructing new knowledge. The major implication here is that effective use of discussion forums can produce important learning effects and is a low-bandwidth strategy, but coupling podcasts maybe a powerful way to enhance the student learning effects. However, with the present technological infrastructure in Tanzania, it is unlikely that live streaming of podcasts is feasible especially in rural areas with limited bandwidth, but that there are other options. For example, the use of the MIL design has shown positive learning effect. Thus, instructors of higher education institutions in Tanzania need to design their e-courses in line with the MIL design which can make quite a difference in learning effect compared with the NIL design.

As noted above, questions played a crucial role in promoting deep learning. Although previous researchers have recognized the role played by questions in learning, they have not accounted for what the nature of these questions should be. Ertmer et al. (2011) demonstrated that crafting questions at higher levels promoted student interaction. The current study advances the argument of Ertmer et al. (2011) by bringing in two categories of reflective pedagogical questions on ideas, and questions for ideas that can be crafted at high levels to promote meaningful interaction. Such questions create inter-subjectivity and cognitive dynamics among students. The growth of social and cognitive dynamics was useful for determining the student interaction patterns. Many scholars use social network analysis (SNA) to examine interaction patterns based on notes written in online discourses without considering the actual ideas contained in those notes. Such analysis addresses a general participation in online discourse (Scott 2012). In this study, SNA was performed quite differently from other research. The SNA nodes not only represented the notes written by participants in the forums, but were also the result of coding of ideas generated during exploration, integration and resolution. This type of analysis was crucial for determining how the ideas developed within the learning community, which is more meaningful than simply linking notes written in forums.

Advances in student thinking

Social learning emphasizes collaboration between students, and is a fundamental process for the construction and improvement of knowledge. The findings from this research revealed that the students in the NIL environment had few opportunities to undertake exploratory inquiry. The educational and cultural context did not provide them with sufficient diversity of knowledge. However, social and cognitive dynamics were evident in the HIL and MIL environments where the students explored the inquiry problems and reached various conceptual advances. Moreover, the additional learning opportunities offered by the podcasts in the HIL environment encouraged the students to move to higher knowledge levels through integrating a repertoire of ideas. They also transcended learning beyond the subject matter they were studying to real-life encounters. Such knowledge advances depict the way students’ reached certain saturation in cognitive processes.

Some previous studies have reported that the use of asynchronous discussion rarely involves the concept of idea improvement (Niu and Van Aalst 2009). In contrast, this study used asynchronous environments to progress past mere discussion to deeper inquiry leading to knowledge advancement. For example, the use of questions on ideas and questions for ideas deepened the level of that inquiry and the students’ cognitive processes (Zhang et al. 2009). Although the students in both of the treatment groups achieved a degree of higher-order thinking in the first iteration, they continued to improve on this in subsequent iterations. To be precise, the instructor was dissatisfied with the initial results of the first round. He was then encouraged in the design team reflective meetings to improve on the pedagogical practices. These meetings acted as teacher professional development (TPD), and were very important for the course instructor to gain an insight into what should be implemented. The instructor invested much time supporting the online discussions by providing students with more questions on ideas. The level of higher-order thinking thus improved from the MIL design to the HIL design in the second and third iterations.

An interesting finding from this study is that students developed higher order learning even by using the discussion forum alone. However, it was not merely the provision of a discussion forum in the MIL design, and podcasts in the HIL design, that promoted student learning. Rather, the instructional strategies grounded in social constructivist theory built into the learning materials guided students in achieving meaningful learning. The new insight that could be realized is that we compared not just the three designs but the development of learning and interaction in each design over several units. However, such development was not straightforward for the instructor to accomplish as he did not have advanced technical knowledge. Thus, the design team supported the instructor technically to develop appropriate pedagogical practices, and it took him a little bit of time to learn how to do such extra instruction. These findings provide a broad view of the possibility of enhancing distance-learning courses that are capable of developing competent graduates who may use the acquired knowledge for socio-economic development.

It has been surprising to see disappointing learning effects, when universities have sufficiently sound technological infrastructure to allow effective pedagogical designs and practices (OECD 2015). The major implication for the design of e-learning courses in Tanzania and in any developing or developed countries is that instructors need a conceptual change regarding what they should be doing in these kinds of design studies that compares development of learning for different interventions. They need different ideas geared toward the efficient use of available technology in a way that might help students’ acquire useful knowledge.

Limitations and further research

Despite the clear findings obtained by this study, some limitations exist. First, this study used pedagogical approaches of only one instructor. It is not known what would happen if different instructors with different pedagogical experiences used the discussion forums and or podcasts. It would be very useful to have additional studies in this area that will investigate kinds of pedagogical aspects that might be critical for making these designs successful. The second limitation concerns the scalability of the designs. In order to know more about whether these designs work well it is crucial to involve multiple courses that can be implemented in multiple settings. Thus, further studies should consider involving more instructors at this university or from different universities with similar contexts. Third, this study used only one example of web 2.0 technologies (podcasts) to foster student learning. It would be useful for follow-up research to broaden the repertoire of web 2.0 technologies and see what educational benefits can be brought using these tools. Finally, critics could argue that the superior learning effects in the MIL and HIL designs, compared with the NIL one, could be caused by greater engagement (i.e., students made additional efforts to learn) since the study was unable to access the learning process in the NIL design. However, if so, we would, like Brown (1992), consider this to be a positive effect of these designs.

Conclusion

Findings from this research reveal that before the introduction of the novel course design, students had few opportunities to interact, and learn course content in isolation. The three developed pedagogical designs underpinned by different theoretical perspectives improved students social interaction—advancing from isolative learning to collaborative learning and progressively improved students academic achievement and thinking over the three iterations. These advances were modest when discussion forums alone were used, but extensive when podcasts were added. In a limited bandwidth context like those in developing countries in Africa, especially Tanzania; this is a promising direction to take in alleviating the disappointing student learning results.