The analysis outcomes are organized around three categories: (1) effective instructional strategies adopted according to the students, (2) strategies not adopted but used by students on their own, and (3) effective online course design features. Specific examples of students’ reflections are also presented to provide insight of students’ perspectives regarding the research question, as well as explanations of the students’ perspectives regarding the emerged categories. Examples provided for each category and its sub-categories were from different participants.
Effective instructional strategies
According to the students, almost all instructional strategies adopted in the online course were helpful. However, case studies were the most effective in helping students learn the statistics concepts and the applications of the concepts in the online course (Fig. 1).
Case studies were perceived to be the most effective and particularly helpful because they allowed students to apply what they learned as well as to connect their learning in a meaningful way. For example, one student reflected,
“… the case study was a great way to apply our new skills to something real-life. I’ve read so many research articles in the past, but I’ve never quite understood the technical side of the analysis. I thought that …seeing a real-life application in a research study helped to put the pieces together – mostly.”
Another student also wrote about how the case studies connected their learning with the applications of what they had learned:
“I enjoyed completing the case study. I think that these [case studies] assignments allow me to apply what I am learning in class in a meaningful way; a way that contributes to my further growth as a researcher and as a reader of research. … It reminds me that teaching using the context in which the knowledge would be used later was very powerful.”
Case studies were also able to help students achieve a deeper understanding of the newly learned concepts. For example, one student reflected:
“They [case studies] force me to work through the details to the point at which I have an in-depth understanding of the concept and can apply it to an actual situation. Answering the cases questions required me to analyze how the research was performed. … The case studies helped me think critically, especially by thinking and responding to the questions…”
The case studies also provided an opportunity for students to compare their learning and understanding with those of their instructor’s and their peers’ through instructor’s feedback and class discussions. In this sense, case studies were an effective instructional strategy for providing opportunity for self-reflective learning that leads to deeper understanding (Penn et al. 2016). For example, one student wrote:
“Case studies have been the most important part of the course for me. I like the problem solving, the sense of a whole and the feeling of acquiring a skill. Most importantly they provided the perfect opportunities to compare my understanding with my professor’s and my peers’ understanding of the same set of questions.”
Following the case studies, video demonstrations of statistical tests and procedures using SPSS were most helpful and effective instructional strategy in the online statistics class. The video demonstrations successfully achieved their intended purpose, teaching students how to perform different statistical procedures in SPSS. It seemed that those video demonstrations effectively replaced a face-to-face, hands-on lab session. For example, a student wrote, “The videos were a life saver with using SPSS. Being able to follow along with someone as they performed all the steps made it much easier to understand this very intimidating program.” In addition, most students agreed that watching a video is much more effective than learning from reading the textbook or a manual for SPSS. For example, one student pointed out: “In most cases, watching a 10-minute video gave me a better understanding than reading a whole chapter in the book.” Interestingly, video demonstrations seemed to be more effective than reading step-by-step instructions and also watching a live demonstration because students could watch recorded demonstrations as much as needed. For example, one student reflected how the video demonstrations helped him,
“I found the video tutorials helpful. I re-viewed them several times to strengthen my understanding. It was nice that the narratives did not just explain what to do, but also why we were doing it and how to interpret the results [output].”
Some students also found that the video demonstrations provided a less intimidating way for them to learn the statistical program of SPSS. This may be due to the fact that students could watch the video demonstrations as much as they needed without worrying about the peer pressure of appearing to fall behind, a situation students might encounter in a face-to-face class. For example, one student reflected:
“The most informative and useful instructional strategy was the videos. As I have gone through this course, I am beginning to look forward to using SPSS. It is not so intimidating to me at this time, which is in large part due to the video demonstrations. I could watch the videos as much as needed just by myself without feeling of dragging down my peers.”
The instructor’s notes emphasizing the key concepts in each chapter and outlining the similarities and differences between and among similar concepts were found to be the third most effective strategy adopted in the course. The instructor’s notes helped students understand key concepts better. For example, one student reflected, “The instructor’s notes were the most helpful for enhancing understanding and emphasizing key points from the book.” Similarly, another student wrote, “Again for me the instructional strategies presented that helped me the most, were the instructional [instructor] notes. The highlighting of core concepts, with clear explanations, aided me in the reading of many chapters...” Some students even gave specific example of where the instructor’s notes helped them the most. For example, one student reflected, “I was struggling to thoroughly understand the difference between the paired and two sample t-tests. The instructor’s notes, particularly, in Chapter 7, were very helpful for understanding the two concepts.”
The instructor’s notes also provided a great guide and acted as an advance organizer (Mayer 1979) for students when they started to read the textbook to decide what content was the most important. For example, one student reflected:
“The instructor’s notes helped me a lot. The textbook is good but wordy and I could read it and read it again and still not be able to figure out what was important and what wasn’t. With the instructor’s notes, I could figure out what was important and let go of things that were unimportant which I did not understand fully.”
Some students found that it was even more effective to combine the instructor’s notes with other available resources, such as course assignments. For example, one student described how he used instructor’s notes,
“The instructional strategies that I found most effective were the videos and instructor notes/summaries. During each module I would read the instructor notes, print the assignments, and refer to each often while reading the assigned textbook chapters. Having access to each resource at once was the most helpful in understanding a concept.”
Based on the students’ reflections, instructor’s notes that focused on the similarities and differences of similar concepts were not only necessary but also one of the most valuable instructional strategies that helped students learn the content in an online statistics class.
The mini projects were several small projects that built into a final project throughout the semester based on the progression of the course content. The mini projects enabled the re-visiting of the same project over the semester and were the fourth most effective instructional strategy adopted. The mini projects served as a valuable building process of the final project and provided students sufficient time and opportunities to digest concepts before applying them. They also allowed the instructor an opportunity to provide focused comments and feedback on specific aspects of the final project while the relevant concepts were under discussion. For example, one student reflected,
“The mini-projects helped me the most in the course because they helped me complete the final project in a much better way, quality wise. I really appreciated the feedback that I received on my first mini-project, and I believe that my next two were much more refined and [more] complete projects due to the feedback received previously.”
Some students also commented that the mini projects provided them the opportunity to experience how a research study worked from beginning to end in a less intimidating way. For example, one student reflected:
“I also liked that we built upon our mini-projects throughout the semester, which was the most effective way to complete a final project. It felt so much less intimidating to be able to chunk it out and work on it piece by piece as we gained new knowledge and insight [about research].”
It seems that mini projects were very effective to teach educational research and the applications of statistics concepts. They also helped students complete a final project that involved different building processes and acquiring the knowledge for building a final project in a progressive way.
Nearly two-thirds (25/39) of the participants considered the online discussion forum to be an effective instructional strategy adopted in the online statistics class. The online discussion forum provided a means for students to seek clarification and answers to their questions and also a way to validate their understanding and promote self-reflective learning. For example, one student wrote:
“Being that this is a reflection … I feel that the discussion forum was the most useful. I was very engaged and was glad that I could not only help other students to problem solve why they were getting an error message when doing their Tukey test but that I understand the concepts being presented. It also validated for me that I am coming to a greater understanding in the class as a whole.”
By contributing to the class via responding to their peers’ questions, most students felt that the online discussion forum was helpful for encouraging peer-to-peer and instructor-to-student interaction. For example, one student reflected, “As most of us are social people and we feel the need to connect with each other through our courses, it is nice to talk with each other, and discuss concepts and ask questions via the online discussions even in this statistics class.”
In addition, online discussions also provided some extra motivation and promoted a healthy competition among peers. For example, one student shared:
“I read a forum post from someone about getting all the research situations correct from Chapter 13. That was all it took to motivate me to try to meet his high standard. I’m happy to say I met that challenge and tied him in his score.”
For those students who did not actively participate in the online discussion forum, they also benefited from having a place to review relevant discussions on course content and check their own understanding via self-reflection. One student reflected:
“The null hypothesis is a difficult concept. By the time I came up with questions, I noticed many had been posed on the discussion board. It was more helpful to me to read the comments of others than to interject a similar question. Eventually, I came to understand it.”
It appeared that an online discussion forum was very necessary. It helped the students discuss course-related problems and provided a way for students to check their understanding and learning against that of their peers.
Other strategies students used on their own
Using outside online resources
Other instructional strategies that were helpful for promoting students conceptual understanding of the statistical content were searching the Internet and using outside relevant resources that were not provided by the instructor. The majority of the students (35/39) used outside resources, such as websites, online videos, and even people to help their learning. The most frequently used outside resources included the Google search engine; the Khan Academy; Research Gate’s Q&A section; iTunes U; and textbooks, such as textbooks on Amazon called Activity-based Statistics and The Cartoon Guide to Statistics.
Some students referred to outside relevant websites when they could not easily make full sense of the required textbook. In fact, more than half of the students mentioned using Google search or external websites and resources in their reflections. The extra research for relevant resources of the course content and reading additional information online or even other textbooks seemed to be one of the most helpful instructional strategies for the students in this online course. For example, one student reflected: “I fully expect to find my own information in many courses, so this was not a concern for me. I do think the textbook was difficult to learn from, which may be why I looked for more outside resources.”
Relevant outside websites, including videos and interpretations of SPSS outputs helped clarify difficult topics and also served to alleviate students’ fear and anxiety towards learning statistics. Some students shared extra resources they found with their peers in the online discussion forum. For example, one student wrote:
“I set up a Diigo site for posting good stats sites and shared the site with my classmates. …I spent a great deal [of time] doing research on the Internet to clarify concepts when I was still uncertain about them, especially for the t-tests.”
However, the use of outside resources may be due to the students’ different backgrounds and/or entry knowledge levels. For example, one student used outside resources because the textbook was difficult for her to understand. Another more advanced student used outside resources due to the lack of depth of the mathematical concepts covered. For example, the more knowledgeable student reflected: “The textbook overlooked the mathematical concepts that underlie the different [statistics] concepts. After reading the text, I relied on online resources from other universities to better understand the mathematical concepts.”
As a result of different backgrounds and the access to different resources, students used a variety of resources. However, the students experienced challenges finding and using outside resources. One challenge was to find the right resource on the exact topic. For example, one student wrote, “The challenge for me was finding the right video about the exact topic I was looking for.” Another challenge was not all resources exactly met the students’ needs. For example, one student wrote, “I think some of the videos [found on the Internet] were focused on “do this” and didn’t add to my understanding of why I was doing what I was doing.” The third challenge was that different sources may use different terminologies for the same or similar concepts. For example, one student reported, “I have found myself using Google to better understand some concepts. However, sometimes, different terms or names have been used by different sources than the terms used by our textbook or instructor. This sometime caused more confusion.”
Practicing textbook examples and chapter problems
More than half of the students (21/39) also tried to figure out the statistical or mathematical process underlying a statistics procedure on their own in order to achieve a better understanding by working out the textbook examples and/or chapter exercises. For example, some students calculated the confidence interval example in the textbook using a pencil and a piece of paper. Working out the textbook examples on their own helped some students understand the concepts better. For example, one student reflected, “I find calculating the statistics by hand gives me a deeper understanding of the statistics since I am forced to work through the data and the resulting variables.” Similarly, working out the chapter problems and exercises on their own helped or reinforced students’ understanding of the statistics concepts or procedures. One student wrote:
“For many of the chapters, I worked most, if not all, of the practice problems at the end of each chapter. Working the practice problems at the end of the chapters helped me to execute and reinforce the statistical concepts presented. Although these were not assigned, I felt as if figuring the statistics by hand often helped me to comprehend the important concepts in the chapters.”
Given more time, more students might have been able and/or willing to work out the textbook examples and the end of chapter practice exercises.
Seeking help from peers, colleagues, and friends
Some students (15/39) also sought out people who were knowledgeable about statistics to help their learning in this online course. For example, one student reflected:
“I have also been working with two other students in the class. Alex and Paul [pseudonyms] have helped me tremendously and I don’t think I would have been able to finish this class without their support. Alex especially helped me to gain the confidence I needed to move forward. … I am very grateful that I didn’t give up.”
Students sought help from various people, including their spouses, friends, and colleagues. For example, one student wrote, “I also sought assistance from two colleagues: A statistics teacher … and a principal who recently completed a doctoral program.” This finding made the researcher wonder if students in a face-to-face class would seek out help from knowledgeable people the same way as those in an online course.
One-to-one phone calls
In addition to e-mails asking for clarifications of course content and weekly assignments, some students (9/39) also contacted the instructor for one-to-one phone calls to help clarify some concepts, such as the dependent vs. independent variables.
It seemed that the majority of students in the online class actively engaged in learning, which was revealed in their thoughtful reflections. The students completed the course tasks assigned by the instructor and also actively sought out extra resources and help during their learning process. Although the outside resources provided students with multiple opportunities to interact and learn the course content, it was not without challenge. Online course instructors need to be conscious of this aspect when designing an online course.
Effective online course design features
For this aspect, the researcher first examined the end of semester course evaluations, specifically the four questions related to most important course structure and learning activities as anonymously reported by 40 students (Table 1). The four course evaluation questions presented in Table 1 related to clear objectives and the alignment between course assessment and objectives and organizations of learning activities (peer collaboration/learning community) in online courses. The course design features covered by the four questions were also determined by the course instructor/designer. Table 1 lists the average means of students’ responses to the four course design questions on a one to five Likert scale with five being the highest score.
Table 1 Student evaluations of course design features by semester/year
Based on the course evaluations, the course objectives were clear and the assessments and course materials were well aligned with course objectives because the weighted averages of students’ responses to the first three questions were 4.3 or more on a scale of one to five. Students’ responses to the peer collaboration question were much lower than those to the other three questions, which again demonstrates the challenge of creating an effective and collaborative learning community in an online STEM course (Bacon and MacKinnon 2016).
Next, the researcher coded the 39 reflections for effective course design features. Overall, the students considered the course well designed. Students identified the following course features as being helpful in this course:
Consistent structure of the course
The majority of the students (31/39) viewed the consistent structure and layout of each module’s content as helpful and effective. The consistent structure or layout of each module helped students get familiar with the course as well as ease their fears of statistics. For example, one student wrote:
“I enjoyed the consistency of the course structure. The first module seemed quite difficult and there was a lot content, but because the following modules were very similar I eventually got the hang of things and felt much more comfortable and confident as I knew what was to expect.”
This shows that a consistent course layout with friendly navigation within an online course is very important to keep students oriented and reduce the fear of a topic that is perceived as challenging.
Various resources, assignments, and activities
Some students (28/39) also liked the overall course design because it offered a variety of resources, assignments, and activities. All assignments and activities organized in the course worked cohesively to help students obtain the learning objectives. This aspect was also reflected in the students’ course evaluations regarding the good alignment between course objectives, materials, and assessments. For example, one student reflected:
“I think this class was set up very well. I think the pace, resources, and lessons all worked together cohesively and helped assuage my fears. I appreciated the instructor’s notes throughout each module (what to pay close attention to and what to skim) …. Considering the complexity of the content I could have fallen behind, but the class was set up to provide multiple resources/assignments to study the content and allow us to apply what we learned.”
Similarly, one student wrote, “I enjoyed the flow of the course. The balance between the bookwork [selected chapter exercises], the SPSS exercises, and external assignments [such as case studies] was good.” It seemed that providing different resources, assignments, and activities in an online course can create a combination of ways for students to learn as well as to check if they understood the materials correctly.
The application focus
Some students (27/39) enjoyed the application focus of the course in an introductory statistics course on educational research. For example, one student wrote, “I appreciate the way this course was structured. We had opportunities to be introduced to the material, engage one another in conversation, and then apply our knowledge.” Similarly, another student wrote, “I thoroughly enjoyed the applied statistics. The topic has made me think about things outside of school, especially politics and news reporting of polls.” In regard to how this course was set up differently from a similar course for STEM majors, one student’s quote provided a perfect explanation, “The case studies and mini-projects were very effective for shifting the understanding of concepts into practice. These added an unexpected dimension [application] to what I anticipated to be a basic math course. I appreciated this addition to the coursework.”