The design of this investigation follows research principles that stem from basic qualitative studies, which focus on meaning, understanding, and process as suggested by Merriam (2009), thus a purposeful sample of information is necessary to provide for data collection (e.g., surveys, journal entries). Typically, this kind of investigation is inductive, leading to richly descriptive findings generally presented as themes. More specifically, the design of this investigation is that of a case study focused on a rare or critical event, so its results should have transferability to similar contexts and settings. Qualitative data analysis and interpretation is an ongoing process that involves a constant reflection about the data, asking analytic questions, and writing notes throughout the study. Furthermore, data analysis is conducted concurrently with gathering data, making interpretations, and writing reports (Creswell, 2009).
Subject of analysis
The primary development tool for instructors adopting online modalities at UNM is the six-week Evidence-Based Practices for Teaching Online (EBPTO) course. It was created in 2016 to support instructors planning to teach in UNM’s Accelerated Online Programs (AOPs), which are fully online degree completion programs. Awareness of the course gradually increased, and it was opened to other interested instructors at UNM, including contingent faculty, instructors from UNM’s four branch campuses, and graduate students. Updates were made periodically to improve the course, and it went through a substantial redesign in summer 2019, with revised learning outcomes, improved alignment, more engaging learning activities, and expanded research on teaching and learning. Since its inception, EBPTO has been offered in the fall, spring, and summer terms, often several times each term, making it available year-round. Two additional sessions of EBPTO were added as a response to the need for instructors to design and build sound learning solutions for both online courses and remote teaching modalities during and after Spring 2020.
With the arrival of COVID-19, there was an immediate and unexpected need for all UNM instructors to complete their courses remotely and to continue teaching remotely or fully online in the subsequent summer, fall, and spring terms. EBPTO registrations rose from normal levels of 11 and 15 in the two sessions that started in January and March, 2020, to a peak of 117 in June, 2020. It is worth noting that the initial registration in June, 2020 was 145, but not all participants continued with the course after Module 1. Previous to June 2020, approximately 365 instructors at UNM had completed EBPTO.
In all of 2020, a total of 286 instructors from different colleges, myriad disciplines, and academic ranks registered for EBPTO. To mention just a few examples, instructors were from departments such as Mechanical Engineering, Sociology, English, Music, Chicana Chicano Studies; Individual, Family, and Community Education, Theater, Foreign Languages and Literature, Film and Digital Arts, etc. In short, all UNM colleges were represented in EBPTO. The availability of this course laid the foundation for an orderly response to an incredibly high demand in instructional design services at UNM due to the COVID-19 pandemic.
The course objectives for EBPTO are that upon successful completion, participants are able to: (1) apply evidence-based practices for online learning, (2) prepare to build and/or facilitate an online course following the UNM Online Course Standards Rubric, and (3) practice using UNM Learn tools to understand the student experience. Each session of the course has two facilitators from the UNM instructional design team (part of the UNM Center for Teaching and Learning), and each participant is assigned an instructional designer who guides them through the course process with key touch points aligned with reflective assignments, a course map, and course building activities. While this may seem like a high touch instructor development experience (and it is meant to feel this way for participants), the added workload for instructional designers is moderate because it is spread across a total of nine designers with a project manager and associate director taking on additional instructors when the need arises. Generally speaking, the course allows instructional designers to begin a relationship with participants they will continue to work with moving forward or to help or connect participants to support resources that they can access long-term. By building rapport early, instructional designers are often saved time in the long run because instructors have an early connection to sound resources and design practices.
An important piece of institutional information that supports EBPTO is UNM’s Online Course Standards Rubric. This document was commissioned by a Provost committee serving from 2005 to 2009 comprised of faculty and distance education experts. It was approved by the Faculty Senate Committee on Teaching Enhancement in 2013, and updated in 2019 to better align it with research on effective online teaching practices. A council of faculty and distance education experts meets twice each semester in continued stewardship of the rubric and institutional processes that rely on the rubric. The rubric provides a framework for online course design based on expectations for quality online courses in line with federal regulations for online courses as well as national standards like those established by entities such as Quality Matters, SUNY Course Quality Review Rubric, and the Peralta College Online Equity Rubric. It serves to guide instructors in developing new online courses and in self-assessing existing online courses for improved design and delivery based on nationally recognized, evidence-based practices for teaching online. Expectations for quality online courses are divided into the following standards: (1) Course Overview and Introduction, (2) Instructional Elements, (3) Interaction and Collaboration, (4) Assessment and Feedback, and (5) Course Evaluation. The learning experiences in EBPTO are designed to illustrate practices defined by this rubric that are applicable to a 6-week long faculty development online course.
Within the Online Course Standards Rubric, the Course Overview and Introduction standard requires instructors to clearly outline the course purpose, structure, policies and performance expectations before the student begins the course, which are typically outlined in the syllabus. The Instructional Elements standard requires that learning objectives and aligned measurable outcomes are clearly defined and communicated, and that active learning is promoted through an interactive learning environment, utilizing different types of media, multiple tools, and different formats of materials.
According to the rubric, online course instructors are also required to orchestrate learning activities that are designed to promote both student-to-student and student-to-instructor interaction and engagement. These activities need to be explicitly aligned with learning objectives and outcomes in order to meet the Interaction and Collaboration standard. The Assessment and Feedback standard requires that assessments be aligned with specific learning objectives and general outcomes of a course, and that performance expectations are clearly defined, but most importantly, that expectations are presented to students upfront, and that the grading and feedback procedure is clearly defined in terms of the amount of time it will take the instructor to grade activities. Last but not least, the Course Evaluation standard indicates that course evaluations such as welcome, or mid-course, or end-of-course surveys are utilized by the instructor to provide opportunities for continuous improvement in future course delivery.
To model one way of meeting the rubric criteria, EBPTO begins with an orientation that includes welcome videos from the facilitators and from the Associate Director of the Center for Digital Learning. During the June 2020 sessions, the complement of facilitators was increased from two to four, to provide additional guidance and to handle the increased volume of facilitation tasks. During the orientation, participants are asked to introduce themselves, post any questions or concerns they may have, and complete a survey in which they share their perceptions of online learning. They are also asked to contemplate and share “enduring understandings” they would like their students to take away from their classes over the long term.
In Module 1, participants are shown the results of the perceptions of online learning survey they took in the orientation. They also view a presentation covering fundamental research on the impact of online instruction. They discuss both their perceptions and the research. The remainder of the course takes them through basic design information, such as learning objectives and the backward design process, followed by modules on collaboration and interaction, active learning, assessment of student work, and finally, the processes for ongoing evaluation and revision of online courses.
These materials are covered using videos, presentations, reference documents, and example documents. Participants interact with one another and with facilitators using text-based discussions, video discussions (one within the university LMS and one using FlipGrid), a reflective journal, and course messages. Course assignments include writing course objectives, designing an active learning activity, and completing a course map for the first four modules of an online course. The course ends with a capstone online teaching portfolio that compiles and illustrates an instructor’s work in the course.
The focus of this case study is the largest EBPTO cohort, a session that began in late June, 2020 and continued through July and early August. There was a total of 117 instructor participants actively enrolled in this session. The peak in demand that occurred with this session was fueled in a general way by the pandemic, and specifically by the unprecedented volume of remote classes that needed to be taught in Fall, 2020. The content of EBPTO remained largely the same and focused on the same core objectives, though there were adjustments to interactive course elements to account for the anticipated volume of discussion posts, assignment submissions, and other course activities.
Data Collection and Analysis
To inform broader perspectives about the research problem, the authors considered different raw data from the online course as a preliminary approach to validating the accuracy of the information. After preliminary work was completed, a purposeful sample of information was necessary, so the authors collected end-of-course survey results, and journal entries from the institutional Learning Management System, Blackboard Learn. Then data were anonymized and deidentified in a spreadsheet, which was securely stored in the university’s file hosting service OneDrive with restricted access to the research team.
To conduct the analysis, the following steps were taken according to Creswell (2009): (1) Organizing and preparing data for analysis, (2) Reading through all data, (3) Coding the Data, (4) Developing descriptions, (5) Interrelating descriptions, and (6) Interpreting the meaning of descriptions. Even though these steps or stages are interrelated, in practice they are iterative, i.e., they are not linear.
The research team included reliability procedures in the study such as collaboratively checking data to make sure that they didn’t contain mistakes made during collection or organization of data. Also, the team held regular documented meetings and shared analysis materials throughout the study. Furthermore, the team incorporated validity strategies to assess the accuracy of findings that included: (1) Triangulation through the use of different data sources of information, (2) Use of detailed descriptions to convey findings, (3) Self-reflection on the bias that the researchers bring to the study provided earlier under the Limitations section, (4) presentation of discrepant information that on occasion runs counter to the themes as explained later under the Implications section, and (5) Spending a prolonged time in the field i.e., each researcher has at least 7 years of professional experience directly related to topic of the study.