Introduction

The COVID-19 pandemic changed the way instructors and students interacted with each other almost literally overnight for many institutions of higher education. Dense Face to Face only classrooms were no longer safe places for students. Of the many challenges universities faced, one was continuing to actively engage students in project-based learning environments where learning hinged on students’ interactions with each other, the instructor and the course content amid a global pandemic. After a brief online only approach, Purdue University resumed Face to Face education in the Fall of 2020 as did about one-half of the other universities in the United States. Along with other universities that returned to Face to Face instruction, we grappled with finding an effective dynamic teaching model that would address the needs of Face to Face students, allow campuses to maintain social distance and not exclude students who were temporarily unable to be in the classroom physically. To facilitate Face to Face instruction in a dynamic and unpredictable Fall semester, where a number of students might be unable to come to the classroom for one or more meetings, the HyFlex model emerged as an appropriate model for our introductory design course based on previous literature (Beatty, 2019; Raes et al., 2020). While recently established, the HyFlex model has many variations and has historically been successful. Literature documenting implementation has been situated in planned environments where students knowingly committed to this approach at the time of enrollment in the program or university. One aspect of this model that instructors struggled with historically was providing students who are participating online with similar experiences as those participating face to face. Some of those challenges included sharing materials, keeping students’ attention, and allowing for group work to create that sense of community (Bond, 2016). Students’ academic success is a key focus of university classrooms and essential for retention (Hamann et al., 2021; Millea et al., 2018). Previous research on the HyFlex model has been primarily implemented in lecture-based environments and summarized in three meta-analyses (Beatty, 2019; Detienne et al., 2018; Raes et al., 2020). These meta-analyses share mixed results such that some studies indicate improved students’ academic performance in HyFlex classrooms (Lightner & Lightner-Laws, 2016), while others show a decrease (He et al., 2015) or no difference (Lakhal et al., 2014; Miller et al., 2013; Rhoads, 2020).

The pandemic provided an opportunity to investigate a variation of the general HyFlex approach we call Interactive Synchronous HyFlex. This model is potentially unique in that students can choose on a daily basis if they would engage in our design thinking course physically in the classroom or remotely online. Each student is in the same section of the course, but the way in which they engage could vary from online to Face to Face daily without advanced planning. Further, all students (regardless of physical location) engage with each other and their instructor synchronously to foster community and teamwork. While this model was developed during the pandemic, this study contributes to understanding what educational approaches may emerge in the new ‘norm’ by contributing to our understanding of how this approach might impact students’ academic performance. To get a better understanding of student academic performance in this setting, we compared a course historically offered as Face to Face only (Fall 2019) to the same course offered as Interactive Synchronous HyFlex (Fall of 2020). The HyFlex version of the course included the same learning outcomes and assignments with only slight adjustments for a blending of concurrent Face to Face and synchronous online delivery using Microsoft TEAMS software running on student devices. While previous literature suggests a HyFlex model was likely to be successful, we have a very active classroom targeting freshmen students in a residential undergraduate degree who were not expecting a HyFlex environment. Based on the unique demographics of our students and university environment, we were concerned about the impacts of this model on academic performance of students. To investigate, we report on two related studies. First, we compared grades from an introductory college level design thinking course offered in two different modalities: Face to Face implemented in the Fall of 2019 and Interactive synchronous HyFlex offered during the Fall of 2020. Second, we more deeply interrogated grade data from the HyFlex course modality (Fall of 2020) to identify if students who came to class physically had grades that differed from students who chose to participate one or more times remotely.

Literature review

HyFlex

HyFlex or Hybrid Flexible Learning Environments began to develop around 2005 according to Beatty (2019). The term “HyFlex” has a broad definition and is used to describe a variety of different approaches. The common theme among these variations is that students have flexibility in how they participate in learning experiences (Hrastinski, 2019; Van Doorn & Van Doorn, 2014). In some cases, a course is offered as a Face to Face course or an Online course where students choose which participation method is best for them prior to the term start and they are in that section of the course with that particular participation mode for the duration of the term (Coates et al., 2004). In other instances, students can choose on a day-by-day basis if they will attend in person or online that day. Online or remote participation also has varying specific methods of implementation from synchronously meeting with the Face to Face peers, to synchronously meeting separately with only remote peers or asynchronously engaging in course material some time during a short period of time (perhaps a day or a week).

The purpose and relationship between the Face to Face learning opportunities and the online learning opportunities can also vary. In some cases, the Face to Face learning experience is the primary delivery method and students who are not able to be there have an online option as a “backup plan”. In other cases, the course materials are online and students can attend class as needed more like a help session or semi-structured office hours/study group lead by an instructor (Ituma, 2011). Motivation to offer HyFlex environments has historically varied from expanding capacity by engaging more learners using the same physical space to maintaining business as usual during emergencies such as campus closures due to local or international events including wildfires and earthquakes (Samuel et al., 2019). HyFlex provides students with additional autonomy. Student autonomy is a key to meeting students basic psychological needs according to Self Determination Theory (Vansteenkiste & Ryan, 2013) and therefore providing autonomy in terms of participation method has been theorized to increase student sense of motivation for learning (Kyei-Blankson & Godwyll, 2010; Malczyk, 2019). Despite the benefit of flexibility and technological competence the HyFlex model offers for students, HyFlex comes with some challenges for instructors who are designing and implementing these HyFlex courses. The main challenge for the instructors is to provide very similar classroom and learning experiences for both groups of students: remote and Face to Face (Binnewies & Wang, 2019; Shek et al., 2022).

Universal principles of HyFlex

Four fundamental principles defining HyFlex environments have been identified by Beatty (2019) which are Learner Choice, Equivalency, Reusability and Accessibility. Learner Choice is a defining feature of HyFlex in that students are the ones who make the choice about how to participate. This is often challenging for instructors who believe they know how to best teach their course materials and their lesson plans include dictating how and when students interact with each other and the instructor. Shifting this level of autonomy from the instructor to the student can present challenges (Weston, 2005). For example, it is hard for instructors to anticipate how many students will be Face to Face and how many will be remote which makes group experiences more difficult to plan.

Equivalency is a critical measure of the similarity between learning activities in Face to Face and online experiences. Regardless of participation choice (Face to Face, online synchronous, online asynchronous), students have the same learning objectives and equivalent learning outcomes. This is particularly challenging as team-based projects in active learning environments assume readily available partners in class (Binnewies & Wang, 2019). Online platforms may provide a level of access to peers in small group breakout sessions, but the logistical setup does require additional steps to ensure success, perhaps involving video conferencing software or collaborative file sharing. Equivalency is further complicated by student autonomy. In a Face to Face only setting, the instructor determines what is the “best” use of classroom time, what outside of class individual assignments look like and how outside of class group assignments are to be completed. In the HyFlex environment, some students may choose online because it is best suited for their learning needs, while others, unfortunately, may choose online for the wrong reasons or lack the self-regulation or time management skills to maintain focus (Raes et al., 2020). Thus, the learning plans may appear to be equivalent, but the student experiences may or may not actually play out that way.

Reusability is related to efficient use of instructional materials such that the same materials can be used in the Face to Face environment as well as in the online environment. Accessing materials may require additional consideration as in the Face to Face only environments handouts, for example, can be brought to class. However, in the online environment, handouts need to be available in digital format via the learning management system (LMS). Recent developments in LMS systems have made file sharing increasingly seamless (Malczyk, 2019; Vilhauer, 2021). Sharing a video in class at just the right time or having a guest speaker requires extra planning to engage online students with the same opportunities to interact with the media or guests. In class discussion prompts, for example, may be reused as online discussion forum prompts with little modification.

Accessibility is the fourth main principle (Foust & Ruzybayev, 2021). Some students are unable to attend Face to Face classes for a variety of reasons such as illness or university related activities that might require transportation. Online environments that are asynchronous require students to have time management skills and self-regulation skills to manage their responsibilities (Alhazbi & Hasan, 2021). Synchronous online experiences require students to be available during the scheduled class time and hardware such as webcams and microphones as well as stable internet connections with significant bandwidth. Accessibility issues surrounding each of these participation modes are different and are essential to consider prior to enabling these participation modalities. The addition of the online participation option in the HyFlex modality presented students with a chance to leverage the benefits of accessing the course online while being physically in the classroom. For example, some students preferred to view the screen share rather than looking at the room projector. Others used the live transcription to improve their ability to follow along with the course conversation.

Challenges and benefits in the HyFlex environment related to student learning

The main benefit of the HyFlex model with respect to student academic performance is that the model provides flexibility for students who have changing constraints such as a dynamic work schedules, challenging family situation, illnesses, or a myriad of other challenges. In addition, according to Miller et al. (2013), the technology implemented to create a HyFlex environment helps promote student performance by equipping students with preparation for a high tech collaborative workplace. With the outbreak of COVID-19, the assumption that learning only occurs when students are seated in the classroom has come under increased scrutiny. The potential for HyFlex to become the ‘new norm’ is an increasing reality which makes us wonder how this modality could impact students’ academic performance. Though recent literature suggests an increasing demand, two main challenges directly associated with learning in the HyFlex model include equity and engagement (Binnewies & Wang, 2019). Equity in the context of a HyFlex environment is mainly related to achieving a balance between content delivered Face to-Face and remotely, which can affect student performance. The engagement concern is related to providing interaction between the portion of the class who is Face to Face and the portion who is remote.

Student academic performance in the HyFlex environment

A literature review on HyFlex course design resulted in discovering primary sources from three existing metanalysis. Dr. Brian Beatty published an e-book titled Hybrid-Flexible Course Design: Implementing student-directed hybrid classes (2019) that provides definition as well as a series of cases in a synthesis HyFlex literature. His book is updated regularly and, as of this review, synthesized the work of 85 related publications. Dr. Raes et al. (2020) published a systematic literature view on synchronous hybrid learning which reported on 47 publications from 37 unique studies. Dr. Detienne et al. (2018) conducted a systematic literature review on benefits, challenges and design guidelines for synchronous hybrid learning synthesizing 46 publications. These three publications served as a filter to identify appropriate research related to student academic performance in a HyFlex environment.

Studies surveyed considered a variety of outcomes from academic performance to student learning to sense of community and used a variety of study designs from quantitative measures of student grades to qualitative investigations of student and faculty experiences. Studies on student academic performance show some mixed results, but generally show the introduction of the HyFlex environment as compared to a Face to Face only environment does not have a dramatic impact on student academic performance negatively or positively. In the years prior to the COVID-19 pandemic, HyFlex offered an alternative educational environment often intended to extend the reach of institutions of higher education through serving a more geographically diverse audience or enabling class sizes larger than the classroom capacity. The HyFlex environment in the following studies was generally part of a program that students purposefully chose to enroll in offered by institutions after careful analysis of the stakeholders’ needs. Our research is situated in a HyFlex environment offered as an emergency response to the COVID − 19 pandemic where students enrolled in programs expecting Face to Face instruction through their undergraduate degree program and may or may not have been prepared for temporary remote participation forced on them by quarantine requirements or at their own discretion such that they could remain in their dorm room to participate in class if they felt like it. Five studies explicitly measuring academic performance of students were extracted from the three published literature reviews to provide foundational structure for our investigation.

Miller et al. (2013) investigated student academic performance in two analyses. First, they compared a Face to Face only version of their course to a HyFlex version of their course. Then they compared student academic performance in the HyFlex course based on student participation defined by three groups: One group of students engaged “live-online” (remote and synchronous), while another group participated Face to Face and the third group online and asynchronous. The large (n = 161) statistics course was primarily lecture based at a public institution. In their study, academic performance of students was measured by gains in a pre to post assessment based on a standardized test of statistical reasoning. While the HyFlex group (consenting students n = 69) gained slightly less than the Face to Face only comparison group, the difference was not statistically significant (consenting students n = 67). Further, they compared the two “live” attendance groups of HyFlex against each other on student academic performance by homework, midterm exam and overall course grade. The Face to Face group (n = 56) performed slightly higher on each of the three measures, but, not significantly as compared to the synchronous online group (n = 21). One limitation worth noting on their study related to our study is that attendance at the individual student level was self-reported, and they suggest more accurate methods of documenting attendance are needed to confirm findings.

Lakhal et al. (2014) also experimented with HyFlex environment where students could participate the entire semester Face to Face, online or alternating based on their interests in an undergraduate course on information systems (n = 439). Similar to the Miller et al. study, Lakhal’s study indicated that there were no significant differences in multiple choice test or exam scores across participation modes. However, they did report that online students performed significantly better on homework scores throughout the semester. Unlike the Miller study, a comparison group taught in a Face to Face only method was not used nor were assessment procedures discussed.

He et al. (2015) created an undergraduate electrical engineering course that was available online via the course learning management system and class attendance was optional, essentially creating a unique version of the HyFlex environment. Students had the autonomy to participate in one of three patterns: Face to Face, online course or alternating between the two modes of participation. Their study was driven by the notion that while previous studies had shown course attendance was essential for learning, these previous studies were conducted in settings where the lecture was the primary source of information. In this study, the Face to Face “lecture” was functionally a supplement to the online source of course information and students were not forced to attend the class meetings. The authors hypothesized that forcing students to attend might negatively impact their motivation. Most students consented to participate (n = 139, 87% of the enrolled students), and most were freshman (73%). Student performance was measured by two mid-term exams and a final exam. Cronbach’s Alpha scores indicated reasonable levels of reliability among the measures. Results from their work suggest that attendance in Face to Face lecture did not increase student performance during the first segment of the course when learning materials were less complex. As the semester progressed and materials became more complicated, attendance did become a significant predictor of performance indicating the online only option was insufficient for many students and the Face to Face supplement was significantly impactful.

Lightner and Lightner-Laws (2016) investigated graduate managerial science and statistics courses delivered in a HyFlex method as a way of mitigating concerns about an achievement gap between online and Face to Face students. Prior to considering the HyFlex approach, they conducted a large review of their students’ historical performance in which they document an achievement gap such that online students did not perform as well as Face to Face students. They note that their student body has largely unpredictable schedules (due in large part to a local military base in which many students were involved) making participation in a Face to Face only course challenging. In their HyFlex pilot and associated study, they blended learning where students (n = 156) could choose Face to Face, online or a combination of both (for example, students could come to class for a lecture on Chap. 3 as well as engage in the course material related to Chap. 3 asynchronously via the online experience – essentially doubling their exposure to that material if they preferred) on a weekly basis to best fit their needs that week. Their version of the HyFlex model reduced or nearly eliminated the performance gaps as measured by end of course grades seen previously at their institution between Face to Face and online student participation.

Rhoads conducted a study (2020) of 81 students from 15 different undergraduate courses that were taught in a Face to Face only mode with the same 15 courses that were taught using a HyFlex approach. Data for analysis included final grade average as a measure of student performance. Students’ grades did not vary significantly with course modality comparing a 16-week Face to Face only mode of participation to a 5-week HyFlex model. Covariates of age and gender were considered, and neither yielded significant differences between delivery methods.

Course context

With conflicts in the literature about the impacts HyFlex environments might have on student academic performance and a variety of variations, we cautiously implemented a model that facilitated Face to Face instruction and engaged learners who needed to be remote on one or more class meetings synchronously. This research effort pragmatically investigates impacts on student academic performance to provide evidence of model functionality and offer insight on future utility of the model. The Purdue Polytechnic introductory design course, Tech 12000: Design Thinking in Technology, is the first course in a minor in design and innovation, a core graduation requirement of the college and satisfies two university core curriculum outcomes which are: Science, Technology and Society and Information Literacy. This course historically has been offered using a flipped format with focus on active learning. According to Roehl et al. (2013), in a flipped classroom, instructors prepare learning activities (such as videos and learning materials) for students to study from home and come prepared for class. Class time is for active learning where students apply what they learned in collaborative work and discussion. As cited in the literature, our design thinking course, engages students in the course content prior to each class session (including videos, materials, quizzes) and class time is used for active discussions and project-based learning. This active learning course engages students in individual, small group and whole group interactions. Section sizes range from 36 to 40 students with approximately 18 sections offered each academic semester.

Learning outcomes for the course operationalize a human centered design approach and require students to be able to write a well-defined problem statement, conduct ethnographic research and synthesize literature to understand a problem, brainstorm, prototype, test and refine a prototype, and communicate informatively and persuasively. The design course is organized around three main projects. “Design Project 1” represents the first quarter of the course. In this project, students interview each other to identify an opportunity to improve their peers’ experiences related to transitioning to college which typically includes time management, nutrition, exercise or navigating campus. Students are prepared for this project during the first week, and spend the following weeks engaging in the design work followed by critique, reflection and the introduction of design thinking skills in the context of the project. “Design Project 2” was the second design project which was about 4 weeks in duration and situated teams of students with similar majors to consider how design fits their academic discipline. Students were challenged to find a problem in their field and create a conceptual prototype as a solution to address the problem. “Design Project 3” was the final project that spanned the last 8 weeks of the 16 week semester. This project challenged students to self-assemble into teams of 3–5 and develop a functional prototype to a problem related to one of the National Academy of Engineering Global Grand Challenges (2021).

Sections were primarily taught by graduate students. Many of whom are graduates of a teacher education program, some of whom are former classroom teachers and nearly all are interested in becoming a university faculty member or a more skilled secondary classroom teacher. A senior lecturer and an associate professor coordinate the course, hire instructors and manage the curriculum as well as teach sections of the course. Instructors are provided with a master course shell in the university learning management system (LMS) where all course content, assessments and schedules are set. The master course shell has evolved with each semester since 2012 when it was transformed to an active learning environment with support from the Purdue University Center for Instructional Excellence IMPACT program. Instructors are oriented the week prior to instruction beginning each semester and meet weekly for 1–2 hours to coordinate, discuss concerns and look ahead to the coming weeks. Based on each copy of the course LMS being made from the master and the weekly instructor meetings, each section provides students with a very similar experience.

Interactive synchronous HyFlex- our model

We define Interactive Synchronous HyFlex model as an instructional model that provides an interactive, engaging, and equitable classroom experience for students regardless of whether they choose to join each class meeting face-to-face or remote synchronously (Mentzer & Mohandas, 2022). The Interactive Synchronous HyFlex model implemented during the Fall of 2020 was piloted first in March of 2020, just prior to spring break, when the pandemic appeared as though it might disrupt Face to Face instruction. Following spring break, the HyFlex configuration functionally continued, but, all students were remote which allowed instructors to become very familiar with managing the online experience. During the summer of 2020, we resumed Face to Face instruction and further developed the Face to Face elements of our HyFlex model in preparation for the Fall. During the Fall of 2020, instructors wore wireless headsets and set up Microsoft Teams for each section of their course. Students were expected to be in the classroom when possible but were permitted to join remotely as needed. All students (both in the classroom and online) were expected to use the LMS software for curriculum access, Microsoft TEAMS for communications (screen sharing, video sharing, audio sharing, etc.) and have a webcam and headset (so each student was visible and audible). This configuration required no additional hardware to be installed in the room and allowed students autonomy to participate Face to Face or remotely as needed for whole group and small group interactions. Students were offered a loaner Chromebook if needed and only one student took advantage of the offer for a few weeks while their device was undergoing repair. Data were not recorded for this study about what devices were used or where/when they were used.

Research design overview

The current investigation consists of two different studies examining student grades. Study 1 compares student grades in a freshman design thinking course offered in two different modalities: Face to Face and HyFlex. Study 2 investigated grades of students in the HyFlex modality for differences based on participation choices - those students who chose to be Face to Face and students who chose to participate remotely on one or more class meetings. A summary of the two studies is shown in Table 1.

Table 1
Full size table

Study 1

Research design

To answer our research question 1, we implemented a quasi-experimental non-equivalent pre and posttest design. Students’ SAT scores served as a proxy pre-test to establish comparability of the groups prior to the semester start. This research design allowed us to investigate the impacts of the HyFlex approach (Fall of 2020) on student academic performance outcomes as compared to the Face to Face only approach (Fall of 2019) to address the first research question: Is the academic performance of students enrolled in a Face to Face version of the course different from students who enrolled in a HyFlex version of the course?

Variables

SAT scores were pulled from university data sources. With the course being a 100-level course, most students were in their first year and therefore, the SAT is a relatively recent and standard measure of academic preparation. We used SAT as a measure to indicate the academic similarities between groups as “pre” test measure. While most students recently took the current version of the SAT exam, a number of students took the previous version of the SAT while others took the ACT exam. Published concordance tables (The College Board, 2009) were used to convert previous SAT and ACT scores to equivalent current SAT scores. The highest of the SAT or SAT equivalent score available for each student was used in this study as is referenced to hereafter as the “SAT score”.

Demographics were also collected by the research team from the University’s application database for analysis which included gender, ethnicity and residency. Class rank was pulled from the University’s database as well. Gender was reported by the University as binary (Male vs. Female) and analyzed as such. Non-responses were not considered in the analysis. Ethnicity was collapsed into white and non-white as research in the STEM fields frequently uses this distinction for under-represented minorities vs. overrepresented. Residency for this analysis was domestic or international. Class rank was measured by credit hours earned per student. Note these ranks include credits earned prior to enrolling at Purdue University and in the Polytechnic college. This can be misleading as the sophomores by credit hour may be “freshmen” by experience as they have transferred multiple dual credit high school courses that may or may not count toward graduation.

Academic performance was measured with student grades which are a common measure of student success. According to an analytic literature review on defining and measuring academic success in higher education, York et al. stated, “Academic performance in the form of academic achievement, accomplishment of learning objectives, and acquisition of skills and competencies were the most frequently measured aspects of academic success” (2015, p. 7). Further, York continued, “…Academic achievement is almost entirely measured with grades (by course or assignment) and GPA” (2015, p. 7). Four measures of student academic performance were considered to answer each research question based on four significant elements of the course structure: semester grade, Design Project 1, Design Project 2 and Design Project 3. Figure 1 illustrates the timing of each academic performance measure relative to the semester duration of 16 weeks.

Fig. 1
figure 1

Course Timeline

Full size image

The overall semester grade included all graded assignments and any extra credit opportunities available to students. Extra credit typically did not exceed 4% of the overall course grade (which was approximately equal to missing one class session). Extra credit was offered for participation in research opportunities and the end of course showcase which featured a few top projects competing for funding. Design Project 1 summarized grades earned during the first quarter and serves as a context for the rest of the term. Design Project 2 spanned work done in approximately the second four weeks of the term (immediately preceding the final project). It included both individual and group assignments, and is significant because it was the first-time students are assessed on their more thorough understanding of design thinking as they engage in the design process related to their major field of study. Design Project 3 spanned the last eight weeks of the course, and included individual and group-based assignments. Some group assignment grades were modified by a measure of team member contribution (CATME) to more accurately reflect the contribution of each student such that students who contributed less to the project received a modified score that was less than other students for the same product submission.

Design Projects 1, 2 and 3 were evaluated through a number of rubrics to measure components of the design process such as problem statement, benchmarking, brainstorming and so forth. Rubric assessments as measures of student academic performance can be validated in a variety of ways including statistical testing or stakeholder validation (Dawson, 2017). Aligned with the work of Timmerman et al. (2011), we validated our rubrics through comparisons with existing rubrics and consulting pedagogical experts and iterative development with feedback from stakeholders. Rubrics for individual components of each design project were informed by the EDPPSR rubric. The EDPPSR rubric is the result of 10 years of NSF funded work and was validated through the efforts of expert workgroups which have applied and refined the rubric on senior capstone high school student portfolios (similar to college freshmen level efforts) (Goldberg, 2011; Groves et al., 2014). The EDPPSR rubric focuses on measuring a student’s ability to engage in key science and engineering design practices including: identification of a significant challenge, specification of requirements, imagining and selecting a promising solution, defining and evaluating prototypes, reporting findings and drawing conclusions, and lastly, reflecting and recommending iteration or implementation. These essential practices are aligned with the course learning outcomes and University undergraduate outcomes-based core curriculum. As additional evidence of validation, our rubrics were developed through a collaborative process with the guidance of the pedagogical experts at the Purdue University Center for Instructional Excellence using the Backward Design Process (McTighe & Wiggins, 2004). Over the last 11 years, a team of faculty and graduate students studying education have evolved the rubrics based on feedback from students, observations of alignment between the rubrics, the educational submissions of the students and the course learning outcomes. Students have access to the rubrics for each assignment prior to beginning the work so they can understand the learning intentions (Wiliam, 2011). Rubrics are used for both individual and group work and assignments are graded quickly to help inform students about their learning in preparation for the next assignment. Rubrics and instructions were the same across sections with negligible changes between years. Graduate students were responsible for assigning grades. To provide a level of fairness and standardization, graduate students were mentored weekly during instructional team meetings. In some sections, the same graduate student taught as graded. In other sections, a graduate student or senior lecturer taught while another graduate student graded. With this paring, the grader and instructor meet weekly to maintain consistency (Dawson, 2017).

Pre-Analysis similarity check

Prior to analysis, the two groups (Face to Face only semester, Fall 2019 vs. HyFlex semester, Fall 2020) were “pre-tested” for comparability to establish the extent to which they were similar at the onset of the academic terms. As shown in Table 2 grade data were available from all students. The university had SAT data from 85% of students. Demographic data were available for nearly all students (~ 94%).

Table 2 Fall 2019 Face to Face only course experience and Fall 2020 HyFlex course experience data available
Full size table

Table 3 shows that the instructional team was reasonably similar during both semesters with about half the instructors continuing from the Fall of 2019 to the Fall of 2020. Of the other half, some of the instructors were different between terms such that the Fall of 2019 instructors had more teaching experience and were replaced by newer instructors during the HyFlex semester who had less teaching experience. Relevant instruction preparation shown in Table 3 includes instructor participation in a four year teacher preparation program, public school teaching experience and previous experience teaching this course.

Table 3 Instructor information for comparison and treatment semester
Full size table

Demographic data were compared between the two groups and are shown in Tables 4, 5, 6 and 7. Student distribution across class rank, ethnicity and residency were significantly different between the Face to Face only experience (Fall 2019) and the HyFlex experience (Fall 2020) as shown in Table 4.

Table 4 Result of Pearson χ 2 of Demographics in the 2019 (Face to Face only course) and 2020 (HyFlex course)
Full size table

Class rank, ethnicity and residency were different between the terms which may be explained by two key factors. Advanced placement testing was difficult to access and often delayed in 2020 due to the COVID pandemic which impacted the number of students who arrived on campus with dual credit. Thus, the number of first year students might have been comparable which the number of credits they brought to campus might have been lower. Data on which students were in their first semester were not available. Differences in ethnicity and residency may have due to VISA and entry restrictions to the US from international students in associated with COVID. Though demographics showed significant difference, SAT scores, as a measure of academic preparedness, did not vary between the Face to Face course and HyFlex course as shown in Table 5 and therefore we continued with the analysis.

Table 5 Independent T Test of SAT score in 2019 and 2020 semester
Full size table

Results

Descriptive statistics are provided in Table 6 for Face to Face only vs. HyFlex for four significant evaluation periods – the overall semester grade, Design Projects 1, 2 and 3. As shown, generally, grades had a wider range during the HyFlex semester with lower low grades and higher high grades. Generally mean scores were similar between Face to Face only and HyFlex environments.

Table 6 Descriptive Statistics
Full size table

Normality tests were run on the outcome variables which were each determined to be significantly different from a normal distribution as shown in Table 7. According to Field (2009) and Anderson (2010), due to the central limit theorem, violations of normality are less concerning with larger sample sizes therefore we interpret our findings by triangulation both analyses.

Table 7 Outcome Variable Test for Normality
Full size table

To answer our research question about the impacts on learning related to Face to Face only course experiences and HyFlex course experiences while being cognizant of the balance between normality violations and sample size, we ran both t-tests and non-parametric Mann-Whitney U tests. While not typically run in concert, parametric and non-parametric tests served different purposes for our analysis. The T-Test (shown in Table 8 indicated that overall semester mean scores were not significantly different for the Face to Face only vs. HyFlex course offering modality. One significant difference is worth noting – the mean score of the Design Project 1 was significantly higher in the HyFlex approach. Non-parametric Mann-Whitney U test results are shown in Table 9 and indicate that mean rank scores were significantly higher for each of the measures during the HyFlex implementation.

Table 8 Independent Samples T Test of Fall 2019 (Face to Face only) vs. Fall 2020 (HyFlex)
Full size table
Table 9 Mann-Whitney U Test of Fall 2019 (Face to Face only) vs. Fall 2020 (HyFlex)
Full size table

The apparent conflict between the T-Test results and the Mann-Whitney U test results lead us to run a Chi-Square test on grade distribution (results shown in Table 10) as well as a visualization (Fig. 2).

Table 10 Result of Pearson χ2 of Grade Distribution Fall 2019 (Face to Face only) vs. Fall 2020 (HyFlex)
Full size table
Fig. 2
figure 2

Grade distribution during the Fall 2019 (Face to Face only) and Fall 2020 (HyFlex) semesters

Full size image

Grade distribution is illuminated by the significant differences shown in the Chi-Square test which indicate that the ratios of students who earned each grade (A-F) are significantly different in the two learning experiences. Descriptive statistics and Fig. 2 show that in the HyFlex model, more students earned a grade of A and more students earned a grade of F than in the Face to Face only model which explains why the mean scores are similar yet the mean ranks are significantly different.

Study 2

Research design

A quasi-experimental design was also used to in study two but with a subset of the sample from study 1 - specifically, we limited analysis to students in the HyFlex course from whom attendance data were available to investigate differences in academic performance based on student choice of participation modality. The second research question was investigated through a comparison of students in the HyFlex approach who chose to participate remotely during one or more class meetings with students who opted not to participate remotely during any of the class meetings. Driven by the research question: When a course is offered as HyFlex, do students who consistently choose to participate in that course face to face have different academic performance than those who choose to participate once or more remotely?

Variables

Along with the variables used in study 1 (SAT, Demographics and Academic performance), attendance data of students from the Fall 2020 (HyFlex semester) were also collected. Attendance data provided information about how each student chose to participate daily which was recorded by the instructor or an undergraduate teaching assistant as: Face to Face, Remote or Absent. To understand how students’ attendance modalities might impact academic performance, two groups were formed from attendance data. One-hundred and thirty-seven students in the HyFlex course had the option of participating remotely, but, never took advantage of the flexibility. This group of students exclusively participated face to face (or were absent from class). Three-hundred and eleven students took advantage of the Highly Flexible attendance policy and chose to participate remotely during one or more class sessions. Note that the actual number of days remote (if it was greater than 1) was not considered in this analysis nor was the number of days absent from class. The University modified the semester calendar such that all students were remote from Thanksgiving break to the conclusion of the term (2 weeks or 4 meetings). These “remote” participating days for everyone were ignored from the determination to classify a student had chosen to participate remotely or not during the semester as students did not have a choice and the majority of the course experience was completed. Attendance data were available from 14 of the 19 sections at the time of analysis.

Pre-Analysis similarity check

Class rank, gender and ethnicity showed no significant differences between students who chose daily not to attend the HyFlex remotely vs. those who choose to attend one or more class sessions remotely.

Table 11 Result of Pearson χ2 of Demographics for Fall 2020 based on student choice to not participate remotely or to participate remotely one or more class sessions
Full size table

Residency did have a statistically significant difference such that international students participated once or more remotely at a higher rate than their domestic peers. While statistically significant, the practical significance of a 2% difference is negligible and therefore we continued with analysis. As with the demographic makeup, students who chose to be remote once or more were not significantly different in their academic preparation as those who had the option to be remote but choose not to use it. Results of this SAT score comparison are shown in Table 12 and indicate that students choosing 0 days remote were not different than those choosing 1 or more days remote.

Table 12 Independent T Test of SAT score in 2020 semester Face-to Face and Remote
Full size table

Results

Descriptive statistics based on student choice is shown in Table 13. Students choosing to participate exclusively Face to Face students had slightly more extreme grades for the semester with lower low grades and higher high grades. Mean scores were slightly higher among students who did not participate remotely compared to students choosing 1 or more days remote.

.

Table 13 Descriptive Statistics for HyFlex students choosing to participate remotely on more or more class sessions and those who exclusively choose Face to Face participation
Full size table

Normality tests were run on the outcome variables which were each determined to be significantly different from a normal distribution as shown in Table 14. Significantly deviating from normal with small sample sizes suggests the use of non-parametric statistical analysis.

Table 14 Outcome Variable Test for Normality
Full size table

A similar analysis approach was followed for the second research question related to student academic performance in the HyFlex approach based on attendance habits. Students who did not participate remotely were compared to students who participated remotely on one or more class meetings. Though Table 14 shows a significant deviation from normal, T-tests were run based on the large sample sizes as rationalized in study one. Table 15 shows no significant differences among the means of the four grade measures considered between the two participation modes. Table 16, on the other hand, highlights non-parametric analysis indicating that students who did not participate remotely performed significantly better than students who chose to participate remotely on one or more occasions on semester grade, Design Project 2 and Design Project 3.

Table 15 Independent T Test of 2020 HyFlex Face to Face vs. Remote
Full size table
Table 16 Mann-Whitney U Test of HyFlex Fall 2020 Face to Face vs. Remote
Full size table

Figure 3 visually illustrates that Face to Face students earned more grades of A and more grades of F than did their remote counterparts. Table 17 confirms the visual differences of the grade distributions are statistically significantly different based on a Chi-Square analysis.

Fig. 3
figure 3

Grade distribution during the Fall 2020 (HyFlex) experience between students who were exclusively Face to Face (did not participate remotely) vs. those who chose to be remote one or more meetings

Full size image
Table 17 Result of Pearson χ2 of Grade Distribution HyFlex Fall 2020 Face to Face vs. Remote
Full size table

Discussion

The criterion referenced rubric measures from our course result in semester grades that indicate students are generally successful in demonstrating high levels of competence in the course with mean scores between 88.5% and 92.3% for the eight means compared in this analysis as shown in Tables 6 and 13. Table 18 presents a summary of the comparisons and findings for each comparison highlighted.

Table 18 Summary Table of Results
Full size table

During the Fall of 2019, the course was offered by an experienced team of instructors in a Face to Face only format. During the Fall of 2020, the course was offered by a slightly less experienced team (a few experienced instructors from the Fall of 2019 were replaced by new instructors who were both new to the course and new to teaching in general) in a HyFlex approach. Students were required to participate at the scheduled course time in person or synchronously online. Active learning strategies engaged Face to Face and synchronous online learners together and in similar ways. Students had the option of participating how they preferred on a meeting by meeting basis. Some students even joined remotely for a few minutes while on their way to the Face to Face meeting instead of being late due to commute times across campus. In comparing the Face to Face only approach to the HyFlex approach, student academic performance (measured by end of semester grades) was different where the median scores in the HyFlex approach were significantly higher. Additional analysis suggested that while mean scores between the two approaches were similar, grade distribution was different. Students in the HyFlex semester earned a greater percentage of the A’s and a greater percentage of the F’s. During the Fall of 2020 in the HyFlex environment, students were encouraged to participate Face to Face unless they were ill or exposed to someone who was ill. For students in communal living environments such as Fraternities and Sororities, long quarantines were common. Many students were cautious and transitioned to remote for a meeting or two while waiting on COVID-19 test results prior to resuming Face to Face instruction based on a sore throat or other mild potential COVID-19 symptom. We do know that not all remote students choose to be remote for COVID-19 related reasons. Some students were remote due to a flat tire or difficulty finding a parking space or they just didn’t feel like walking across campus for class. Regardless of the reason, there were 367 students of 483 (76%) who chose to participate online one or more times during the semester. Overall semester grades for students who were online one or more times were compared to students who did not choose to participate remotely. Mean differences were minute and insignificant practically and statistically, but median scores and grade distribution were significantly different. Face to Face student earned a greater percentage of the A’s and F’s than did their remote counterparts.

Situating these findings in the literature is challenging due to the unprecedented nature of COVID-19. Existing literature has suggested that attendance is correlated with student success in comparison to missing class (Miller & Baham, 2018). Literature is mixed on the impact HyFlex has on learning and seems to vary based on context. In situations where students are unable to participate Face to Face, participation online is beneficial such that it is better to be online than absent (Green, 2021). Other studies suggest that in situations where a course is designed to be online and students can attend a Face to Face meeting to supplement their understanding, students do significantly better (Coman et al., 2020). Most of the existing studies were situated in planned environments. Students knew they were enrolling in a program that offered a version of the HyFlex model when they committed to the program (Coates et al., 2004). In the Fall of 2020, students in this study signed up for a residential program and were presented with, and perhaps even surprised by, the option of participating online as needed at the start of the semester.

Previous work identified that substantial time management skill and self-regulation skills required for students to be successful in an online environment (Alhazbi & Hasan, 2021). We celebrate that the percentage of students earning an A grade actually went up slightly during the pandemic and those earning an A or B (88.2%) was very similar to the pre-pandemic numbers (89.2%). The interpretation of this is complex, however, as we do not know if HyFlex works equally well for all students regardless of a pandemic as hinted by the number of students who failed the course jumping from 1.6% in the Face to Face only environment to 3.6% in the HyFlex environment. The range of scores indicates that, in the Face to Face only environment, no students earned a zero in the course - either because everyone performed to some extent or dropped the course prior to the term end. In the HyFlex environment, the range indicates that some students actually received a zero in the course. Data are not available to explain to what extent the HyFlex experience, the pandemic or access to advisement may have played a role the students’ decisions not to drop the course while not participating. Semester grade data indicate that of the six students earned less than 5%, each was enrolled during the HyFlex semester. Fourteen students earned a semester grade of 25% or less in the course, and of those, only one was from the Face to Face only semester (2019). Worth noting, available attendance data for these 14 students indicate they only participated in about 35% of the course meetings and, when they did, some were Face to Face exclusively and some were remote one or more times. If these very poorly performing students would have normally dropped the course, but, remained enrolled due to extreme lack of engagement with the University (so disengaged they didn’t seek advisement from an advisor related to dropping the course or respond to inquiries from their course instructor), potentially even stronger results may have been masked by these extremely low scores resulting in an even more positive outcome related to the HyFlex course offering. We also speculate that students may have been more tolerant of the well documented technical challenges of the HyFlex model or students may have been so socially deprived that they eagerly participated Face to Face when possible and online as needed. In the absence of a global emergency, we wonder if the option to participate remotely will lure students who are less engaged into an online environment that demands additional self-regulation, intrinsic motivation and time management skill.

Implication for the classroom

We used these findings to revise the course offering during the 2021–2022 academic year. We continued the HyFlex participation options in much the same way we did during the Fall of 2020. Based on data here, we anticipate student academic performance to continue to be high as usual in the HyFlex environment for most students. However, we are cautiously optimistic about students’ ability to make good choices about their attendance habits as shown by the increased failure rates (which seem to be more related to participating itself than how students are choosing to participate). Given the vaccine successes and COVID mutations, infections rates are unpredictable for the immediate future, though precautions are relaxing toward more normal life experiences. As students are able to engage socially in more normal and fulfilling ways, we wonder if the motivation to attend class physically will drop and the option to attend remotely will look overly tempting?

For students who are unable to attend class due to illness or university sponsored travel, participating remotely is likely to be far more beneficial than missing class. Students who were unable to participate even remotely or those who may have forgotten something discussed in class or benefit from review have the option of accessing the class recordings. The flexibility of how to attend class daily may yield additional benefits such as reduced travel time to the classroom that can be used more productively. We speculate that a HyFlex approach such as ours may have affordances particularly advantageous for learnings with disabilities. Anecdotally, a few students did share with instructors that they were struggling with migraines, anxiety, and other mental health issues and, the ability to participate remotely provided them with a safe space to engage with the course as needed. Further, Face to Face students were observed leveraging the screen share, transcription tools and chat features afforded by the HyFlex approach. In addition, to foster a HyFlex modality, we incorporated tools such as Google Jamboard as an alternative to the traditional place-based whiteboards to foster collaboration. Therefore, the autonomy to attend remotely or in person as needed and, regardless of participation mode, access to screen sharing, live transcript and recordings may be useful to students with disabilities.

There are costs, however, for both students and instructors that need to be considered. For instructors, even though there are few initial challenges related to software learning curve and making the course content equitable for different modes of student participation (Romero-Hall & Ripine, 2021), the HyFlex option can potentially allow students who would otherwise be absent to participate. Hardware and software are required. In this study, Microsoft TEAMS was used which required minimal setup and a few button clicks to schedule a meeting that the instructor and students could join and record automatically. However, the software, like other software platforms, competes for valuable CPU power with other programs jeopardizing the speed of our lightweight laptops and draining their batteries. Students in the classroom can only communicate with online students if they too run the software and use headsets – thus – the burden generated by adding online participation as an option is partially shouldered by students who have made the journey to the classroom. We wonder how tolerant Face to Face students will be with their peer who “don’t feel like coming to class” today for no apparent reason. Students who have options on how they wish to participate do have the autonomy to make bad choices. Perhaps the ability to navigate the course online is a skill that requires development and support. We ponder, to what extent is it the instructor’s responsibility to ensure the educational environment is conducive to learning for each individual student based on their needs? Students may choose to be remote without realizing it might be overly tempting to succumb to distractions in their lives such as roommates, their phones, or attempt to participate from locations that do not afford the bandwidth to maintain adequate audio and video communication. If students chose to participate in the HyFlex course by physically attending class, they can be redirected when they are distracted. If they participate online, the instructor’s ability to notice their off task behaviors and redirect may be limited.

Further research

Our research answers the research questions but prompts us to wonder about additional complexities. We compared students who chose to be remote one or more class meetings against those who did not choose to participate remotely during the semester. We have not investigated actual participation in during class time. The number of meetings where students were absent in relation to the number of meetings they were present in the classroom or remotely could be potentially enlightening as some students who were not remote were also not in class. Conversely, some students who were remote one or more meetings, may have had significant absences on the other meetings. Plotting attendance data as a frequency count based on the number of days remote showed a positively skewed continuum where most students were Face to Face most of the time. The number of students who participated remotely tapered off quickly as the number of days being remote increased. In only a few cases did students attempt to treat the course as an online only course and attend the entire semester or much of the semester online. Investigating student motivation to choose to be Face to Face or remote on any given day could be helpful to inform the design of future classroom expectations and policies. We hope students are making the best choice for them based on optimal learning, but students may be valuing other criteria such as convenience or the opportunity to travel and attend class from home (saving money, but potentially lacking the structure of a daily University routine) or the beach (providing overwhelming distractions and potentially poor internet bandwidth). While not obvious in our analysis, we wonder if there is a correlation between attendance patterns and student success that may or may not be linear and may be bimodal. For example, perhaps, very academically focused students may participate remotely instead of being absent which yields significant benefits for them and saves their instructor’s time because they don’t have to help get students caught up or create alternative assignments to compensate for their absence. Students who are easily distracted may choose to participate remotely so they can be in a quiet environment without peers which may allow them to focus on the course more intently without classroom-based distractions. Less academically focused students who choose to be remote may engage in other activities during class time while attempting to participate in class concurrently resulting in not being completely focused on class. Thus, we envision investigating the potential of a bimodal distribution where the HyFlex option has noticeable learning gains for some students while reducing the learning of other students. Future research might consider the impact of attendance patterns on student academic performance and how device or WIFI access impacts success. For example, our attendance records show some students started the semester by participating remotely and then transitioned to be in the classroom physically, while others did the opposite. Some students were remote in what appeared to be random patterns of a day here and a day there while others were remote only at one point in the term and only for 1–2 weeks which was consistent with quarantine requirements. These kinds of attendance patterns may represent different types of students with different needs and could be worth categorizing for further analyzing data on how remote options support or distract these learners.

Limitations

The instructional team was less experienced in the Fall of 2020 due to a few experienced instructors transitioning away from teaching who were replaced by new instructors. This may have masked some of the benefits of the HyFlex environment as first year, and especially first semester as instructors are developing their teaching skill set. Further potentially masking the impacts of the HyFlex environment may have been the substantial number of students who catastrophically failed the course (course grades of 25% or less) but did not withdraw from course prior to semester end. There may also have been a network effect impacting student choices that we were unable to account for such that if a student was scheduled for another course directly before or after our course, and it was online, they may have been “forced” to participate in our course online due to transportation issues beyond their direct control. When students are physically on the academic side of campus, 10 min between classes is minimal but sufficient to travel between classrooms. When students are in an apartment, at home or in the dorms participating in another course that is online only, having 10 min to get to the next class may be difficult. Further, we did not measure instructor influence on student participation choices. While our general message to students was that we expected them to be in class Face to Face if they were able, some instructors were more adamant and vocal that their students must be physically in the classroom unless they are sick with documentation while others were just glad students participated, regardless of the modality.