Introduction

Since 2019, school districts across the United States and globally have experienced overwhelming challenges as they attempted to flexibly transition between in-person and remote instruction. As schools shut down globally in the spring of 2020, more than 1.6 billion students had to make rapid adjustments to their learning routines as they attended classes remotely (UNESCO, 2020). Remote learning, also known as online learning, distance learning, and virtual learning, occurs when students and teachers must rely on technology to engage in instructional activities (Moore-Adams et al., 2016). Although some remote learning occurred prior to the pandemic, the majority of students and teachers had little to no experience with online learning or instruction until the shutdown occurred (Shamir-Inbal & Blau 2021).

The transition from traditional classroom instruction to online learning during a pandemic has affected students and teachers in many ways. Students and teachers had to cope with increased stress, anxiety, and fear as they remained socially isolated from peers at home (Son et al., 2020). Online learning was not possible for many students who did not have access to necessary electronic devices and reliable Internet (Di Pietro et al., 2020; Graves et al., 2021; Tadesse & Muluye, 2020). Even when students were able to attend online classes, many teachers struggled to provide instruction due to inexperience, burnout, and insufficient technology and pedagogical support (Huck & Zhang, 2021). Instruction time decreased overall, and students received less direct support from teachers and peers (An et al., 2021). Thus, students were also likely to be less motivated to learn and participate during class (Ferri et al., 2020).

These challenges may have negatively influenced students’ learning in a variety of different content areas. Although researchers have made predictions about students’ learning trajectories during the pandemic (Bao et al., 2020; Kuhfeld et al., 2020), actual data on students’ learning trajectories are needed. In this study, we evaluated kindergarten students’ inference-making learning trajectory as they completed the Early Language Comprehension Individualized Instruction (ELCII) intervention (McMaster et al., 2019; Butterfuss et al., 2022) before and during the pandemic (specifically, in the 2020-21 academic year, when schools provided a range of in-person, hybrid, and online learning). Administering ELCII to two different cohorts of kindergarteners pre- and in-pandemic provided a unique opportunity to examine how the changes in instructional modality and the overall pandemic environment may have influenced students’ learning of how to make inferences. In this context and because technology is a core component of online learning, we sought to understand whether that core instructional components of the intervention, scaffolding and feedback, supported student learning. It is important to note that due to the pandemic, we were unable to control for several differences in between the samples, educational environment, and ELCII implementation across the two cohorts (see Method section for a transparent overview of differences). In fact, it was our ethical responsibility to adjust implementation of the intervention given the pandemic to ensure no harm to participating students (e.g., following guidelines for social distancing and remote learning; Chenneville & Schwartz-Mette, 2020). Thus, the results of this study only provide preliminary evidence for the extent to which scaffolding and feedback support students’ learning to make inferences. Still, we argue that the dearth of data evaluating young students’ learning of foundational skills such as inference-making during the pandemic in conjunction with the changes that it introduced to educational systems worldwide make this investigation worthwhile. The findings of this “natural experiment” (Skar et al., 2021) could provide some insight as to how teachers and researchers could leverage educational technologies such as ELCII to provide quality instruction to students in online environments.

Early language comprehension individualized instruction

ELCII is a cloud-based software application that provides Tier 1 (core general classroom) inferencing instruction to kindergarteners. Inference-making is crucial for reading comprehension because it allows students to fill in gaps that are not explicitly stated in the text (Oakhill, 1984). To make inferences, students must connect words and sentences they are currently reading or hearing in a text with information from prior knowledge, including information learned from prior text as well as general world knowledge and episodic memory (Kintsch, 1988). Researchers have found inference-making to be a significant and unique predictor of reading comprehension, and studies indicate a causal link between poor inference-making and poor reading comprehension outcomes (Kendeou et al., 2008; Cain & Oakhill, 1999, 2006, 2011; Cain et al., 2004; Oakhill, 1982, 1984; Oakhill & Cain, 2012). These findings suggest that teaching students to make inferences is important in improving their reading comprehension outcomes.

ELCII is grounded in the Inferential Language Comprehension Framework (ILC; Kendeou et al., 2020), which posits that inference-making is a domain-general skill that can transfer across different contexts and media. Thus, different types of media such as videos can be leveraged to teach inference-making in young children. In ELCII, instead of having students read texts and make inferences, students watch videos to make inferences. By using videos, ELCII can teach students how to make inferences independent of their decoding skills. ELCII consists of 20 instructional modules and 20 read-aloud lessons. In each instructional module, students (a) learn three vocabulary words necessary to understand module content, (b) watch an age-appropriate fiction or nonfiction video, (c) answer five inferential questions at different points in the video, and (d) receive scaffolding and feedback tailored to their answers.

Introduction to ELCII. Before interfacing with the learning modules, kindergarteners are introduced to ELCII through a researcher designed e-book, ELCII’s Mission to Earth, that introduces an alien peer tutor aptly named ELCII. Students learn that ELCII has been sent to earth to teach them how to make inferences. ELCII also teaches students what an inference is and provides them several examples of inferences throughout the story.

Instructional Modules. Immediately before students begin the instructional modules, ELCII teaches students how to navigate the platform and engage in each intervention module, including how to use features such as replay and pause buttons. In each instruction module, students watch a fiction or nonfiction video approximately five minutes in length. Each fiction video is a truncated episode of either Shaun the Sheep or the Berenstain Bears television shows (Berenstain & Berenstain, 2003–2004; Park, 2007–2020). Nonfiction videos were developed by the ELCII team and consist of short documentaries about social studies or science topics based on core academic standards. Before watching each video, ELCII provides students with a preview of its content and definitions for the three key vocabulary words used in the video. As students watch the video, they are asked five inferential questions at pre-specified segments of the video where an inference must be made to understand its content. All questions are multiple-choice with four answer options.

Individualized Scaffolding and Feedback. An important instructional component of the instructional modules is the built-in individualized scaffolding and feedback provided to students who need help making inferences. Scaffolding and feedback is defined as information provided regarding one’s performance that is meant to reduce the distance between current and intended performance (Hattie & Timperley, 2007). For several decades, researchers have recognized that inference instruction that provides scaffolding and feedback can increase students’ inference-making skills and overall comprehension performance, both in online and in-classroom contexts (Brownfield & Wilkinson, 2018; Dole et al., 1991; Hansen, 1981; Meyer et al., 2011; Neumann, 2020; Rodgers et al., 2016; Swart et al., 2019, 2022; Yuill & Oakhill, 1988). For young K-2 students, popular instructional strategies included explicitly teaching students to make connections between prior knowledge and the text, practice in answering inference questions, generating questions, and making predictions about events in texts (Hansen, 1981; McGee & Johnson, 2003; Palincsar & Brown, 1984; Yuill & Oakhill 1988).

In general, the type of scaffolding and feedback that could be provided to students in literacy instruction varies widely (Brownfield & Wilkinson, 2018). However, research has suggested the importance of providing individualized scaffolding and feedback in supporting inference-making (Pérez-Segura et al., 2022; McMaster et al., 2014; Swart et al., 2022). In order to maximize the benefit of scaffolding and feedback, timing of when to provide scaffolding and feedback should be individualized to help students benefit from it. Immediate scaffolding and feedback may be more beneficial to students’ learning rather than delayed feedback (Carnine et al., 1982; McMaster et al., 2014, 2014; Meyer et al., 2011; Rojas et al., 2019; Swart et al., 2019, 2022). In addition to the timing, scaffolding and feedback may be more beneficial when its content is individualized to be contingent to students’ needs (Wisniewski et al., 2020). For example, Rodgers and colleagues’ (2016) in-depth analysis of teachers’ scaffolding in an early literacy intervention suggests that teachers of students with higher learning outcomes tend to accommodate to students’ knowledge needs and guide students to the appropriate information sources.

To support inference-making, scaffolding and feedback in the present study was individualized by providing students with immediate feedback and scaffolding of which content was contingent to their responses. The goal of scaffolding and feedback was to support students to activate prior knowledge or other information presented earlier in the video and integrate that information with the information they were currently encoding to generate accurate inferences (Oakhill, 1984). Specifically, the content of the feedback was contingent to students’ responses. When students answer a question correctly, a pedagogical agent (i.e., ELCII) explains why their response is correct. When students answer a question incorrectly, ELCII provides the student with immediate feedback that their response is incorrect and the reason their response is incorrect. Following the feedback, students also receive two pieces of scaffolding, which consist of segments of the video that contain the information necessary to make the targeted inference. ELCII repeats the information contained in both scaffolding pieces, then asks the student to try again to connect the pieces to make an inference. If students’ second response is still incorrect after feedback and scaffolding, ELCII explains again why the response is incorrect and why the correct response is correct. When students do not understand how to answer the question and would like to leverage scaffolding immediately (rather than after they respond), they are provided an option to instantly receive scaffolding.

Based on prior studies and reading comprehension theory, the individualized scaffolding and feedback component in ELCII should support students’ learning of inference making in remote contexts. To our knowledge, this study is also the first study to assess the role of scaffolding and feedback for inference-making in an online environment for kindergarten students.

Read-Aloud Lessons. In addition to the instructional modules, the ELCII team developed 20 read-aloud lessons for teachers so that they can further guide students in learning how to make inferences in the context of book reading. Read-aloud lessons were designed to mirror the structure of the instructional modules. In each read-aloud lesson, teachers read an age-appropriate fiction or non-fiction book to their students. Prior to reading, teachers provide three vocabulary words. Teachers also ask five inferential questions during reading, and provide scaffolding and feedback based on students’ responses. In providing the read-aloud lessons, teachers model inference-making to the class for the first inference question and then ask students to make inferences for the remaining questions. Students are encouraged to work with a partner to answer the questions and then share their responses with the class. All questions are open-ended.

Rationale for the study

Changes in and interruptions to students’ learning environments during the pandemic have prompted deep concerns about the learning challenges that students are experiencing as a result (Spitzer & Musslick, 2021). Simulations of students’ learning based on past data in the United States have indicated that kindergarten students as well as students in third through seventh grade would show lower rates of growth in reading (Bao et al., 2020; Kuhfeld et al., 2020). Indeed, an analysis of reading and mathematics assessment outcomes for more than 1.6 million K-12 students across 40 states in the U.S. indicates that students’ scores were lower during the pandemic than they were in previous years. Early analyses of students’ learning progress during the first academic year of the pandemic in other countries also appear to corroborate these findings (Donnelly & Patrinos, 2021; Skar et al., 2021). While many studies provide support for a general “learning loss” phenomenon (Skar et al., 2021, p. 1), studies also provide a more nuanced perspective of student learning during the pandemic.

Contrary to expectations, several other studies have not reported ‘losses’ in academic performance for students during the pandemic. For example, studies evaluating international university students’ academic performance in various disciplines (e.g. science, engineering, business) revealed that students’ performance did not differ or even improved during the pandemic compared with their performance before the pandemic (César Vargas-Ramos et al., 2022; Iglesias-Pradas et al., 2021; Limniou et al., 2021; El Said, 2021). Additionally, Spitzer & Musslick (2021) reported that German students (grades 4–10) participating in an online learning environment for mathematics showed improved mathematic performance during the pandemic compared to their performance in the year before. According to these studies, whether or not the pandemic influenced students’ academic outcomes depended on a variety of factors, including but not limited to students’ digital skills, access to necessary support and tools, and the quality of online instruction. Unfortunately, low-income, Black, and Latino students as well as multilingual learners and students with learning disabilities are less likely to have access to technology and support necessary for participating actively in online environments (Dorn et al., 2020; Hebert et al.,2020; Spitzer & Musslick 2021). Thus, the pandemic is likely to have had a greater impact on students who are already experiencing marginalization and learning difficulties, further reinforcing existing educational inequalities.

The majority of studies evaluating students’ academic outcomes during the pandemic focused on older K-12 students as well as students in higher education (e.g., César Vargas-Ramos et al., 2022; Spitzer & Musslick, 2021). In contrast, there appears to be a dearth of information regarding younger elementary students’ learning outcomes during the pandemic (K-2). This lack of data is concerning, as early learners build core skills that are critical to later academic success. There is some evidence that Norwegian first-grade students receiving remote instruction had lower scores for writing quality, handwriting fluency, and attitudes towards writing than students receiving instruction before the pandemic (Skar et al., 2021). Such findings raise questions about whether and to what extent the pandemic may have influenced younger students’ outcomes in other cognitive skills. Learning to read, for example, requires students to be able to decode and recognize words, extract their meaning, combine words in syntactically meaningful ways, and make inferences. The development of these critical skills begins at a young age (Catts et al., 2016; Kendeou et al., 2008). Additionally, early reading and language abilities in pre-school and kindergarten are predictive of students’ reading comprehension skills in later academic years (Catts et al., 2016; Hjetland et al., 2019; Kendeou et al., 2009). Considering how high-quality early language comprehension is critical for later academic success (Silverman et al., 2020), there may be some concerns regarding long-term consequences of the pandemic on young students’ academic progress. Providing online instruction for young students also appeared to bring forth additional challenges, as these students have less experiences with online learning and may have less academic motivation compared with older students (Roy et al., 2022). Younger students are also likely to require more assistance from teachers and parents. Indeed, parents of younger children reported more negative experiences and greater challenges with remote instruction than parents of older children (Roy et al., 2022).

Overall, our review indicates that there is a greater need to assess younger students’ learning during the pandemic. The difficulty of providing quality online instruction to younger students also warrants further investigation into how educational technology such as ELCII could be used to support online instruction for these students. At least for older students, it seems that access to quality educational technology, digital competency, and technological support may mitigate to some extent any learning challenges students face during the pandemic. Whether or not these findings can be applied to young students remains unknown. Mixed findings regarding the efficacy of online learning in comparison with face-to-face instruction for K-12 students also indicate that the mere access to and inclusion of educational technologies in the classroom is not enough to restore quality education (Huck & Zhang, 2021). Thus, it is important to assess further the conditions under which educational technology may support remote learning remains unclear (Huck & Zhang, 2021). Educational technologies provide many instructional features that are found in traditional instruction, but lack of community, student engagement, and student participation may reduce their efficacy (Tamim et al., 2011). Given the plethora of educational technologies available to teachers, the efficacy of educational technologies in supporting learning during the pandemic likely also depends on the specific design and affordances of these tools for instruction (Hauge & Norenes, 2015). In this study, we assessed the potential promise of the individualized scaffolding and feedback component of ELCII in supporting kindergarten students’ learning of inference-making during the pandemic.

The present study

We implemented ELCII in kindergarten schools during the 2019–2020 (pre-pandemic) and the 2020–2021 (during pandemic) spring semesters. As a cloud-based intervention, ELCII was well suited to be administered to students during online and hybrid instruction. Assessing students’ inference-making trajectories during both implementation periods can provide much needed information about students’ learning during the pandemic and whether that differed compared with students from the previous year. Additionally, by assessing students’ performance before and after they received scaffolding, we can also assess whether the individualized instruction component of the intervention (providing scaffolding and feedback based on individual responses) supported students’ inference-making performance throughout the intervention. In this study, we specifically addressed the following research questions:

(1) Do pre- and in-pandemic students show different rates of growth in inference- making as indicated by their responses to inferential questions during the ELCII video modules?

(2) Do scaffolding and feedback provided during the intervention mitigate learning challenges due to the school shutdown in inference growth that may have occurred during COVID-19?

If COVID-19 has negatively impacted students’ ability to learn from the ELCII instructional modules, we expect slower rates of growth/no growth as students complete the intervention. However, if scaffolding and feedback mitigates learning challenges due to the school shutdown, we expect differences in learning rates only to be evident when students do not receive scaffolding and feedback, independent of the pandemic context.

Method

Participants

Students. All students (n = 241) were recruited from elementary schools located in a suburban Midwest district in the United States. Two cohorts of kindergarteners were recruited, one in the 2019–2021 school year (pre-pandemic cohort), during which instruction was provided in person; and one in the 2020–2021 school year (in-pandemic cohort), during which instruction was provided in a variety of in-person, hybrid, and online learning modalities. See Table 1 for demographic information of all students in both cohorts.

Table 1 Demographic Information for all ELCII Students by Cohort
Full size table

The data for the pre-pandemic cohort is from a larger quasi-experimental study evaluating the efficacy of ELCII (n = 407) that was interrupted because of the onset of the pandemic and school closures. Given the focus of this study on how the core components of scaffolding and feedback influence student learning, only the treatment students in this cohort were included in this study (n = 221). Of this pre-pandemic treatment sample, 2 students did not provide signed parent consent forms and 8 students were excluded from the study as they had moved prior to the beginning (n = 2) of the intervention or could not participate fully in ELCII due to enrollment in center-based programs (n = 6). An additional 20 students (9%) were removed from the dataset for the following reasons: moved to another school during the intervention (n = 13) or due to teacher request (n = 7). The final pre-pandemic cohort consisted of 191 students (100 females) from three elementary schools.

The data for the in-pandemic year is from a larger quasi-experimental study assessing the feasibility and efficacy of ELCII in remote contexts. The sample consisted of 152 students. Of the 152 participants, 53 students were assigned to complete the intervention. Of this in-pandemic sample, only 50 students participated in the intervention. Two students moved to another school and one student was removed due to teacher’s request. The final in-pandemic cohort consisted of 50 students recruited from one of the three elementary schools that also participated in the pre-pandemic school year.

Schools and Teachers. The participating schools across both cohorts are part of the same school district and receive Title 1 funding to support student learning and increase parent involvement. All schools also use the same reading curriculum (Lucy Calkins Units of Study in Reading) and all students have access to Chrome books in the classroom. Ten teachers implemented the ELCII intervention in the pre-pandemic year and four teachers implemented the ELCII intervention during the pandemic. Two of the four teachers during the pandemic had prior experience implementing ELCII during the pre-pandemic year. Of all four participating teachers during the pandemic, three teachers began with distance instruction. During the intervention, the school district transitioned to hybrid instruction. In-pandemic students who opted to stay in distance learning were placed in another classroom to be instructed by the fourth teacher. Table 2 provides demographic information of teacher participants by cohort.

Table 2 ELCII Teacher Demographic Characteristics by Cohort
Full size table

Measures and procedures

Minnesota Inference Assessment. Both cohorts completed the Minnesota Inference Assessment (MIA, Kendeou et al., 2021). The pre-pandemic cohort completed MIA at the end of October 2019 while the in-pandemic cohort completed MIA at the end of October 2020. Students completed MIA before starting the ELCII video modules. MIA was designed to measure students’ inference-making skills. Students watched a fiction and non-fiction video, each approximately 5 min in length. The fiction video was adapted from a Blinky Bill cartoon episode (Granny’s Glasses; Palmer & Gross, 1993). The nonfiction video was adapted from a documentary about bald eagles designed to be appropriate for students without prior knowledge. As students watched each video, they also answered eight inferential multiple-choice questions (16 questions in total). The questions interrupted the video at pre-determined points. The assessment had adequate reliability (αForm1 = 0.74). IRT-derived theta scores were used in the analysis. Theta scores are standardized z-scores, with a mean of 0 and a SD of 1. In this study, students’ MIA theta scores were included in the final analyses to control for variance due to students’ baseline inference-making skills.

Teacher professional development (PD)

The implementation of teacher Professional Development was adjusted from pre-pandemic to in-pandemic cohorts. In both cases, though, teachers indicated that they learned effectively from the PD and were well-positioned to implement ELCII, which is the ultimate goal of PD independent of how it was implemented. After the PD training, pre-pandemic teachers completed a brief evaluation of the training workshop. In-pandemic teachers were also asked to complete a brief evaluation of ELCII after implementation. In general, pre-pandemic teachers expressed confidence in their ability to implement ELCII in their classrooms. In-pandemic teachers also found ELCII easy to implement and indicated that they would use ELCII again in the future.

Pre-Pandemic. In December 2019, the research team provided a 7.5-hour PD session for the participating teachers. The PD included an overview of the intervention, the study, and a number of activities that allowed teachers to use the ELCII platform. Teachers also practiced implementing the small-group read-alouds on their own and with another teacher. To do so, teachers also viewed a video demonstration of an ELCII read-aloud and learned about the key instructional components of the read-alouds.

In-Pandemic. In December 2020, three personnel from the research team hosted an online PD session for the teachers. Prior to the meeting, the team sent out an orientation manual that described the intervention timeline, students’ login details, and the teacher demographic survey. Additionally, teachers were provided with instructions to set up their own accounts on the ELCII platform prior to the PD session. Training was designed to last about 35 min, plus additional time for teacher questions. During the session, the team provided an overview of ELCII, the new ELCII platform, and the intervention website. The team also provided a walkthrough of the ELCII platform to familiarize teachers with the interface. Finally, the team provided a quick overview of the data that teachers can access on the platform.

ELCII Intervention.

Pre-Pandemic. Teachers implemented ELCII from the beginning of January 2020 to March 2020. The ELCII video modules were designed to be completed over a 10-week period. Students were expected to complete two video modules per week (total of 20 modules); however, students could also complete the videos at their own pace. The genre of the videos in the intervention modules changed every week. Specifically, on weeks 1, 3, 5, 7, and 9, students watched fiction videos. On weeks 2, 4, 6, 8, and 10, students watched nonfiction videos. In addition to the video modules, pre-pandemic students also participated in two small group read-alouds every week. As with the genre of the video modules, the genre of the books in the read-alouds switched every week. Although students were supposed to complete all 10 weeks of the intervention, they were only able to complete 9 weeks due to the COVID-19 lockdown. Thus, students could only complete 18 video modules and 18 read-aloud lessons.

Each intervention module and read-aloud was designed to last 15 to 20 min. Teachers participating during the pre-pandemic school year implemented the intervention in the classroom typically within the first hour of school. Teachers were given the option to implement the video modules and read-alouds anytime, as long as students finished two modules and two read-alouds per week. As the number of Chromebooks available for student use was limited, most teachers had two groups of students completing video modules on different days of the week. The first group would complete the video modules on Monday and Wednesdays, and the second group would complete the video modules on Tuesdays and Thursdays.

In-Pandemic. The ELCII intervention during the pandemic was administered through a new platform that was designed to allow students to complete the intervention modules without teacher assistance. Students were able to log-in and continue their progress on the intervention modules by themselves. From January 2021 to March 2021, students completed the ELCII intervention. Students started ELCII remotely as the district had mandated full distance-learning during that time. However, beginning on January 21st, the district transitioned into hybrid learning. Most students gravitated between in-class and remote instruction. A handful of students (n = 7) remained in full distance instruction the entire school year. Students in hybrid instruction completed ELCII at home and/or at school. Whether students completed ELCII at home and/or at school was based on teacher discretion. As in the pre-pandemic school year, in-pandemic students were expected to complete two video modules per week. However, to accommodate teachers during the pandemic, students were given more flexibility in terms of pacing as long as they completed all modules within a reasonable amount of time. Students could complete 20 video modules. In both years, the platform was also designed so that students had to complete modules according to the intervention schedule.

Differences in ELCII Implementation. Before evaluating students’ learning trajectories, it is important to explicitly acknowledge that the pandemic by default influenced the implementation of ELCII pre- and in-pandemic in significant ways. First, pre-pandemic students completed small group read-aloud lessons in addition to the instruction video modules while in-pandemic students only completed the instruction video modules. Although the inclusion of read-alouds in the pre-pandemic year raises issues about whether differences in students’ trajectories can be attributed to differences in instruction and context, the exclusion of small group read-alouds during the pandemic arguably provides a more realistic comparison between the two cohorts as remote instruction did not allow such interactions in a school context (see discussion for further interpretation regarding the impact of the read-aloud lessons). Second, three of the instruction video modules pre-pandemic were replaced with new ones during the pandemic after receiving teacher feedback. Specifically, the three video modules that were replaced were two nonfiction modules about Jane Goodall and arachnids and one fiction module about spiders adapted from an episode in the show Shaun the Sheep. These video modules were replaced with two nonfiction modules about elephants and oceans and one fiction module about puzzles adapted from a different episode in Shaun the Sheep. While the content of these modules differed, the research team ensured that the new modules contained videos in the same genre and/or were adapted from the same show. See Table 3 for an overview of the instructional modules students completed pre- and in-pandemic.

Table 3 Title and Genre of ELCII Video Instructional Modules by Week
Full size table

Fidelity

Video Modules. Given that ELCII modules were administered through a web-based platform, all students received the same instructions and completed the same video modules. All pre-pandemic students completed the assigned intervention modules prior to the lockdown, whereas 80% of in-pandemic students completed the assigned modules. All students were only allowed to complete the required modules for a given week and were unable to move ahead and complete additional modules. This ensured that students were completing the modules at a similar pace. Pre-pandemic teachers were also given a form to assess students’ overall engagement with the ELCII video modules, which is an important aspect of fidelity (Sanetti & Kratochwill, 2009). Teachers indicated whether students showed high, medium, or low engagement, and their responses were converted into a 3-point scale (3 = high engagement, 1 = low engagement). Teachers completed the form on average four times, for four different video intervention modules (range: 1–6). Overall, pre-pandemic students seemed to express medium to high engagement with the module content (M = 2.59, SD = 0.47). Teachers also commented on the forms that most students were engaged and interested in learning from the videos. Any information about in-pandemic students’ engagement with the module content came from in-pandemic teachers’ evaluations of ELCII after module implementation. According to teachers’ evaluations, in-pandemic students seemed to find ELCII engaging. Teachers also indicated that students were able to navigate the ELCII platform independently. All teachers also commented that students were interested in learning from the video modules. As an additional measurement of fidelity, we attempted to evaluate the pace in which students completed the video modules using computer log data. If students completed the modules in the weeks they were expected to, this would provide some indication of whether students were completing the video modules as intended. Students’ pacing was calculated as their average absolute deviation from the recommended dosage every week. The following formula was used to calculate students’ pacing: mean(abs(number of modules completed each week – 2)). Two modules were subtracted from the number of modules a student completed each week because students were expected to complete two modules per week. On average, pre-pandemic students’ pacing was 0.30 (SD = 0.36) and in-pandemic students’ pacing was 1.12 (SD = 0.55). In general, some deviation from the intended schedule (i.e., deviation = 0) is expected given the realities of implementing an intervention in a classroom setting. These numbers indicate that in-pandemic students showed greater deviation from the intended module implementation schedule than pre-pandemic students. Greater deviation (corresponding to less fidelity of module implementation) was to be expected during the pandemic given the pandemic environment and students’ remote completion of the video modules.

Read-alouds. Pre-pandemic students completed 18 small-group read-alouds in addition to the ELCII video modules. To assess fidelity of read-aloud implementation, the research team developed a form that identified key components of the read-aloud lessons: lesson set-up, classroom environment, vocabulary instruction, modeling of first inference question, scaffolding and feedback of inference questions 2–5, lesson closing, and dosage. Based on these key components, the team identified and operationalized specific actions within each of the components they expected teachers to perform. These actions were scored on a 3-point scale (0, 1, or 2). Four members of the research team were trained to use the fidelity form by scoring implementation of a sample read-aloud lesson previously shown during the teacher PD training. Average inter-rater reliability at the item level was 95%, ranging from 92 to 97%.

Throughout implementation of ELCII, all teachers (except one) were observed at least twice. The first observations took place between weeks 3–5 of ELCII implementation. Teachers’ average fidelity score in the first observations was 94.2% (SD = 6.3). The second observations took place between weeks 6–9 of the ELCII implementation. Teachers’ average fidelity score of the second observations was 97.8% (SD = 2.1%). These scores indicate that pre-pandemic teachers were implementing the core components of the read-alouds with high fidelity.

Analytic plan

We applied linear mixed-effects modeling using the lme4 package in R (Bates et al., 2015) to evaluate students’ trajectories on the ELCII video modules across time while also accounting for the nested structure of the data. Because time was nested within students, we sought to use a two-level mixed-effects model with level 1 representing the measurement time and level 2 representing students. The baseline linear mixed effects model included a time variable (measurement times), a year variable (pre- vs. in-pandemic), and an interaction between time and year:

Scoreij = (β0 + b0i) + (β1) * timeij + β2(year) + β3(timeij * year) + eij.

In this model, the subscripts i and j denote student and measurement time, respectively. Students’ scores over time are the outcome variable, β0 is the fixed intercept, β1, β2, β3 are the fixed linear slopes, b0i is the random intercept for student, and εij is the residual error. We could not include a random slope of time for student as the model would not converge. In our analysis, students’ score on the video modules were evaluated every two weeks of the intervention (total of 4 measures over time). A time of 0 represented students’ scores after the first two weeks of the intervention. At every measurement time, students completed two fiction and two nonfiction video modules. Students’ scores were aggregated across the modules for a total score out of 20. We did not include students’ performance on the fifth measurement time (weeks 9 and 10) pre- and in-pandemic as we halted data collection for the pre-pandemic students during week 9 of the intervention. The decision to aggregate students’ performance every two weeks into a measurement time was informed by the fact that students completed fiction or nonfiction modules every other week. The fiction and nonfiction modules correspond to informational and narrative texts, and these genres tend to differ in difficulty (Botsas, 2017; Best et al., 2008; Duke, 2000). Evaluating students’ performance in a measurement time across genre would accommodate the likely differences in inference-making performance due to genre.

The baseline model addressed whether students’ trajectories across time differ depending on the year (pre- vs. in-pandemic) (RQ1). To this model we added additional covariates to control for differences between the two different cohorts of students. These covariates included students’ age, pretest MIA scores, gender, English learner status (yes/no), special education status (yes/no), race/ethnicity (e.g., American Indian, Asian, Black, Latinx, Multiracial, White), and free or reduced lunch eligibility (eligible, not eligible). Students’ demographic data were provided by the schools and teachers participating in this study. Students’ school was also added as a covariate to account for the fact that pre-pandemic students were recruited from two additional schools. For categorical covariates, pre-pandemic students, female students, non-English language learners, non-special education students, and students not eligible for free or reduced lunch were dummy coded as 0. Students’ school and ethnicity were dummy coded as 1. All quantitative covariates were mean-centered prior to analysis. Welch two sample t-tests and chi-square tests indicated that students in the two cohorts differed significantly across all covariates (p < .05).

To assess whether individualized scaffolding and feedback supports students’ learning (RQ2), we ran an additional model that added a scaffolding covariate to the baseline model. As a result, the new model also included a scaffolding term, two-way interactions between scaffolding and time and scaffolding and year, and a three-way interaction between scaffolding, time, and year.

Results

Descriptive statistics for students’ performance on the ELCII video modules before and after scaffolding by year at each time point can be seen in Table 4. In both cohorts, students’ mean score increased over time and after they received scaffolding and feedback. The table also shows the proportion of missing data for students at each time point. A nontrivial proportion of data was missing for in-pandemic students compared with pre-pandemic students. The missing data were presumed missing at random (MAR). Finally, parameters estimates as well as model fit indices for the mixed-effects models (with and without the scaffolding covariates) can be seen in Table 5.

Table 4 Scores on the video intervention modules
Full size table
Table 5 Parameter Estimates for Combined Models with Covariates
Full size table

RQ1: trajectories of students’ inference-making pre-and in-pandemic

Our first research question examined whether students’ inference-making trajectory on the ELCII video modules differed pre- and in-pandemic. Specifically, we hypothesized that students completing ELCII during the pandemic would exhibit slower growth in inference-making compared with students completing ELCII before the pandemic. The results of the mixed-effects model showed a significant main effect of time and year, qualified by significant time x year interaction (see Fig. 1). Consistent with our hypothesis, in-pandemic students showed attenuated inference-making trajectories during the ELCII intervention compared with pre-pandemic students [t(445) = 3.11, p < .01], even after accounting for their age, school, pretest MIA scores, gender, English learner status, special education status, ethnicity, and free or reduced lunch eligibility. Regarding the covariates, MIA, English language learner status, and free or reduced lunch eligibility were statistically significant. Students with lower MIA scores (1 SD decrease), English learners, and students eligible for free or reduced lunch performed, on average, lower on the ELCII video modules after accounting for other covariates.

Fig. 1
figure 1

Pre- and In-Pandemic’s Inference-Making Trajectories over Time

Full size image

RQ2: the role of individualizing student instruction on inference-making performance

Our second research question examined whether the scaffolding and feedback component of the ELCII intervention would support students’ learning during the pandemic by individualizing student instruction. We hypothesized that individualizing instruction in ELCII through the scaffolding and feedback component would help mitigate any attenuation in learning that may occur during the pandemic. The AIC for the mixed-effects model with scaffolding as a covariate was lower than the AIC for the baseline model, suggesting the improvement of the former over the latter model. Additionally, the results of the new model revealed a significant three-way interaction between scaffolding, time, and year (t = 2.73, p < .01). We obtained estimated marginal means using the ggPredict function in R (Lüdecke, 2018) in order to visualize the three-way interaction (see Fig. 2). Additionally, we ran a simple slopes analysis using the sim_slopes function in R (Long, 2021). The simple slopes analysis showed that pre- and in-pandemic students’ inference-making trajectories differed significantly before they received scaffolding. Specifically, in-pandemic students did not show significant improvement in inference-making over time (slopein−pand = 0.32, p > .05), and pre-pandemic students showed significant improvement over time (slopepre−pand = 0.92, p < .001). As indicated by the plot, pre- and in-pandemic students’ inference-making trajectories after they received scaffolding did not differ (slopein−pand = 0.70, p < .001; slopepre−pand = 0.69, p < .001). Together, the interaction plot and simple slopes analyses suggests that the scaffolding and feedback component significantly influenced students’ inference-making trajectory by cohort, even after accounting for all other covariates. In-pandemic students exhibited little-to-no improvement in inference-making when compared with pre-pandemic before receiving scaffolding and feedback. However, both cohorts showed positive and similar learning trajectories when students received scaffolding and feedback. These results suggest that the scaffolding and feedback component supported students’ performance on the ELCII modules during the pandemic to the extent that their inference-making trajectory was similar to that of pre-pandemic students.

Fig. 2
figure 2

Pre- and In-Pandemic’s Inference-Making Trajectories over Time by Scaffolding

Full size image

Discussion

The purpose of this study was two-fold. First, we evaluated kindergarten students’ inference-making trajectory on ELCII before and during the pandemic to assess how students’ learning environment and the overall pandemic environment may have influenced their learning progress. Second, we assessed whether providing individualized scaffolding and feedback to students through ELCII helped support students’ inference-making performance during the pandemic. The results of this study provide some initial insights into younger students’ actual learning trajectory during the pandemic. Additionally, the findings shed some light on the role of individualized scaffolding and feedback on student learning in remote environments.

Assessing students’ inference-making trajectories pre- and in-pandemic showed slower rates of inference-making grown for in-pandemic students compared with pre-pandemic students, even accounting for a variety of demographic characteristics and students’ baseline inference-making skill. This finding extends previous research that documented students’ learning during challenging times, including natural disasters (e.g., NAEP, 2022; Ward et al., 2008). Consistent with previous findings (e.g., Bao et al., 2020; Kuhfeld et al., 2020; Skar et al., 2021), the pandemic appears to be negatively associated with young students’ learning of inference-making. This finding is alarming, although it is not surprising given unprecedented hardships during the pandemic, because inference-making is a core general skill that can influence not only reading but also learning of different subject areas (Kendeou et al., 2020). The finding suggests that moving forward district or school leaders need to support teachers to provide more inference instruction to young students.

We also found that the individualized scaffolding and feedback feature of ELCII supported in-pandemic students’ inference-making performance. Without scaffolding and feedback, in-pandemic students did not show any growth in inference-making. However, with scaffolding and feedback, students showed significant growth in inference-making over time. In fact, with scaffolding, pre- and in-pandemic students’ inference-making trajectories did not differ. These results corroborate prior studies highlighting the importance of scaffolding and feedback in providing inference-making instruction to young students (Hansen, 1981; McGee & Johnson, 2003; Palincsar & Brown, 1984; Yuill & Oakhill 1988). This study adds to the current literature by showing that individualized scaffolding and feedback via a digital platform in remote contexts can support kindergarten students’ learning of inference-making. That is, individualized scaffolding and feedback that caters more to students’ learning needs may be critical to improving students’ inference-making skill. In order to individualize scaffolding and feedback in remote contexts, the teaching agent must be aware of students’ knowledge needs, elaborate on the scaffolding and feedback based on those needs, and guide students to make inferences without simply giving them the answer. In the future, it may be interesting to assess how greater individualization of scaffolding and feedback with more advanced educational technology may benefit students’ learning of inference-making and learning across different domains in remote contexts.

In interpreting our results, it is important to acknowledge several limitations of our study. First, there were differences in implementation of ELCII pre- and in-pandemic. Teachers during the pandemic received less intensive PD training. Providing teachers with intensive PD training would have placed additional burden on teachers on top of all the other challenges they were facing during the pandemic. Data on students’ pacing of module completion suggested that fidelity of module implementation (defined as completion of modules in the expected time) was lower during the pandemic, due, in part, to logistical issues and high stress introduced by the pandemic – students needed more flexibility and time. Still, in-pandemic teachers indicated that students were engaged during completion of video modules and could independently navigate the ELCII platform. Pre-pandemic students also completed the instructional modules and small-group read-alouds while in-pandemic students only completed the instructional modules. This was because teachers during the pandemic were unable to provide small read-aloud lesson. Therefore, the results of this study should be interpreted with caution, as these analyses do not preclude the possibility that differences in the implementation influenced in-pandemic students’ learning trajectories. Still, the adjusted implementation during the pandemic likely also provides a more realistic comparison between the pre-pandemic and in-pandemic cohorts. In addition, we would like to note that regardless of these differences, pre- and in-pandemic students exhibited similar inference-making trajectories after they received scaffolding and feedback. Therefore, despite lower fidelity and the lack of read-aloud lessons for in-pandemic students, the scaffolding and feedback still improved in-pandemic students’ inference-making performance to levels that were comparable with that of pre-pandemic students.

Second, our analyses involved comparison of two cohorts of students that displayed differences in demographic information. Although we attempted to account for differences in students’ baseline inference-making skills (i.e., MIA pretest theta scores) and student demographics (i.e., age, school, gender, English learner status, special education status, ethnicity, and free or reduced lunch eligibility), there may have been other important differences between cohorts that account for differences in students’ inference-making trajectories. The sample size of the in-pandemic students was also small compared to that of the pre-pandemic students due to difficulty in recruiting schools and teachers during the pandemic. There was also greater attrition during the in-pandemic school year. In general, it is important to acknowledge that low power due to low sample size reduces the likelihood that statistically significant results in our study reflects a true effect. Thus, our findings should be evaluated and interpreted in conjunction with other studies evaluating students’ learning during the pandemic as well as the role of technology in remote instruction.

Third, our findings would have been enriched by having more information regarding in-pandemic students’ activities during remote instruction and at home. Most in-pandemic students started ELCII remotely for almost a month before they transitioned into hybrid learning. Students completed ELCII either at school or at home based on teacher’s preference. In-pandemic teachers estimated that students’, on average, completed ELCII about half of the time at home (range: 30–75% at home). While in-pandemic students were not fully remote throughout the intervention, their environment and learning contexts were vastly different as a function of the pandemic in comparison with pre-pandemic students. We also do not have any data regarding students’ activities at home during the pandemic. Parents may have had students only complete school activities or complete other supplemental activities to support their education. One study modeling students’ reading trajectories during the pandemic suggested that having parents read to their children every day would mitigate any potential loss in reading ability gain (Bao et al., 2020). Home activities and support may have supported some of students’ learning during the pandemic.

Despite these limitations, our study provides some much-needed initial insight into young students’ actual learning trajectories during the pandemic. As schools transition to remote instruction and/or hybrid instruction, it is important to consider further how educational technologies could provide quality instruction to students. Delivery of effective remote instruction depends not just on the inclusion of educational technologies, but the specific affordances of that technology. The results of this study suggest that individualized scaffolding and feedback is one potentially important affordance of educational technologies. However, there may be other affordances of technology that effectively support student learning. Future studies should investigate usage and affordances of educational technology during the pandemic. The results may provide some clarify as to why some students’ academic outcomes may have increased rather than decreased during the pandemic (Spitzer & Musslick, 2021).

In considering the role of educational technology during the pandemic, it is also necessary to consider teachers’ ability to provide quality instruction through educational technologies. In this study, students could complete the ELCII video modules without assistance from teachers. However, most educational technologies require teacher cooperation and knowledge to support student learning. Teachers not only need access to evidence-based educational technologies, but further training and support in integrating educational technologies to meet students’ specific needs. Ultimately, supporting student learning during the pandemic will require quality instruction from teachers in conjunction with supplementary lessons provided through educational technology.