Introduction

Covid-19 created unprecedented disruptions to our nation’s education systems, including the near-universal closure of school buildings for the last three months of the 2019–20 school year. At that time, scholars projected dramatic learning loss and a sharp increase in educational inequality (Kuhfeld & Tarasawa, 2020). Yet the disruption also generated a range of efforts to use remote instruction to mitigate these effects. The best available evidence suggests instruction that includes a face-to-face component is preferable to purely virtual learning for student engagement (Gallagher & Cottingham, 2020) and achievement (Bueno, 2020; Escueta et al., 2020). During the pandemic, Kofoed et al. (2021) found that West Point students randomly assigned to an online class struggled to concentrate and connect with their instructor and peers and received lower final course grades than in-person students. More generally, higher education students transitioning to remote instruction due to Covid report greater amounts of busy work and reduced perceptions of academic success (Motz et al., 2021). However, limited research has emerged to date on efforts to innovate in the virtual learning space amid the pandemic for K-12 students. This is a significant omission as the expansion of online schooling during Covid-19 presents an unprecedented opportunity for the field to learn about best practices for K-12 students in virtual environments, particularly among a wider population of students that previously have had limited exposure to online learning.

Much of the pre-pandemic online learning literature comes from higher education (Martin, Sun & Westine, 2020; Barbour, 2018) and generally finds null (e.g., Hoffman & Elmi, 2020; Wagner et al., 2011) or negative effects on student academic outcomes (e.g., Bettinger et al., 2017). Scholars examining student engagement with online learning in higher education have found that students value interactive environments where they can engage with peers and instructors and that these characteristics are associated with higher levels of student satisfaction (Caskurlu, Richardson, Maeda & Kozan, 2021; Martin & Bolliger, 2018). However, it is unclear the extent to which these findings generalize to K-12 settings. Again, this is unfortunate as the pandemic has generated greater openness to online learning, and a research-informed expansion could increase the success of such efforts.

Additionally, much of the pre-pandemic research on K-12 virtual learning has focused on specialized circumstances such as advanced placement courses, credit recovery, or remote rural regions (Toppin & Toppin, 2016; Yang, Yu & Chen, 2019). Reviews of the literature on online charter schools do not provide much optimism about the benefits of virtual K-12 charter schooling (Waters et al., 2014), with specific studies finding lower levels of academic performance for online versus in-person students (Fitzpatrick et al., 2020; Woodworth et al., 2015), even when comparing students within the charter sector (Ahn, 2016). The negative effects of these programs have been greater among economically or academically disadvantaged students (Kwon et al., 2019; Liu & Cavanaugh, 2011). Less is known about how students and teachers who have not specifically chosen to participate in online schooling over an in-person option might fare in virtual environments.

Despite overall low levels of engagement in virtual learning, researchers have found some characteristics of online programs that are associated with higher levels of student success, such as greater student engagement (Kwon et al., 2019), time students spend on online courses, amounts of teacher feedback (Liu & Cavanaugh, 2011), high-quality project-based curricula (Zheng, Lin & Kwon, 2020), and individualization (Cavanaugh, 2001). Generally, students appear more engaged with online learning when the curriculum includes work they consider to be meaningful (Darling-Aduana, 2021), when they are able to engage synchronously with teachers and peers (Barbour, 2012; Turley & Graham, 2019), and when parents are available to monitor and motivate (Curtis & Werth, 2015). It is important going forward to examine whether these findings on online learning methods replicate among the more generalizable population of K-12 students experiencing online learning during the pandemic and whether new models may have cropped up from which the field can learn.

One primary potential advantage of online learning over in-person formats is the possibility of expanding access to educational opportunities given the lack of geographic constraints. In short, a student could theoretically use an online course to access material not offered at their local school. In higher education, Goodman et al., (2019) find Georgia Tech’s online master’s degree program in computer science substantially increased enrollment, opening opportunities for students who otherwise would not have pursued a degree in person. However, this concept has been understudied in the K-12 arena and likely has important implications for equity given the well-documented inequitable distribution of highly effective teachers (Boyd et al., 2005; Lankford et al., 2002) due to teachers’ tendancy to sort into residential communities where fewer low-income students of color live and learn (Boyd et al., 2011). In short, online learning could be a lever for expanding access to exceptional teachers for the students least likely to currently have facetime with these educators. Teacher quality is the strongest in-school predictor of student academic achievement (Hanushek, Kain & Rivkin, 2005) and therefore represents a critical resource in any effort to reduce educational inequality. Such strategies for extending the reach of our nation’s best teachers have been proposed by education policy analysts and advocates (e.g., Dwinal, 2015; Hassel & Hassel, 2009) but have been the subject of only limited rigorous research to date.

There are also big open questions related to teacher perceptions of online learning, which is essential to effectiveness given the longstanding body of research suggesting teacher buy-in is critical to the successful implementation of reform (e.g., Weatherley & Lipsky, 1997). In nationally representative post-Covid surveys, teachers and principals expressed significant need for support in the implementation of online learning, particularly around ways to motivate and engage students virtually (Hamilton, Kaufman & Diliberti, 2020). Additionally, the pandemic has delivered a blow to teacher morale, with over 80 percent reporting feelings of burnout (Diliberti & Kaufman, 2020). However, it is unclear whether the virtual nature of instruction has contributed to that burnout versus other aspects of pandemic life. Therefore, it is important to know whether and how it is possible to preserve (or improve) teacher morale in an online environment. Understanding teacher perceptions of efforts to expand their reach using technology is critical to determining the viability of the concept.

Therefore, given the limited existing work on K-12 virtual learning in the post-pandemic context, this study examines an early effort to develop new approaches to serving students, promoting educational equity, and expanding the reach of talented teachers via virtual learning. Specifically, we study the National Summer School Initiative (NSSI), a virtual program run over five weeks (25 days) in summer 2020 that served 11,769 3rd to 8th grade students across the nation with the goal of making up for lost learning time due to Covid-19 (NPR, 2020). The roughly 50 partner schools or networks serve student populations that, on average, are 90% Black or Latinx and in which 79% qualify for subsidized meals (see Table 1). Roughly 43% of schools were operated by charter management organizations. The program was designed by current and former leaders of high-performing charter schools and a large school district home to a high-performing charter sector, in partnership with the nonprofit consultancy Bellwether Education Partners.

Table 1 Describing NSSI Schools Relative to All U.S. Schools
Full size table

The program was unique from other K-12 online learning programs not only in its target student population but also in the way it sought to capitalize on the virtual nature of instruction to expand access to excellent teachers. Leaders recruited a group of “mentor teachers” they considered to be among the nation’s most talented educators, recommended from among their networks, to videotape themselves teaching each lesson to their own “fishbowl class” of students via Zoom. Mentor teachers were assigned “partner teachers” selected by the partner school or network, with each mentor working with all of the roughly 50 English language arts or math teachers at each grade level. Partner teachers received access to lesson plans and video of the mentor teacher’s class session before teaching the same lesson to students from their home school. They also received professional development (PD) from their mentor teacher. The program further aimed to prepare teachers for a possible virtual 2020–2021 school year and ease the burden on schools of creating robust summer programming while they focused on the operational challenges of fall reopening plans. At the program’s conclusion, leaders revised the NSSI model into an initiative called Cadence Learning that allowed schools, districts, networks, and learning pods to gain access during the school year.

We examine stakeholder perceptions of the program to inform the design of future online schooling interventions, amid disruptions to in-person instruction as well during more typical times. We focus on participant perceptions of the program, rather than impacts on student achievement. We view this as an important contribution given student and teacher satisfaction and engagement has been a key challenge during periods of online learning (Barbour, 2012; Keaton & Gilbert, 2020), leading teachers to express the need for professional development related to student engagement in online settings (e.g., Hamilton, Kaufman & Diliberti, 2020; Zweig & Stafford, 2016). Additionally, previous research draws a link between engagement in virtual learning environments and performance (Al-Azawei & Al-Masoudy, 2020; Boulton et al., 2013), as well as a connection between enjoyment of learning when using educational technology and positive learning behaviors (Hashim & Vongkulluksn, 2018). Therefore, it is essential for the field to capitalize on the widespread use of K-12 virtual learning during the pandemic to gain a better understanding of promising practices that could encourage satisfaction, enjoyment, and engagement with online learning.

Preliminary data on actual learning in the aftermath of pandemic-induced school closures shows students in the U.S. achieving at lower levels in Fall 2020 than in prior years, especially in math and among students from schools with greater concentrations of disadvantage, though a lack of testing data for many students leaves uncertainty about the full scope of the decline (Kuhfeld et al., 2020b). Given the likely growth in educational inequity, it is essential for the field to learn from early efforts to implement high-quality virtual instruction among those student populations most hard hit by the pandemic to inform future efforts to mitigate learning loss and minimize the growth of inequality as a result of lost learning time.

In this study we seek to understand whether it is possible for virtual learning to successfully engage K-12 students, to improve teacher morale and confidence in their teaching abilities, and to more equitably expand the reach of excellent teachers. We draw on survey data from teachers, students, parents, and administrative coaches, as well as interviews with partner and mentor teachers, to address the following research question in the context of a rare case of a post-pandemic effort to innovate in the virtual learning space: What did teachers, students, and parents perceive were the strengths and weaknesses of the NSSI program?

THE NATIONAL SUMMER SCHOOL INITIATIVE

The student day at NSSI ran about three hours and forty-five minutes and the teacher day extended for an additional 90 min of preparation and PD. There were three core academic classes: novel studies, close reading, and math stories, all of which were taught synchronously, one feature of the program that made it unique relative to many other K-12 online learning programs. Novel studies focused on reading and discussing what leaders described as an “exceptional novel” and developing pleasure in analyzing texts. In close reading, students discussed and wrote about shorter selections of poetry, short fiction, and nonfiction. The text selections and implementation of the curriculum was tailored to each grade level. In math stories, students solved a problem of the day designed to be “real world” applicable not by using particular algorithms but by applying their existing knowledge of math to the context. Teachers then led students through a “discourse” in which students discussed how they solved the problem, with the goal of developing conceptual understanding. The theory of action was that lower-performing students would be able to contribute ideas about how to solve the problem while also seeing a peer student’s more sophisticated solution. More advanced students would in turn be challenged to think of more than one method for solving. Novel studies and math stories were held daily, while close reading was held three days a week. The other two days, students had an enrichment class that included self-directed educational activities and synchronous virtual science labs. A daily asynchronous movement and mindfulness class included yoga, fitness, and dance. Leaders recommended 20–30 students per section with a maximum of 40. The average class size was 27, as we report in Appendix Table 6 and 7, and students were typically grouped with other students in their grade from the school they attended during the regular school year.

The program was operated not for profit and funded by philanthropists. Leaders recruited networks and individual schools to participate in the program and then the schools identified partner teachers willing to teach in the summer program. Schools were also responsible for identifying the families that wanted their children to participate. Partner schools received the curriculum, training, and mentor teacher services free of charge. They had to pay partner teachers, for any technology students needed to participate in the program, and for a local administrative coach to serve as their point of contact. Prior to the program, teachers attended a week-long virtual training by the Lavinia Group, an organization that has previously served several leading charter networks. During the program, teachers had a daily PD session focused on intellectual preparation for the next day’s lesson and analysis of student work.

Methods

We interviewed partner and mentor teachers and analyzed internal survey data as these methods have the advantage of providing information from stakeholders directly about their perceptions of NSSI. Incorporating multiple types of data (interviews and surveys) from a variety of sources (administrators, mentor teachers, partner teachers, students, and parents) helped us triangulate results to reduce systematic biases that could arise when relying on a single type of data or informant (Maxwell, 2005). The survey data were collected prior to the interview data but the two sources of data were analyzed simultaneously which allowed us to toggle back and forth between them as tentative findings emerged.

For interviews, we identified a stratified sample (Seidman, 2006) of 60 partner teachers from the total group of 513 and 12 mentor teachers from the full group of 15 based on the teacher’s subject and grade and whether the teacher worked at a charter management organization (CMO) school. Our interview sample ultimately included 28 teachers (22 partner teachers and 6 mentor teachers) who responded to our request. One recruitment challenge was the timing of our interview phase which occurred at the end of summer 2020, in the midst of the pandemic, and during the start of what was sure to be a challenging school year. Luckily, Table 2 indicates that the interview samples were representative of the full populations of teachers on all dimensions for which we have data.

Table 2 Describing the NSSI Teacher Study Sample
Full size table

A team of eight researchers conducted interviews via Zoom using a semi-structured protocol (Merriam, 1998) that began with open-ended questions to avoid steering the conversation based on our hypotheses and then included more specific probes regarding that could be posed in the event that the informant did not raise them in response to the open-ended prompts (Seidman, 2006). For example, we begin by asking respondents to tell us about their experience with NSSI, then how NSSI compared to regular school, then strengths and weaknesses of the program. More specific probes tackled topics related to student engagement and teacher morale. Interviews lasted between 45 and 60 min. Teachers received a $50 gift card for their time. Interviews were video and audio-recorded and transcribed for coding. We provide the protocols in the Appendix. The video recordings allowed the lead author to watch the first interview conducted by each team member, provide feedback for future interviews, and spot-check interviews as they were occurring to ensure quality and consistency.

We relied on content analysis to identify themes emerging from the data. Coding and analysis occurred immediately after the data collection phase. We began analysis with the lead author coding a random selection of five transcripts—two of interviews with mentor teachers and three with partner teachers—and generating codes directly from the themes emerging from those interviews. All coding was conducted using Dedoose software. These initial five interviews were then double coded by a second member of the coding team (made up of seven of the eight interviewers) who had been trained on the intention behind each code. The lead author met with each individual coder to discuss and reconcile any discrepencies between the codes applied. This also had the benefit of checking the lead author’s interpretations against the full group of coders. Given the discrepencies were relatively minor, the coders were provided with feedback (the most common piece of feedback was to highlight larger selections of text from the transcripts to provide context helpful for the analysis phase), the lead author’s coding tweaked, and then all continued coding additional interviews.

Ultimately, all interviews were double coded and the lead author was one of the coders on 40 percent of the interviews and spot checked the coding on the remaining transcripts to help ensure consistency across interviews. For each interview, the two coders met one-on-one to discuss and reconcile any differences in their coding. The full research team met weekly to discuss any questions or difficult-to-resolve questions that came up in the one-on-one meetings, to clarify codes, and to discuss the addition of new codes that emerged from the interviews. This provided another check on the lead author’s interpretations and the ability to adjust as needed. Our team kept a codebook providing notes on any clarifications or key decision rules by code.

We ended up with 130 primarily inductive codes—listed in full in the Appendix–which fell into 16 broader categories: curriculum, diversity, feedback, fishbowl students, leaders, mentor teachers, morale, operations, overall/general, schedule, students, teacher collaboration, teacher development, teachers, lessons about virtual learning, and workload. We re-coded transcripts that had been coded early in our process to ensure that new codes that were added during the coding phase could be applied to all transcripts (Maxwell, 2005). Luckily, the addition of new codes was relatively rare after the first week of coding which gave us confidence not only in our coding scheme but also that the recruitment of additional interviewees would not likely lead to the identification of novel themes (Merriam, 1998). Typically, for each code created, we also added the inverse to our list of possible codes (even if it was not relevant for that particular interview). For example, after an interview prompted the creation of the “Curriculum – engaging for students” code, the lead author also created a “Curriculum –not engaging” code. This meant that the codes had a “directionality” which aided interpretation. In other words, we knew not just the frequency with which respondents discussed curriculum but the frequency with which they discussed it in positive versus negative terms.

To analyze the coded data, we examined the frequency with which each code was applied to generate a list of tentative findings that eventually became the nine major findings that we discuss below. For each of the most commonly applied and related codes, we read through each quotation to which that code had been applied and selected quotations that evidenced that finding (ultimately selecting representative quotes from this larger set of quotes to include in the paper). We did this both across and within the mentor and partner teacher groups. We then scanned the list of codes and examined quotes from any codes that seemed to potentially contradict the tentative finding. This allowed us to add nuance to our findings and to actively search out disconfirming evidence that was inconsistent with our emerging findings or that supported alternative explainations (Maxwell, 2005). We also asked our team of interviewers and coders for their reflections on major takeaways that came out of their interviews and asked for their feedback on our emerging hypotheses and findings to ensure that what we reported was not broadly inconsistent with what they had heard from interviewees.

We also conducted member checks to ensure that the inferences we were making were supported by the data and that we were interpreting all quotes in the way that respondents had intended. We did this by providing all respondents a draft of the paper with quotations and inviting feedback. However, ultimately, we made no changes based on this step as the only feedback we received was generally positive in nature.

In addition to the interview data, we also analyzed data from surveys conducted by the NSSI team in weeks one, three, and five of the program. Our focus is primarily on the week five, end-of-program results to assess perceptions based on the full program. Surveys were completed by administrative coaches who served as the primary school- or network-based contact at each partner school (n = 42), mentor teachers (n = 7), partner teachers (n = 188), students (n = 2,484), and parents (n = 892). Table 2 shows that there are no statistically significant differences between the mentor survey sample and the full population of mentor teachers on observable characteristics. The partner teacher survey sample is generally representative but more likely to be missing data on the grade level taught (11% for the survey sample vs. 5% for the full population) and less likely to teach at a CMO (44% vs. 56%). We do not have data on the full population of students or parents to assess representativeness. All survey findings should be interpreted with caution as they may not reflect student, parent, and teacher opinions more broadly. The student and parent surveys ask about satisfaction with the program, student learning, perceptions of virtual learning, and student in terest and engagement with school. The administrator and teacher surveys asked similar questions as well as questions about teacher learning, confidence, and the quality of program materials and the mentorship experience.

Limitations

Our study has a number of limitations. First, we are constrained to examine self-reported participant perceptions and cannot speak definitively to changes in participating students’ academic performance. We also urge caution in drawing conclusions from our survey data since respondents may not be representative of all participants. Our study cannot speak to the effectiveness of virtual learning as a whole, nor does it provide a comparison between the efficacy of virtual versus in-person instruction. Future research should test whether our findings generalize to an era in which most schools are reopened for in-person learning.

Importantly, our findings may not generalize to students who differ in important ways from NSSI participants. Specifically, these students were drawn from schools serving large concentrations of low-income children and students of color. Additionally, many of them came from the charter sector where parents must opt their children into their particular school. One implication is that the participating students and their families are likely different on unobservable dimensions from typical students attending a traditional public schools. Similarly, the teachers participating in the program are likely more representative of those working in the charter sector than in traditional public schools. Our findings may not generalize to those settings where technology access and technical support is more of a challenge than it seemed to be in the contexts we studied. That said, one of the major theories of action behind the existence of the charter sector is that these schools should serve as laboratories of innovation where educators could be given greater autonomy to develop new effective practices that could then be shared out and implemented more widely in the traditional public sector. Indeed, there is evidence that effective charter sector practices can be transplanted to traditional district schools (e.g., Fryer, 2014). Therefore, our findings still may provide useful lessons for the traditional public school sector.

Results

Stakeholders perceived that students made academic improvements

Partner and mentor teachers overwhelmingly perceived that students benefitted academically from participating in NSSI, although several acknowledged that there were limits to what could be accomplished in five weeks. This theme related to positive perceptions of student learning came up with 23 of our 28 interviewees, and was tied for the most frequently used code.

One ELA partner teacher described progress her students made with close reading and drawing connections between texts and real life:

I honestly felt like all of my kids, they improved… The idea that you can read a poem, and it could have a literal meaning and a deeper meaning… you're not just reading to get done and look for your teacher to ask you some very basic questions, but that this article is supposed to make you think about life… things that you're experiencing today. That skill was not even there and I saw them develop that. And most of them told me… I read differently now. And I pay attention to what I read. And I'm going to be thinking about what I read and what it means for my life. –Partner Teacher 1

Math teachers similarly indicated that students at a variety of skill levels were able to become more flexible mathematicians:

My students benefited a ton through this program. Even my students that may have caught on to the concepts quicker… they were more flexible… when they finished, they … would try a second or third method because they had learned different ways of doing it from the discourse the day before. Even my kids who might not have had access to multiplication or division or are not as strong with those facts, they would then feel more confident with answering the questions because… even though they can’t do the quicker and most efficient method, they still were able to pull out other methods that they felt more confident in. –Partner Teacher 2

Survey results presented in Tables 3 and 4 echo these themes. Among partner teachers responding to the end-of-program survey, large majorities agreed that students improved their academic skills (82%), gained confidence in their academic abilities (83%), and became more interested in school and learning (72%) due to NSSI. Mentor teacher survey respondents unanimously agreed that students gained academic skills, confidence, and interest in learning as a result of NSSI. Eighty-six percent of both partner and mentor teachers indicated NSSI improved their perceptions of virtual learning, although some interviewees were quick to point out that certain things are either not possible or not optimal in virtual settings.

Table 3 Partner teacher endline survey results
Full size table
Table 4 Mentor teacher endline survey results
Full size table

A majority of student survey respondents agreed that they grew as readers (81%) and mathematicians (75%), became more confident in their reading (68%) and math (65%) skills, and became more interested in school and learning (54%) during summer school, at least among those who completed the end-of-program survey (see Table 5). Parent responses (see appendix) exhibit a similar pattern. Most agreed their children improved in reading (75%) and math (77%), gained confidence in their academic abilities (75%), became more interested in learning (71%), and discovered a new interest (64%) due to summer school. That said, we acknowledge the difficultly of interpreting these findings without a comparison group as it is unclear whether NSSI participants perceived they grew more academically than non-participants (and if so, how much more).

Table 5 Student Survey Endline Results
Full size table

Stakeholders perceived that the content was rigorous, relevant, and engaging—especially in ELA

Teachers told us that a key program strength was the content covered by the curriculum. Themes related to the high quality of the curriculum were among the most frequently mentioned in our interviews. More than twenty interviewees emphasized that the curriculum was rigorous, culturally relevant, and engaging.

Teachers were especially effusive about the novel choices, describing them as high-quality texts that elicited significant student investment. Students were gripped by these novels, attached to the characters, and eager to find out what happened next. Teachers emphasized that, while engaging content is always important, this is especially critical in virtual settings where promoting engagement can be challenging. One ELA mentor explained:

In an ELA classroom, the text is the most important thing... You need to pick something that students are going to want to talk about. Being online… you need to pick a short text… not only do you want to choose content that's going to be really engaging, challenging for students, but really relevant and something that they can latch on to, it also needs to be something that they can feel like we're moving fairly quickly through this versus oh we’re slogging through… The curriculum for [my grade’s] ELA class was amazing. It was an incredibly relevant and poignant text. Students were really invested… there is a particular scene in the book where you find out that two characters have died … as soon as they read that portion, [the partner teacher’s] class was kind of silent. And then a student who really hadn't spoken for most of the summer unmutes himself and goes, ‘Really? … Both of them had to die?’ and then hits mute again, and she could just tell that they were so upset… so affected by what happened… We want students to feel that invested and that connected. –Mentor Teacher 1

Several teachers emphasized the cultural relevance of the ELA curriculum and ways the texts allowed students to draw connections to current events or their own lives. Teachers also highlighted that the novels and close reading texts were thoughtfully paired to reinforce concepts. One partner teacher put it this way:

The novel we read basically was about oppression and a girl fleeing from her native land to the United States. That opened up many different conversations because we know what's going on in our world today, as far as from Covid, to black and brown people being racially profiled, racial injustice… us reading that novel—and it was a 12-year-old girl relating to the students—this girl had times where she was hiding in a closet, not able to go anywhere. Students were like, look I'm indoors because of Covid. There were so many connections that I feel were so intentional. The students recognized that which helped spark that interest… Also, there were nonfiction articles we read, and related to the novel… And when students know… the why behind things, they're wanting to continue to investigate and learn, but if they don't see the connection, … it's more like… why am I learning about this? … At first though, it was a mystery. We started out with the nonfiction piece, and they were like, whoa, this is some deep stuff. Wow... I just don't understand. And then when I started reading the novel, they're like, oh, that's who that leader was? Oh my gosh, the leader! So it's like we gave them a suitcase filled with information. And once they got to the to the novel, they're like, oh, I get why we packed that in our suitcase, they started to make the connection, and it got them to the greater theme in the end. –Partner Teacher 3

Teachers further suggested that the curriculum expanded students’ knowledge and awareness of global historical events:

A fairly straightforward strength [of the] program is they started with really good books… The book we read was about the partition of India, this super powerful and important moment in history in a beautifully told book, kids learned a lot about history, kids discussed Gandhi's ideas. I don't know, what do you want your own kids to be doing? Probably reading a really good book, discussing important ideas about the world …you want books to be both mirrors for kids that they can see their own culture affirmed and honored, but you also want books that can be windows for kids and help them see into new experiences that are different from what they know. And I thought the books that they chose did a pretty great job of providing a little bit of both. –Mentor Teacher 2

A large majority (84%) of partner teacher survey respondents agreed that the curricular materials provided by NSSI were strong. Students gave additional indications that they found the program engaging, with 77% reporting that the energy in their online classrooms was positive and 65% indicating they were happy to be in summer school. In sum, stakeholders perceived that the ELA curricular content and novel selection was a key program strength.

Stakeholders found the math pedagogy accessible and rigorous but believed the math content could have been more culturally relevant

Teachers told our interviewers that the approach to math instruction was engaging and rigorous for students at a variety of skill levels. The focus on developing flexibility and multiple methods of answering the same problem allowed lower-performing students to engage with the material by finding their own way to solve. It allowed higher-performing students to continue engaging even after they had discovered their first method of solving. One math partner teacher shared:

The rigor level was, for the most part, ‘low floor high ceiling,’… any kid can access it, and then they all allowed for multiple ways of solving which is really what made the discussion and the math really rich because even if this problem seems kind of easy on face value… there were so many ways of representing all this stuff, my high [performing] students weren't bored the whole summer... If you are coming in at a lower level, you can still access the problem. If you are coming in very strong in math already, there's still more you can do to make your work even better and to build your flexibility in solving problems. Especially because there was no order or progression in the way that they were presented. It could be anything from the whole year any day and that in itself was pretty rigorous. –Partner Teacher 4

An 81% majority of student survey respondents agreed that they learned new strategies to solve math problems, and 72% agreed they became increasingly comfortable solving math problems in more than one way. Teachers also argued that the discourse approach to mathematics, encouraging students to talk through their reasoning and ask questions of one another, fostered high levels of engagement. One partner teacher explained:

Because the conversations were being led by [the students], I think it was just way more enjoyable for them as well. And it was so much more enjoyable for me, because I was like, ‘man, this is like, actual fun and the kids don't hate it.’ And the kids that would come and had no clue, if they didn't understand the question the day before, they were the ones that then came in, and were asking so many questions when other people would share their work. ‘Why did you do it this way? What is that?’ Usually, I'm used to those kids just sitting there silently, like, ‘I didn't get it, so I'm not going to participate’ but I think the discourse opened up that opportunity. If you didn't get the question, you still had so much opportunity to participate… It was not like a normal summer school at all. –Partner Teacher 5

By the end of the program, 63% of student survey respondents agreed they were more comfortable sharing ideas in math class.

Despite these strengths, teachers indicated that they believed the math problems themselves could have been more relevant to increase engagement further. Interviewees described the problems as “bland,” “boring,” “standard” and “not particularly innovative.” One mentor teacher described it this way:

I love the teaching approach… but the actual problems we put in front of kids were very ‘blah,’ to put it bluntly. There was a lack of cultural relevance and a lack of just like, ‘I'm a teenager or preteen and I want to do math that actually is interesting to me or sparks some kind of interest versus some random problem about someone selling lemonade.’ –Mentor Teacher 3

Several teachers noted that at least one mentor teacher tried to infuse the math curriculum with greater cultural relevance. While fellow teachers appreciated the intentions behind this effort, they noted that one math problem she developed—with a set up related to the “three-fifths compromise” under which enslaved people were treated as less than a full person in allocating representation under the U.S. Constitution—backfired. Without introduction to put the problem in context, some partner teachers and parents found the exercise offensive. A handful of interviewees suggested that NSSI leaders, despite their overall anti-racist orientation, could have done more to address this incident head on.

Stakeholders reported lower levels of student enthusiasm for the asynchronous movement and mindfulness content

An important feature of NSSI was its emphasis on synchronous instruction. One exception were the movement and mindfulness classes which were pre-recorded and distributed to students for independent viewing. Administrators did not find this approach to work especially well, with fully half of respondents disagreeing with the statement “movement and mindfulness was effective” (see Appendix Table 7).

Teacher interviewees also indicated low levels of engagement with this content. One math partner teacher suggested this was due to the lack of synchronous interaction:

The one thing they could kind of beef up a bit—but it was actually a great concept—was the movement and mindfulness. We were able to eliminate that block because our scholars did not respond to it. They really were not interested … when it's all just a video, and they can't have any input, our scholars tend to zone out. –Partner Teacher 6

Student survey respondents indicated low levels of engagement, with 27% by the end of the program indicating they had never attended in the past week. Open-ended survey responses suggest that the asynchronous nature of the content was to blame. One student wrote, “I would go more if it was live.” Others described the class as “boring” or needing more variety.

Partner teachers perceived that the program improved their instruction

Overall, partner teachers felt that they improved their teaching as a result of participating in NSSI. Interviews revealed this was, in large part, due to access to the mentor teachers, who they generally perceived to be talented educators. The most common mechanism through which partner teachers suggested this occurred was by watching videos of mentor teachers leading classes through the same lessons partner teachers would teach four to five days later. Even more experienced teachers said this was helpful both in providing models for teaching the lesson and anticipating how students were likely to respond to particular parts of the curriculum. Representative comments from ELA teachers include:

It was helpful—the mentor videos especially—to watch them and learn, okay, this is when she asks this question, and those transferable questions were really helpful, that was newer to me. So it was helpful to see how they would take a paragraph or part of the book and kind of break it down … to see another teacher who's more experienced than me teach her students and where she would pause and what the key points were. –Partner Teacher 7

It was great to see someone else already roll out the lesson. In part because seeing other students’ responses helped me anticipate what direction my kids may or may not take. And then I think it's always helpful to watch other teachers teach. That's actually not something we have a ton of time or opportunity to do in a regular school setting… Those mentor teachers are now some of the people I've observed teaching the most, in my entire six years of teaching, because I got to see them do a full 45-minute lesson every day. Whereas really thinking about like any colleagues I've had over the past six years, I've never seen anyone do a complete 45-minute lesson, let alone every single day. Just having that experience of getting to observe another really excellent teacher, was just great. –Partner Teacher 8

Survey data echoed these themes. Among partner teacher respondents, 80% agreed the “daily lesson videos and lesson flow documents provided by the mentor teacher are strong” and 79% agreed, “I am learning from my mentor teacher,” including 53% who strongly agreed. Among administrators, 67% agreed the “daily videos and lesson flow documents provided by mentor teachers are strong.”

Some partner teachers indicated that access to mentors from across the country was particularly valuable for those whose home districts had a limited supply of highly effective teachers. One partner teacher explains how this was true for hard-to-staff subjects:

Where I live, the math teachers I feel generally don't really understand the Common Core content. And I think there's a shortage of math teachers. So I think [NSSI] gave teachers a time to learn from people who really understand the content… and who really could teach them how to teach. –Partner Teacher 6

Partner teachers also appreciated that mentors taught the same lesson before they did and informed them about what worked and what did not. Mentors indicated that teaching these lessons built their credibility with partner teachers and improved the PD they provided. One math mentor explained:

I've done some whitewater kayaking in my in my life—and it's like the first run, right? You go out there, you get knocked over, you figure out where the eddies are, where the rocks are, and the currents, and then you come back and you say, ‘Look, I survived the thing and let me tell you all about it.’ So I think the teachers on the whole really appreciated that approach. I wasn't with them. I was one of them. I was just going four days earlier, trying to try things out, and some work, some didn't. I came back and reported on it, showed them the video, and then they could learn... And I would say you know never in the history of education has every second of every piece of instruction been recorded. Right? And that's what we what we accomplished... the promise for teacher development in that is really powerful.” –Mentor Teacher 4

In terms of the skills partner teachers developed, interviewees indicated that the program helped them prepare to teach more effectively online in preparation for a virtual or hybrid school year. Many suggested that they went on to become leaders at their home schools, providing guidance and support to colleagues who were less experienced with virtual learning. A large majority (87%) of partner teacher survey respondents agreed with the statement, “I will be a better teacher in the 2020–21 school year because of teaching at NSSI,” including 57% who strongly agreed.

In interviews, many teachers told us that they believed they improved their ability to promote student engagement during NSSI. This was an important area of growth given it is something teachers saw as a major challenge of virtual instruction as evidenced by the fact that our code “virtual lessons – engagement is key” (indicating a takeaway about virtual learning) was among the most commonly applied codes across interviews. Partner teachers further argued that they improved on analyzing student work, giving feedback to students, featuring examples of student work to increase engagement, creating joyful online classrooms, and raising their expectations for what students could accomplish virtually. Others suggested that the program helped them realize that culturally relevant curriculum is important and possible to implement.

Mentor teachers enjoyed the program and felt their own practice improved

Mentor teachers were some of our most enthusiastic interviewees, suggesting that they appreciated the opportunity to participate in the program—despite the heavy commitment it involved—and even that their own teaching practice improved as a result. Mentors especially appreciated the ability to develop relationships with and learn from a community of other excellent teachers across the country, as two teachers (one math and one ELA) explained:

The quality of people that were hired was unbelievable—a lot of really talented, smart, thoughtful, hardworking folks of a variety of different backgrounds really helped us have a pretty rich conversation about many topics, ranging from pedagogical approach to what does our organization stand for when we talk about fighting for racial equity ... The people component was really wonderful. –Mentor Teacher 3

I really loved it… I would not say that it was just a walk in the park but one of the things that I really enjoyed was the chance to collaborate with other educators and to hear their perspectives from a number of different contexts… being able to learn from [a fellow mentor teacher] and the vast amount of experience that he has and being able to collaborate together. I feel like the professional relationships that I developed, I wouldn't trade those for anything. That alone made the summer worth it. –Mentor Teacher 1

In addition to working with partner teachers and providing guidance based on their own efforts to implement the curriculum, mentor teachers appreciated having a role in creating the program with NSSI leaders and felt they were given substantial autonomy to experiment:

What I loved about the organization was that it basically said, look, we have an idea of what we think the model is, but a big part of the model is hiring what we think are amazing teachers and letting them run with it… There was this kind of founders, innovation kind of spirit that was pervasive. There was a structure, and then within that structure, I felt like a musician who could riff. I could be like, ‘oh, I'm going to try this, I'm going to put on a costume, I'm going to say this thing, I'm gonna spend eight minutes on this or 12 minutes tomorrow. I felt a real sense of freedom and innovation in the model that was really powerful. And that innovation done in small ways over and over led to a better and better experience for the kids over time. And because of our size, relatively small, the sharing of those best practices across the teachers, you just saw things get adopted and shared really quickly. –Mentor Teacher 4

Finally, mentor teachers appreciated that NSSI allowed them to expand their reach and have greater impact on students and teachers without leaving the classroom:

[NSSI] provides a really big platform for a classroom teacher. This whole debate in my field of, well, if you teach in the classroom, you only get to teach this many kids but if you become the superintendent or the director of this or that, you teach thousands of kids, but you don't really teach them you do things that allow them to learn more. This is a way to stay in the classroom but then have way more kids be impacted by the stuff you're doing, which is unique. –Mentor Teacher 4

Survey data was consistent with the interviews. Mentor teacher respondents unanimously agreed with the statement “I am happy that I am a mentor teacher this summer with NSSI,” including 57% who strongly agreed, and also unanimously agreed that “NSSI increased my enthusiasm about being a teacher.” Satisfaction appeared to increase over time, as we show in Appendix Table 10. Our interview data suggest that this was due to the heavy workload as the inaugural program launched as well as mid-program adjustments leaders made based on early feedback.

Teachers appreciated receiving adaptable curricular materials

Another key program strength that partner teachers highlighted was the fact that NSSI provided them with what they perceived to be high-quality lessons rather than asking individual teachers to develop their own. This saved a significant amount of preparation time that they could instead use on other aspects of their teaching. However, they appreciated the flexibility they had to use these materials as they saw fit and to adapt them to their own teaching style and student needs. While some teachers actually played parts of the mentor teacher videos for their students, hit pause, ran the discussion, and then returned to the lesson, others simply watched the videos on their own in preparation for teaching their classes. As two partner teachers explained:

I've taught summer school loads of times through those 31 years and most of the time I had to just create stuff—so being able to have access to high-quality materials really streamlined what we were doing at summer school. We were really able to focus on meeting kids where they were at and meeting their needs, and really providing some extra education. –Partner Teacher 9

Not only did we have a very detailed and well thought out lesson plan to read, we also had a video of a person actually teaching this lesson. And so that just made preparation, I honestly don't know what more you could want… Teaching is hard and there’s not enough time… so much time goes into crafting lessons and planning. If more teachers had all of that time to (1) really internalize the lessons, (2) really focus on student work, give feedback and adjust for the next day… That’s how you get the best outcomes for students is when the teachers really deeply know what they’re teaching and where they’re going with it. And I still think the NSSI model is adaptable and leaves lots of room for people to be teachers in their own way. It’s not scripted… I could see the same lesson being carried out different ways at different schools or classrooms. –Partner Teacher 8

One of the ELA partner teachers (Partner Teacher 11) who taught a group of students at NSSI who he had not worked with in the past, when asked whether he was able to build a relationship with them, responded, “Totally. And part of that was because I wasn't spending so much time setting up lesson plans and worried about this and that. The fact that I knew certain things are already set up for me, it allowed me to spend time just getting to know the kids, their strengths, their weaknesses, their likes, and it just kind of flowed.”

Teachers wanted the professional development to be more differentiated

Although partner teachers generally felt NSSI helped them improve, they had suggestions for how to improve the pre-program and daily PD sessions. At the program’s start, teachers felt there was too much time built into the schedule for PD. Leaders responded by reducing the time—which was appreciated—but some teachers indicated that it was still too frequent and the placement in the schedule made it difficult to get the most of the PD because there was limited time between class and PD sessions. Partner and mentor teachers agreed expectations for PD attendance were also unclear.

Furthermore, several teachers suggested that components of the PD could have been further differentiated by teacher experience level, particularly to increase the value for more experienced partner teachers. One ELA partner teacher explained:

Everyone I worked with from my particular school, most all of us are veteran teachers. And I remember going through even the trainings and we were like saying this was not catered to teachers who've been core veteran teachers. This is catered to teachers who don't really have content knowledge. The questions that were being brought up by some teachers from other networks were questions about like, how do you do these simple things that are first year teacher questions. So all of us had similar feelings that this was not differentiated. –Partner Teacher 12

Mentor teachers we interviewed agreed that this was an area for improvement in terms of both differentiation and using the time in a less top-down way to engage the partner teachers more actively. One math mentor teacher argued:

The areas of improvement would just be around differentiation for teachers in terms of how we supported them in supporting providers was more geared towards novice teachers, and not kind of like support for more veteran teachers, or at least space for collaboration for veteran teachers. –Mentor Teacher 5

This was consistent with the survey data. Among administrators, only a bit more than half (54%) agreed that the “daily intellectual prep PD made my teachers stronger” and only 41% agreed that the “daily student work analysis made my teachers stronger.” Among the mentor teachers, 57% thought the intellectual prep was making teachers stronger, and only 29% agreed that the student work analysis time was making partner teachers stronger. Partner teachers were a bit more positive about these components, with 61% agreeing the intellectual prep was helping and 57% agreeing that the student work analysis time was helping. Unfortunately, we do not have data on teacher experience for the survey sample to test whether newer teachers were more satisfied with the PD.

Discussion

The findings from this study suggest that it is indeed possible for virtual learning to improve participant perceptions of student academic outcomes and engagement with school, to improve teacher morale and perceived teaching abilities, and to more equitably expand the reach of excellent teachers. The National Summer School Initiative program provides a unique positive proof point of an online learning experience that rising 4th–9th grade low-income students of color perceived as engaging, that their teachers considered to be a positive experience, and that capitalized on virtual platforms to extend the reach of talented teachers. Participating teachers, students, and parents believed that learning is possible in a virtual environment and felt that the program improved their perceptions of what can be accomplished with online education. We find that the program generally succeeded in providing what students, teachers, and parents believed was a high-quality learning experience for thousands of students whose schooling had been interrupted by the Covid-19 pandemic while also providing professional growth opportunities for participating educators. Partner teachers and students who completed surveys reported marked perceived growth in student learning and intellectual confidence. Participants perceived the curricular content to be high-quality, engaging, and culturally relevant, especially in ELA.

Our study therefore adds to the existing literature by capitalizing on a unique innovative effort amidst the pandemic to promote educational equity via online learning, and ultimately provides an example of virtual learning among a more generalizable population of students than has typically been studied in previous research on K-12 online schooling. Our findings in many ways diverge from the existing literature which suggests that student satisfaction is a major challenge for virtual learning by suggesting practices that can promote online engagement. Several of the tentative prescriptions for increasing engagement emerging from this pandemic-era study are consistent with earlier research identifying exceptional online environments where engagement levels were high, suggesting that meaningful conent is critical and suggesting the superiority of synchronous over asynchronous formats.

At first glance, our work may seem inconsistent with the previous research on online charter schools which has been generally gloomy. However, this may be explained by the different circumstances through which charter school students came to NSSI relative to those opting into online charters that have been previously studied. More specifically, NSSI students who came to NSSI from charters had opted into an in-person charter school that in turn opted into a specialized virtual summer program due to a public health emergency. In contrast, students in the online charters had opted into full-time online learning pre-pandemic. These populations may be quite different in ways that could influence the generalizability of results (from both our study and the existing literature on virtual charters).

Our study also reveals potential lessons for teaching and learning more generally, regardless of whether it be in a charter, traditional public, virtual or in-person setting. Our findings suggest that high-quality, rich, timely content that students can see connecting to their lives or current events can be useful for generating student engagement. Pedagogy that allows students at multiple levels to access the content may also be valuable for promoting engagement in math. The results suggest that teachers perceive culturally relevant content to be helpful for student engagement, consistent with evidence on the causal effects of programs such as ethinic studies on engagement and achievement (Dee & Penner, 2017). However, our findings also illustrate how efforts to incorporate such content can backfire. This suggests the need for thoughtful design from the start, and that cultural relevance can be challenging to accomplish on the fly.

Beyond the perceived direct effects on student learning and engagement, we sought to understand teachers perceptions of online teaching and professional development. Indeed, we found that NSSI partner teachers valued the chance to work in sustained partnership with an expert mentor who was teaching the same content, while the mentor teachers appreciated the unique opportunity to extend their reach. Mentor teacher videos provided powerful models according to partner teachers. This could be accomplished within in-person settings but virtual learning facilitates the creation of videos that can be curated into libraries of teaching practice. In the future, online learning could facilitate more systematic observation of partner teachers by talented coaches than has historically been possible in traditional in-person schools. This would be consistent with other research demonstrating the positive effects of video-based observation on teacher perceptions of the evaluation process (Kane et al., 2020; Quinn et al., 2018).

We also see implications for teacher development within and beyond virtual learning environments. Mentor teachers’ experiences suggest that programs that differentiate teachers’ roles based on experience and effectiveness can generate enthusiasm among talented educators and multiply their impact. Partner teachers’ experiences suggest that sustained partnership with a talented mentor who shares materials and practices was perceived as a powerful instructional improvement strategy, with potential for both online and in-person instructional formats and consistent with prior research (e.g., Coggshall et al., 2012; Ehrich et al., 2004; Garet et al., 2001; Kennedy, 2016; Kraft et al., 2016; Solis et al., 2012; Steiner et al., 2021). This is consistent with previous research suggesting the key role that instructional coaches play in mediating education policy (e.g., Woulfin, 2018; Woulfin & Rigby, 2017; Hashim, 2020). Our findings also provide pandemic-era evidence consistent with previous research suggesting that professional development differentiated to teachers’ needs and levels of experience is perceived as more valuable thant general efforts (Gulamhussein, 2013). Teachers also appreciate coaching that is delivered by someone who has previously taught the material and can therefore provide instructional guidance with credibility.

Importantly, our results strongly suggest that teachers need not reinvent the wheel when it comes to curriculum. Centralized efforts to provide high-quality—but still adaptable—lessons to teachers can save them valuable time that they can instead devote to other important tasks such as internalizing lessons prior to class sessions, reviewing student work, providing feedback to students, meetin with students one-on-one, spending time getting to know students better, and ultimately building relationships. This seems especially relevant in the times of coronavirus when educators are overwhelmed with the logistical challenges of physical distancing and simultaneously developing virtual teaching skills but is probably a useful lesson for leaders regardless of the broader circumstances. Notably, our findings on teacher perceptions are consistent with existing evidence on the causal effects of providing curriculum for student achievement (Jackson & Makarin, 2018).

Finally, our results provide suggestive evidence that online learning can be harnessed to provide more equitable access to high-quality teaching. One advantage of virtual over in-person learning is that physical geography is not a constraint. At NSSI, talented mentor teachers from across the country were able to extend their reach, working with students and teachers outside of their home states through an online platform. Given greater openness to virtual learning in the aftermath of its widespread use during the COVID-19 pandemic, educational leaders could capitalize on this feature of online learning to increase the access low-income students of color have to the highly effective teachers who are currently inequitably distributed across schools (Boyd et al., 2005; Lankford et al., 2002). Although there continue to be downsides to virtual learning that will likely make in-person instruction preferable in the post-pandemic era, the field should consider carrying forward the use of technology to expand the impact of the limited pool of highly effective teachers to those students who otherwise have the most limited access and those teahers who can learn from these talented educators.