Advertisement

Efficacy of a blended learning approach to elementary school reading instruction for students who are English Learners

  • Elizabeth R. KazakoffEmail author
  • Paul Macaruso
  • Pam Hook
Open Access
Development Article
First Online:

Abstract

U.S. Department of Education data continually indicate that students who are English Learners (ELs) underperform in reading compared to students who are not English Learners (non-ELs). This study examined whether a personalized, adaptive blended learning approach can support reading development in ELs and non-ELs. There were 442 ELs and 442 non-ELs matched on grade level, beginning-of-year aimsweb tier status, and placement level in the blended learning program (Lexia Reading Core5 ®). After 1 year, both groups showed significant gains on aimsweb with no differences between groups for kindergarten, and second through fifth grade. In first grade, students who are ELs outperformed students who are non-ELs. For students who continued the blended learning program for 2 years, ELs and non-ELs showed similar changes in aimsweb tier status. Notable reduction in the percentage of students identified as at-risk for reading failure was found in both the EL and non-EL groups.

Keywords

Elementary Technology English Learners (ELs) Blended learning 

Introduction

Proficient reading relies on a solid foundation of oral language skills including phonological awareness, vocabulary, structural analysis (morphology), discourse processing, and pragmatics (Lesaux and Geva 2006). In addition, effective reading requires the ability to master phonics and demonstrate fluent word recognition skills. Difficulties in one or more of these areas can pose challenges in learning to read (August and Shanahan 2006). Unfortunately, difficulties in these areas are frequently seen among students in the United States. According to the National Center for Education Statistics (U.S. Department of Education 2015), only one-third of fourth graders scored at a proficient level in reading and scores did not improve between 2013 and 2015. Weaknesses in reading are even more pronounced for students identified as English Learners (ELs). Based on the National Assessment of Educational Progress (U.S. Department of Education 2015), a striking 92% of students who are ELs scored below the proficient level in reading compared to 62% of non-ELs.

Approximately 10% of students in the United States are ELs (U.S. Department of Education 2015), and this percentage is expected to rise to approximately 25% by 2030 (Cheung and Slavin 2012). A key contributor to the academic difficulties faced by ELs is that nearly 70% are from a low socioeconomic status (SES) background (National Education Association 2008), which includes lower family incomes, poorer neighborhoods, lower parental education levels, and more limited educational resources than students who are non-ELs (Goldenberg et al. 2006). These social inequalities contribute to discrepancies in academic achievement as students progress through elementary school (Vadasy and Sanders 2011). For students from low SES backgrounds, 80% failed to meet reading proficiency milestones (Campaign for Grade-Level Reading 2014) as students from low-SES families are less likely to have experiences that develop fundamental reading acquisition skills, such as phonological awareness, vocabulary, and oral language (Buckingham et al. 2013). In 2011, students who were eligible for free or reduced lunch (an indicator of low SES) scored 17 points lower in reading than students not eligible (U.S. Department of Education 2011) and the achievement gap between National School Lunch Program (NSLP) and non-NSLP students has not narrowed since 2003 (NEA 2008). These reports indicate that SES is closely intertwined with EL status, and that low SES factors are likely contributors to academic difficulties faced by ELs.

Studies have shown that students who are ELs typically follow the same trajectory as non-ELs in learning to read. For instance, ELs and non-ELs take similar paths in developing phonological awareness (Gersten and Geva 2003) as well as phonics and word recognition skills (Lesaux et al. 2006). However, in terms of comprehending written materials, non-ELs generally outperform ELs, often due to limitations in vocabulary and word knowledge seen with ELs (Lesaux et al. 2006). These limitations contribute to academic discrepancies found in ELs compared to non-ELs. For instance, students who enter kindergarten with limited English skills have lower achievement levels than non-ELs by grade five, with large to moderate effects, depending on demographic risk factors such as SES (Kieffer 2008).

The best approaches to teach reading to non-ELs often apply to ELs, including explicit instruction in phonological awareness, phonics, vocabulary, structural analysis, fluency, and reading comprehension strategies (August et al. 2014; Gersten and Geva 2003; Vadasy and Sanders 2011). Classroom instruction should incorporate clear objectives and frequent assessments of student progress (Goldenberg 2012), be applicable to students at all reading levels, allow for personalized learning, and utilize texts matched to reading ability (Espinoza 2008; Goldenberg 2012). Instruction should also provide ample opportunity for exposure to academic printed materials, encourage independent reading, and foster oral/written communication about reading contents (Francis et al. 2006).

The features described above set the bar for quality reading instruction. However, even with most of these features in place, ELs may still lag behind non-ELs, especially in the area of reading comprehension (August and Shanahan 2006). As highlighted by Goldenberg (2012), traditional classroom practices alone may be insufficient to aid ELs in learning advanced academic materials and thus closing the achievement gap with non-ELs.

Research has shown that ELs may benefit from intensive interventions that extend beyond typical classroom instruction, including use of 1-to-1 tutorial sessions or small group instruction. For instance, Vadasy and Sanders (2011) implemented a supplemental phonics program in 1-to-1 tutorials lasting 120 min per week over 20 weeks. The program produced significant gains in word reading for EL first graders, but only for students with higher oral vocabulary scores. In an earlier study, Kamps et al. (2007) examined the effects of direct phonics instruction in a year-long, small group intervention with first and second grade ELs and reported that intervention students showed superior oral reading fluency scores compared to ELs receiving typical classroom instruction. More recently, Begeny et al. (2012) implemented a reading fluency program called Helping Early Literacy with Practice Strategies (HELPS). This program requires 1-to-1 instruction with a trained adult and is implemented outside of the classroom. ELs receiving HELPS showed significant benefits in reading fluency and comprehension compared to ELs not receiving the intervention (Begeny et al. 2012).

A few studies have reported that technology-based interventions may be beneficial for ELs. For instance, Troia (2004) found that Fast ForWord Language, a highly-intensive program featuring acoustically modified speech, helped improve expressive language and word recognition in low performing ELs. Macaruso and Rodman (2011a) examined Lexia Reading, a supplemental phonics program, and found significant benefits for EL kindergartners in phonological awareness and word recognition. Rodríguez et al. (2012) also used Lexia Reading and found superior gains in reading comprehension when EL first graders received instructions in their native language (Spanish) rather than English. In general, these results are consistent with other studies showing that use of technology can support reading development in at-risk students (e.g., Cassady and Smith 2005; Hecht and Close 2002; Macaruso et al. 2006; Macaruso and Walker 2008; Macaruso and Rodman 2011b; Mitchell and Fox 2001; Saine et al. 2011; Torgesen et al. 2001).

The studies reviewed above point to the benefits of interventions for ELs, including use of technology-based programs. However, with educational budget cuts, teacher shortages, and increasing percentages of ELs, the need for quality instruction without the time commitment and resource-heavy elements often seen in intervention studies is becoming increasingly important. Blended learning, an approach that integrates technology with teacher-led instruction in the context of a typical classroom, may be particularly well-suited to address these issues and has recently grown in popularity in grades K-12 (Staker and Horn 2012; Powell et al. 2015). Blended learning allows one to address all aspects of effective instruction for ELs in a more efficient manner, using technology to support teacher-led instruction, including personalization and data collection to inform differentiated instruction (Christensen et al. 2013). Preliminary findings from a small-scale study with a subsample of ELs (Schechter et al. 2015) suggest that a blended learning approach may be particularly effective for ELs; however, more evidence is needed.

Blended learning

Blended learning incorporates face-to-face, teacher-led instruction in conjunction with student-led digital activities in order to provide students with a personalized educational path (Horn and Staker 2011). With blended learning, students have some degree of control over the content, pace, time, and location of their learning (Powell et al. 2015). Real-time data often provided through digital technology in a blended learning approach help teachers differentiate instruction according to students’ varied progress (Horn and Staker 2011; Hilliard 2015). Based on the potential benefits of blended learning over traditional instructional models, blended learning is gaining popularity not only for targeted populations, but also in general education settings (Horn and Staker 2011).

Blended learning can take various forms, thus allowing users to adapt a program that best fits their pedagogical goals and physical setting. Blended learning may include a station rotation, computer lab rotation, flipped classroom, enriched virtual, or individual rotation among other forms (Horn and Staker 2011; Christensen et al. 2013). In elementary schools, such as the current study, a station rotation is a commonly implemented form of blended learning. This form is considered a good fit for elementary schools because it builds upon the traditional classroom model of activity-centers (Evans 2012). In a station rotation, students rotate in small groups within the classroom to stations, including at least one digital component (Powell et al. 2015). A lab rotation, also implemented in elementary schools, consists of students visiting a computer laboratory for the digital component of blended learning. In some cases, schools take an eclectic approach to blended learning, utilizing both station and lab rotations (and even home use of the digital technology); together with teacher-led whole class and/or small group instruction.

Research regarding the potential benefits of blended learning is limited, especially in elementary school settings. In studies that have explored blended learning in higher education, students in a blended learning program self-reported as more motivated (Vaughan 2014), more supported (Lim et al. 2014), and provided with more helpful resources (Kim 2014) than peers in traditional classes. In both higher education and high schools, blended learning is being used to personalize learning by providing students with a larger variety of courses than can be offered in traditional classes (Hilliard 2015; Picciano et al. 2012).

Blended learning approach used in study schools

The current study explored the benefits of a blended learning approach to support reading skills in ELs and non-ELs at the elementary school level. The blended learning approach utilized in this study, Lexia Reading Core5 ® (Core5; Lexia Learning 2014) was chosen based on prior studies of demonstrated efficacy (Schechter et al. 2017; Schechter et al. 2015; O’Callaghan et al. 2016) as well as features that address key areas of effective instruction, including clear objectives, frequent assessments of student progress, personalized learning, and ample opportunities for exposure to academic printed materials (Espinoza 2008; Francis et al. 2006; Goldenberg 2012). The program is designed to accelerate reading skills in students who are behind their peers as well as sustain and increase progress for on-level and above-level readers. The contents of the program were developed for students in preschool through grade five and provide a structured (i.e., explicitly teaches word identification and decoding), sequential approach to instruction in the domains of phonological awareness, phonics, structural analysis, automaticity/fluency, vocabulary and comprehension/higher order thinking skills. (See the Appendix for a detailed description of the skills/strategies covered in each domain as well as activity examples). Core5 is aligned by grade level to the Common Core State Standards in reading (corestandards.org). It contains 18 levels of material, subdivided into 89 activities and 1,243 units, each of which takes 4-8 min to complete. Core5 was integrated into the classroom’s standard English Language Arts (ELA) program.

Integrations of Core5 into each school’s standard ELA curriculum often involved use of a station rotation or lab rotation model. Within a station rotation model in the classroom, students’ recommended program use time can be used to prioritize which students should spend time on the online program. In a lab rotation, a whole class can use Core5, allowing each student to work on skills appropriate to his or her needs. Students who need extra help can be pulled for a Lexia Lesson, either in the classroom or lab with a teacher or support personnel. Students in this study began Core5 through an adaptive auto placement tool, which places a student at their actual skill level, which may or may not align with their grade in school. For example, students may begin working on a Core5 level two or more grades below their grade level, one grade below their grade level, or in/above their grade level. Mastery of skills (90–100% accuracy) within a level is required to advance to the next level. The program provides immediate corrective feedback and explicit instruction when students struggle with an online activity.

For each activity in Core5, students begin in the standard step of instruction. For example, in Level 8 (Mid-End Grade 1 Skills), students address depth of vocabulary knowledge by focusing on multiple meanings of words. They read regular, single-syllable words (e.g., bat, pen, kid) and choose the two pictures (e.g., a baseball bat and a small black flying animal) out of six that illustrate two different meanings of these words. If students make an error, they move to the guided practice step, which simplifies the task to support skill acquisition. Scaffolds or modifications in this step may include removing choices, simplifying visual components, adjusting the complexity of language, changing the presentation of the task, or providing embedded support. In the above activity, the number of words is reduced to the ones they are struggling with and there are only three pictures to choose from. If students continue to make errors, they move to the instruction step, which explicitly teaches the skill, again focusing on students’ specific errors. In the example, the two meanings of the word “bat” are defined for students and they choose the appropriate picture based on the word “bat” used in a sentence (e.g., She swings the bat at the ball.). This scaffolding provides students with an opportunity to learn and practice skills in a supported environment.

Based on performance in the online program, Core5 specifies an individualized Prescription of Intensity (ranging from 20 to 80 min per week) designed to increase students’ chances of completing reading levels corresponding to their grade level [i.e., reaching End-of-Year (EOY) Benchmark]. Prescription of Intensity is based on students’ Performance Predictor, which indicates students’ likelihood of reaching EOY Benchmark. Performance Predictors were derived from logistic regression analyses of performance data from normed samples of users. Predictors are grouped into three risk levels (On Target, Some Risk, and High Risk) and can be interpreted as: “If this student continues to work at the same pace for the same amount of time as last month, his/her chance of reaching EOY Benchmark is X %.” Larger Prescriptions of Intensity are given to Some and High Risk students in order to increase their chances of reaching EOY Benchmark.

Core5 also contains online teacher reports which allow for careful monitoring of students’ progress and provide teachers with data-driven action plans and paper–pencil activities to differentiate instruction. Paper–pencil activities which include Lexia Lessons and Lexia Skill Builders allow students to meet the full set of Common Core State Standards by providing opportunities to work on skills that cannot be addressed through online activities alone. Standards that involve expressive language are particularly well-suited to the format of these off-line activities. Through the Lexia Lessons and the Lexia Skill Builders, students are encouraged to integrate spoken and written language skills as they build their listening, reading, speaking and writing abilities. Lexia Lessons can be taught in small groups with the teacher, and students may work independently or with peers using Lexia Skill Builders to further develop automaticity and expand expressive language skills. Before/after school and home use are also possible means of implementing aspects of Core5 as a blended learning approach.

Professional development training, comprised of two 90–180 min sessions, was delivered to participating schools by an implementation staff affiliated with Lexia Learning who have backgrounds in K-12 education. The first training session provided strategies for integrating Core5 within the classroom setting and the second focused on effective use of online teacher reports to track student progress and provide instruction based on the data collected through the program. The importance of integrating Lexia Lessons and Skill Builders with the online program was highlighted. During the school year, the implementation staff monitored student use of the program and provided support to schools to boost program use if schools were not meeting usage fidelity.

Schools were able to choose when they wanted to receive each of the two sessions. Typically, the first session of training occurred prior to the first use of Core5 by students. The second training usually occurred after a few months of data had been collected within the program. Schools also had access to online training guides, reference manuals, a video library called Training On Demand, and customer support services (via phone or email).

Research questions

Given that data continually indicate that reading skills for students in the U.S. are not improving and that students who are English Learners underperform in reading compared to students who are not English Learners, this paper explores if there is a benefit to using a blended learning approach to reading instruction and if the blended learning approach can support reading growth for both ELs and non-ELs. As such, a blended learning approach to reading instruction was evaluated over two school years and the following research questions were addressed:
  1. 1.

    To what extent can a blended learning approach support reading growth over the course of one school year for:

    1. a.

      ELs compared to non-ELs?

       
    2. b.

      At-risk ELs compared to at-risk non-ELs?

       
     
  2. 2.

    To what extent can a blended learning approach support reading growth over the course of two school years for:

    1. a.

      ELs compared to non-ELs?

       
    2. b.

      At-risk ELs compared to at-risk non-ELs?

       
     

Method

A large-scale, quasi-experimental group design was used to evaluate the effectiveness of a blended learning approach to reading instruction for ELs in comparison to non-ELs. The first “Year One” study compared reading growth over one school year for ELs and non-ELs matched in reading ability at the beginning of the school year, including additional analyses for at-risk students. A second “Year Two” study was also conducted, examining reading growth for matched ELs and non-ELs from the Year One study who continued with the blended learning approach to reading instruction during a second school year.

Assessment of reading growth was based on a standardized reading measure, aimsweb (www.aimsweb.com). Aimsweb is a universal screener and progress monitoring assessment for fundamental reading, math, spelling, and writing skills for grades K-12. As a universal screener, aimsweb establishes students’ abilities at the beginning, middle, and end of the academic school year. Aimsweb can also be administered more frequently as a progress monitoring tool for at-risk students. Using assessment results, students are categorized into a 3-tier benchmark system of on-track (Tier 1), at risk (Tier 2), and severely at risk (Tier 3). Additional details on aimsweb are detailed below in the “aimsweb variable” section.

Year one study sample

The sample consisted of matched groups of kindergarten through fifth graders (442 ELs, 442 non-ELs) from 64 schools that used Core5 in their blended learning approach to reading instruction and provided information about EL status. All students were from the same Midwestern state. The non-EL group was randomly selected from a population of over 2000 non-ELs to create matched EL and non-EL groups. ELs and non-ELs were matched by grade, initial Core5 placement level and beginning-of-year tier status on aimsweb. All students in the sample had beginning-of-year and end-of-year scores on aimsweb and used Core5 with fidelity, meeting usage requirements (i.e., Prescriptions of Intensity) for at least 20 weeks over the 2013–2014 school year. Included in the matched sample were 82 kindergarteners, 244 first graders, 146 s graders, 170 third graders, 140 fourth graders, and 102 fifth graders. Due to varying demographics by geography within the state, 34 schools contributed both ELs and non-ELs, 27 schools contributed only non-ELs, and three schools contributed only ELs. Eighty percent of ELs in the Midwestern state speak Spanish as their first language. For the five schools with the highest number of ELs in the matched sample, over 90% of the students in the schools also received free/reduced price lunch.

Year two study sample

Of the 884 students in the Year One study, 130 students (71 ELs and 59 non-ELs) continued to use Core5 with fidelity during the 2014–2015 school year. These students were part of the Year Two study. In the Year Two study, eight schools contributed both ELs and non-ELs, 18 schools contributed only non-ELs, and 20 schools contributed only ELs (Table 1).
Table 1

Sample description

 

Year 1

At-risk

Year 2

At-risk

EL

442

141

71

17

Non-EL

442

141

59

19

Total

884

282

130

36

Measures

Data were collected from the online component of the blended learning program (Core5) and performance on the standardized reading measure (aimsweb).

Core5 variables

Performance measures obtained from the program included initial Core5 placement level, number of hours using the online program, number of units and activities completed, number of Core5 levels advanced, and whether students reached their EOY Benchmark.

Aimsweb variables

Performance on aimsweb served as an outside measure of reading growth. Aimsweb is a universal screener and progress monitoring tool for fundamental reading, math, spelling, and writing skills for grades K-12. As a universal screener, aimsweb measures students’ abilities at the beginning, middle, and end of the school year. Aimsweb can also be administered more frequently as a progress monitoring tool for at-risk students. Individually administered, standardized subtests are available through aimsweb to measure students’ academic abilities. Three commonly used reading subtests are Letter Naming Fluency (LNF) for letter identification, Nonsense Word Fluency (NWF) for word decoding, and Reading Curriculum-Based Measurement (R-CBM) for oral reading fluency, each subtest is administered in 1 min (http://www.aimsweb.com).

Analyses were based on the beginning-of-year (BOY) and end-of-year (EOY) scores on the LNF subtest for kindergarten, NWF subtest for first grade, and R-CBM for second through fifth grade. Raw scores on these subtests were converted into percentile scores and tier status. Reliability indices for the early literacy subtests (LNF, NWF) range from 0.80 to 0.94 and fall in the low 0.90 s for R-CBM. Criterion validity correlations for the early literacy subtests range from 0.50 to 0.63 and from 0.47 to 0.81 for R-CBM, which is well within the acceptable range (See NCS Pearson, 2012, for details of all reliability and validity studies).

The percentile cutoffs for tier classification differ between the early grades—kindergarten and first grade—and the older elementary grades—second through fifth. There is a difference between the percentile cutoff for Tier 1 across these two groups because the earlier elementary grades were administered Letter Naming Fluency and Nonsense Word Fluency, whereas the older elementary grades were administered R-CBM. The aimsweb cutoffs for tier classification were developed through research on the specific subtests (Pearson Education 2011). For kindergarten and first grade, scores above the 35th percentile are considered Tier 1, scores between the 16th and 35th percentile are Tier 2, and scores below the 16th percentile are Tier 3. For second through fifth grade, scores above the 45th percentile are Tier 1, scores between the 16th and 45th percentile are Tier 2, and scores below the 16th percentile are Tier 3. Both raw scores and tier status were used in analyses. (For additional details on cut scores, please see http://www.aimsweb.com/wp-content/uploads/AIMSweb_Default_Cut_Score_Guide.pdf).

Analytic plan

A priori power analyses were conducted to determine the necessary sample sizes for the current study. Sample size, effect size, significance level, and power are intricately intertwined concepts (Cohen 1988). A priori power analyses determine appropriate sample sizes. In general, detecting a small effect size at a significance level of 0.05 requires larger sample sizes. Conversely, detecting a large effect size at a significance level of 0.05 requires smaller sample sizes (Cohen 1988). A priori power analyses test for the minimum recommended sample size given the hypothesized effect size at the given significance level (typically 0.05; Cohen 1988). Previous work on ELs’ reading abilities suggests that EL status has a moderate to large effect on reading achievement (Fry 2007; Kieffer 2008). For ANOVAs, f values of 0.25 and 0.4 are considered medium and large effect sizes, respectively (Cohen 1988). Using the pwr package in R 3.2.4 for Windows (Champely et al. 2015), power analyses were run for moderate and large effect sizes assuming two groups with a p = 0.05 significance level and a probability of finding the effect (power) of 0.8.

To examine student performance over time on aimsweb, repeated measures analyses of variance (RMANOVAs) were conducted in SPSS 24.0 for Windows. Separate analyses were run for kindergartners, first graders, and second through fifth graders because of the differing subtests given for each group. For kindergartners and first graders, initial Core5 placement level was used as a covariate. For second through fifth graders, the covariates included the grade level of students and initial Core5 placement level.

Greenhouse–Geisser adjusted F statistics are reported for tests within groups for all RMANOVAs to control for possible sphericity (Abdi 2010). The interaction between time (pretest, posttest) and EL status (EL, non-EL) was tested to examine whether the degree of growth from pretest to posttest differed for ELs and non-ELs. In order to examine pre- and posttest differences between groups, post hoc tests were run with Bonferonni corrections where the interaction between time and EL status was significant.

Results

The first set of analyses examined reading growth for the matched sample of ELs and non-ELs in the Year One study. The second set of analyses examined reading growth for ELs and non-ELs in the Year Two study. For both studies separate analyses were conducted for all students and for subsamples of at-risk students.

Year one study: Core5 results for all students

Based on the matching process, the ELs and non-ELs had the same Core5 placement levels at the beginning of the Year One study: 70% of students in each group began working on Core5 levels below their grade level and 30% began working on Core5 levels in their grade level. ELs and non-ELs used the online program for an average of 52 and 50 h over the Year One study, respectively, and both groups made similar progress in Core5. ELs averaged 431 units and 29 activities completed and advanced 5.9 Core5 levels. Non-ELs averaged 413 units and 28 activities completed and advanced 5.5 Core5 levels. By the end of the Year One study, 63% of ELs and 63% of non-ELs reached their EOY Benchmark.

A priori power analysis

Assuming a moderate effect size (f = 0.25), the suggested group size for the following analyses was 63.77. Assuming a large effect size (f = 0.4), the suggested group size for the analysis was 25.52. Therefore, the analyses presented have group sizes above 26 at the minimum, and group sizes above 64 were utilized where the data were available.

Year one study: aimsweb growth for kindergarten

In the matched sample, there were 82 kindergartners (41 EL, 41 non-EL). The overall effect of time was significant (Greenhouse–Geisser adjusted F(1,79) = 181.981, p < 0.001; f = 1.51), indicating that the kindergartners grew significantly from beginning to end of the Year One study. Both the overall effect of EL status (F(1,79) = 1.762, p = 0.188; f = 0.14) and the interaction between time and EL status were not significant (Greenhouse–Geisser adjusted F(1,79) = 0.910, p = 0.343; f = 0.11), indicating that ELs did not differ from non-ELs and the two groups grew at a similar pace (Table 2).
Table 2

Estimated marginal mean of LNF (controlling for BOY)

 

Time 1

Time 2

ELL

30.02

63.05

Non-ELL

36.27

65.90

Total

33.15

64.78

Year one study: aimsweb growth for first grade

In the matched sample, there were 244 first graders (122 EL, 122 non-EL). The overall effect of time was significant (Greenhouse–Geisser adjusted F(1,241) = 481.855, p < 0.001; f = 1.41), indicating that first graders grew significantly from beginning to the end of the Year One study. The overall effect of EL status was also significant (F(1,241) = 6.449, p < 0.05; f = 0.16), indicating differences in the performance of ELs from non-ELs (Table 3). In addition, the interaction term between EL status and time was significant (Greenhouse–Geisser adjusted F(1,241) = 5.743, p < 0.05; f = 0.15). Post-hoc testing with Bonferonni corrections indicated that ELs were not significantly different from non-ELs in the beginning of the Year One study (p = 0.195), but ELs performed significantly better than non-ELs at the end of the Year One study (p < 0.01).
Table 3

Estimated marginal means of NWF (controlling for BOY)

 

Time 1

Time 2

ELL

40.35

98.00

Non-ELL

37.30

86.15

Total

38.82

92.07

Year one study: aimsweb growth for second through fifth grade

In the matched sample, there were 558 s through fifth graders (279 EL, 279 non-EL). The overall effect of time was significant (Greenhouse–Geisser adjusted F(1,554) = 427.725, p < 0.001; f = 0.88), indicating that second through fifth graders showed significant growth from beginning to end of the Year One study. The overall effect of EL status was also significant (F(1,554) = 7.909, p < 0.01; f = 0.12), in which ELs performed better than non-ELs overall. Finally, the interaction between time and EL status was not significant (Greenhouse–Geisser adjusted F(1,554) = 0.465, p = 0.496; f = 0.03), indicating that ELs and non-ELs grew at a similar pace (Table 4).
Table 4

Estimated marginal means of RCBM (controlling for BOY and Grade)

 

Time 1

Time 2

ELL

65.80

106.43

Non-ELL

60.45

100.04

Total

62.12

103.24

Year one study: aimsweb tier status for at-risk students

This section examines changes in tier status on aimsweb for students who began the Year One study in Tier 3, thus considered at-risk for reading failure. There were 141 at-risk ELs and 141 at-risk non-ELs. At-risk students in the two groups showed similar changes in tier status. As seen in Fig. 1, 67% of at-risk ELs and 64% of at-risk non-ELs moved up tiers by the end of the Year One study. Notably, at least 40% of at-risk students in each group advanced to Tier 1 by the end of the Year One study.
Fig. 1

Tier Status on aimsweb for ELs and Non-ELs in a year one study and year two study

Year two study: Core5 results for all students

The two groups of students (71 ELs, 59 non-ELs) in the Year Two study had similar Core5 placement levels: 70% of ELs and 64% of non-ELs began working on Core5 levels below their grade level, and 30% of ELs and 36% of non-ELs began working on Core5 levels in their grade level. Both ELs and non-ELs used the online program for an average of 87 h and showed comparable progress in Core5 over the Year Two study. ELs averaged 668 units and 45 activities completed and advanced 8.5 Core5 levels. Non-ELs averaged 656 units and 46 activities completed and advanced 6.8 Core5 levels. By the end of the Year Two study, 79% of ELs and 83% of non-ELs reached their EOY Benchmark (Table 5).
Table 5

Estimated marginal means of RCBM (controlling for BOY and Grade)

 

Time 1

Time 2

Time 3

Time 4

ELL

57.05

94.12

76.74

115.55

Non-ELL

52.41

96.44

77.62

116.90

Total

54.73

95.28

77.18

116.23

Year two study: aimsweb growth on R-CBM

Growth on aimsweb in the Year Two study was analyzed (RMANOVA) for students in grades two through five (38 ELs, 30 non-ELs) who were administered the same subtest (R-CBM) at four time points (beginning and end of year: 2013–2014; beginning and end of year: 2014–2015). The overall effect of time was significant (Greenhouse–Geisser adjusted F(2.539,162.513) = 34.299, p < 0.001; f = 0.73), indicating that the students showed significant growth on the R-CBM over the Year Two study. The overall effect of EL status (F(1,64) = 0.000, p = 0.996; f = 0.00) and the interaction between time and EL status (Greenhouse–Geisser adjusted F(2.539,162.513) = 0.887, p = 0.435; f = 0.12) were not significant, indicating that ELs and non-ELs grew at a similar pace over the Year Two study.

Year two study: aimsweb tier status for all students

As seen in Fig. 1, ELs and non-ELs showed similar aimsweb tier status at the beginning of the Year Two study: 31% of ELs and 25% of non-ELs were in Tier 1, 45% of ELs and 43% of non-ELs were in Tier 2, and 24% of ELs and 32% of non-ELs were in Tier 3. The difference between ELs and non-ELs in tier status at the beginning of the Year Two study was not significant. There were, however, significant changes in tier status by the end of the Year Two study (χ 2 = 26.57, df = 4, p < 0.001). Nearly half of ELs and over half of non-ELs ended the Year Two study in Tier 1. There was no significant difference between ELs and non-ELs in terms of tier status at the end of the Year Two study. The percentages in Tier 3 at the end of the Year Two study (11% ELs, 7% non-ELs) were lower than the percentages in Tier 3 at the end of the Year One study (15% ELs, 19% non-ELs).

Year two study: aimsweb tier status for at-risk students

At the beginning of the Year Two study, 24% of ELs (n = 17) and 32% of non-ELs (n = 19) were in Tier 3, thus considered at-risk for reading failure. As seen in Fig. 1, at-risk students in both groups grew significantly over the Year Two study. More than 80% of at-risk ELs and non-ELs moved up at least one tier. There was no significant difference between ELs and non-ELs in tier status at the end of the Year Two study. The percentages of at-risk students in Tier 3 at the end of the Year Two study (18% ELs, 16% non-ELs) were lower than the percentages of at-risk students in Tier 3 at the end of the Year One study (33% ELs, 35% non-ELs).

Discussion

Through use of a highly structured, personalized blended learning approach implemented with fidelity (i.e., meeting recommended usage requirements), matched groups of ELs and non-ELs in kindergarten through fifth grade showed large improvements on aimsweb, a standardized test of reading skills. This was quite evident in the Year One study in which both groups across grades showed significant growth from beginning to end of the school year. In kindergarten and second through fifth grade, EL and non-ELs grew at a similar pace. In the case of first grade, ELs showed significantly greater gains than matched non-ELs. The finding that ELs benefitted to the same extent or, in one case, to a greater extent than non-ELs from the same instructional program contrasts with studies showing that ELs generally do not fare as well as non-ELs in terms of reading performance (e.g., Kieffer 2008; U.S. Department of Education 2015). These findings suggest that a blended learning approach to reading instruction, as implemented in this study, can lead to significant advances on a standardized reading test for ELs and non-ELs alike.

At-risk students

The blended learning approach was found to be particularly beneficial for at-risk students in both EL and non-EL groups. The at-risk students were all in Tier 3 of aimsweb at the beginning of the Year One study. By the end of the Year One study, there was a noticeable improvement in tier status for at-risk students, with only one-third of at-risk ELs and non-ELs remaining in Tier 3. Further, by the end of the Year Two study, less than 20% of at-risk ELs and non-ELs were still in Tier 3. There were no significances between at-risk ELs and non-ELs on aimsweb at the end of the Year One and Year Two studies. These findings point to equivalent benefits of the blended learning approach for at-risk ELs and non-ELs, with increased effectiveness when implemented over two years.

Benefits of blended learning

It is important to consider reasons why a blended learning approach may be effective for at-risk ELs and non-ELs. When faced with students who present with diverse profiles of skills, an intervention that is able to accurately and efficiently differentiate instruction is highly desired, especially if students need to work on skills to fill reading gaps well below their grade level. The digital, online component of the blended learning approach used in this study has the ability to automatically adapt based on student performance, meaning the program is able to provide individualized practice as well as scaffolded instructional steps when needed. Differentiation of instruction and scaffolding of material with the detail and consistency a digital tool can provide may be difficult for classroom teachers to do alone, particularly considering that many students struggle with skills that are originally taught at different grade levels than the students’ current grade. For example, expecting a fourth grade curriculum to cover first grade phonological awareness skills for a fourth grade student who performs at a first grade level is unrealistic. Therefore, differentiation of instruction made possible by the digital, online component of a blended learning approach may allow students who are behind to “catch up” to their peers. The ability of the digital component to report real-time data on a students’ strengths and weaknesses allows a teacher to know exactly where students excels and where they may need extra support.

A blended learning approach that provides scripted offline lessons tied to the digital content, as in this study, allows students who struggle in the online program to receive individual or small group instruction on specific skills that can be delivered by a trained staff member. For students who may have trouble generalizing what they have learned in the online component of the program, the provision of paper–pencil activities linked to skills in the program serves to reinforce and extend application of those skills. It is possible that the blended learning approach in this study was so beneficial for at-risk students because through the use of the online component, scripted lessons, and paper–pencil activities, teachers could more effectively administer personalized instruction than they could without the blended learning program.

In many instances, the types of instruction demonstrated to be most effective for at-risk students, in particular ELs, often require 1-to-1 tutorial sessions and/or must be conducted outside of the regular classroom environment (e.g., Begeny et al. 2012; Vadasy and Sanders 2011). A blended learning approach when properly implemented may provide a viable alternative to these types of instruction. A blended learning approach capitalizes on the ability of technology to deliver systematic practice and instruction that targets individual student’s skill gaps and can reduce the amount of time spent in tutorial sessions. Furthermore, the adaptive component of the online instruction in this study ensured students optimized their time by focusing on specific skills where they needed extra practice. Outcomes consistent with this study were recently reported by O’Callaghan et al. (2016) when Core5 was implemented outside of the United States as part of an intervention study with at-risk 4- to 6-year olds in Northern Ireland.

A further component of the blended learning approach in this study which may have been particularly valuable for ELs is its breadth of content. Vadasy and Sanders (2011) suggest that when a school’s reading program is already strong in phonics, at-risk ELs may potentially benefit from interventions that target reading comprehension through enhancing oral language skills (see August et al. 2014). The blended learning approach in this study helps build comprehension first through listening comprehension activities designed to enhance vocabulary and an understanding of story structure (e.g., main idea, sequencing) and then progresses to reading comprehension strategies in later grades. Especially important for ELs, the blended learning approach in this study leverages picture-based activities to strengthen vocabulary, including academic vocabulary (Francis et al. 2006) and figurative language such as idioms, similes, and metaphors. Through these embedded supports, ELs demonstrated gains in reading equivalent to gains made by a matched sample of non-ELs, resulting in no differences between groups (or a difference favoring ELs) at the end of the Year One and Year Two studies.

Limitations

A main limitation in this study is the absence of a non-intervention comparison group—for example, students in the same schools who did not receive instruction with the blended learning approach. Thus, determining the extent to which the blended learning approach contributed to reading gains in the intervention schools relative to traditional classroom instruction was not tested in this study. A second limitation is the uneven distribution of ELs and non-ELs across schools in this study. ELs were clustered to a greater extent in some schools and non-ELs in others, which is a common demographic pattern in the United States. For instance, one study noted almost 70% of elementary-age ELs attended 5000 schools, out of approximately 50,000 elementary schools nationwide (Fry 2007). Thus, it is possible that group comparisons are confounded to some degree by school effects. To mitigate against other group differences, ELs and non-ELs were matched in terms of beginning-of-year placement in Core5 and tier status on aimsweb.

Lastly, although implementation of the blended learning approach was systematically covered in the professional development trainings, the ability to monitor implementation was restricted in this study. Given that there were 64 different schools in the study, data on specific school implementations of the blended learning approach were limited to keeping track of use and progress in the online program. Researchers were able to gather minimal information about other elements of the blended learning approach, including the extent to which station rotation or lab rotation models were used, how systematically the scripted lessons and pencil–pencil activities were utilized, what other components of classroom instruction and interventions were implemented in addition to Core5, and the extent to which elements of the blended learning approach were accessed after school or at home. Some of these elements are being investigated in forthcoming studies on the implementation of blended learning approaches to reading instruction.

Conclusion

The population of ELs in the United States continues to increase, yet ELs historically perform below non-ELs on reading assessments (Kieffer 2008; U.S. DOE 2015). Results of this study point to a potentially effective way to curtail the trend. It was found that a blended learning approach—integrating teacher-led instruction with online, digital activities—contributed to significant gains in reading in a large sample of ELs. Gains for ELs were similar to (or greater than) those found in a matched sample of non-ELs, including equivalent gains for at-risk ELs and non-ELs. These promising results suggest that a blended learning approach can serve as a viable option for ELs, resulting in higher percentages scoring at a proficient reading level and closing the gap with their non-ELs peers. Of course, effective use of a blended learning approach with ELs rests on solid implementation, where teachers make use of online reports to track student usage and progress but also apply key elements of offline instruction to address the specific needs of students.

Notes

Acknowledgement

The authors would like to thank Jen Elise Prescott, PhD for her consultation on the analyses.

Compliance with ethical standards

Conflict of interest

This submission evaluates the effectiveness of a commercial product. Two authors of this paper are employed by Lexia® Learning, a Rosetta Stone® Company, and one serves as a paid consultant and is employed by Community College of Rhode Island. None of the researchers receive commission on sales of the products. Teachers and school personnel carried out the implementation of the blended learning program independently.

References

  1. August, D., McCardle, P., & Shanahan, T. (2014). Developing literacy in English language learners: Findings from a review of the experimental research. School Psychology Review, 43(4), 490–498.CrossRefGoogle Scholar
  2. August, D., & Shanahan, T. (2006). Executive summary: Developing literacy in second-language learners. In D. August & T. Shanahan (Eds.), Developing literacy in second-language learners: Report of the national literacy panel on language-minority children and youth. Mahwah, NJ: Lawrence Erlbaum Associates Publishers.Google Scholar
  3. Begeny, J. C., Ross, S. G., Greene, D. J., Mitchell, R. C., & Whitehouse, M. H. (2012). Effects of the helping early literacy with practice strategies (HELPS) reading fluency program with Latino English language learners: A preliminary evaluation. Journal of Behavioral Education, 21(2), 134–149.CrossRefGoogle Scholar
  4. Buckingham, J., Wheldall, K., & Beaman-Wheldall, R. (2013). Why poor children are more likely to become poor readers: The school years. Australian Journal of Education, 57(3), 190–213.CrossRefGoogle Scholar
  5. Campaign for Grade-Level Reading. (2014). 3rd grade reading success matters: The campaign. Retrieved from http://gradelevelreading.net/wp-content/uploads/2012/01/009-14-GLR_Brochure_2-27.pdfx.
  6. Cassady, J. C., & Smith, L. L. (2005). The impact of a structured integrated learning system on first-grade students’ reading gains. Reading & Writing Quarterly, 21(4), 361–376.CrossRefGoogle Scholar
  7. Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Wunder, J., De Rosario, H., & De Rosario, M. H. (2015). Package ‘pwr’. Retrieved from https://cran.r-project.org/web/packages/pwr/pwr.pdf.
  8. Cheung, A. C., & Slavin, R. E. (2012). How features of educational technology applications affect student reading outcomes: A meta-analysis. Educational Research Review, 7(3), 198–215.CrossRefGoogle Scholar
  9. Christensen, C. M., Horn, M. B., & Staker, H. (2013). Is K-12 Blended Learning Disruptive? An introduction of the theory of hybrids. San Mateo, CA: Clayton Christensen Institute.Google Scholar
  10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum Associates.Google Scholar
  11. Espinosa, L. M. (2008). Challenging common myths about young English language learners. (Policy Brief: Advancing PK-3). New York: Foundation for Child Development.Google Scholar
  12. Evans, M. (2012). A guide to personalized learning: Suggestions for the race to top-district competition. San Mateo, CA: Clayton Christensen Institute.Google Scholar
  13. Francis, D. J., Rivera, M., Lesaux, N. K., Kieffer, M. J, & Rivera, H. (2006). Practical guidelines for the education of English language learners: Research-based recommendations for the use of accommodations in large-scale assessments. Portsmouth, NH: Center on Instruction. Retrieved from http://www.centeroninstruction.org.
  14. Fry, R. (2007). How far behind in math and reading are English language learners? Washington, DC: Pew Hispanic Center. Retrieved from http://files.eric.ed.gov/fulltext/ED509863.pdf.
  15. Gersten, R., & Geva, E. (2003). Teaching reading to early language learners. Educational Leadership, 60(7), 44–49.Google Scholar
  16. Goldenberg, C. (2012). Research on English Learner instruction. In M. Calderón (Ed.), Breaking Through: Effective instruction & assessment for reaching English Learners (pp. 39–61). Bloomington, IN: Solution Tree Press.Google Scholar
  17. Goldenberg, C., Rueda, R. S., & August, D. (2006). Sociocultural influences on the literacy attainment of language-minority children and youth. In D. August & T. Shanahan (Eds.), Developing literacy in second language learners: Report of the national literacy panel on language-minority children and youth (pp. 269–318). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  18. Hecht, S. A., & Close, L. (2002). Emergent literacy skills and training time uniquely predict variability in responses to phonemic awareness training in disadvantaged kindergartners. Journal of Experimental Child Psychology, 82(2), 93–115.CrossRefGoogle Scholar
  19. Hilliard, A. T. (2015). Global blended learning practices for teaching and learning, leadership and professional development. Journal of International Education Research, 11, 179–187.Google Scholar
  20. Horn, M. B., & Staker, H. (2011). The rise of K–12 blended learning. Mountain View, CA: Innosight Institute.Google Scholar
  21. Kamps, D., Abbott, M., Greenwood, C., Arreaga-Mayer, C., Wills, H., Longstaff, J., et al. (2007). Use of evidence-based, small-group reading instruction for English language learners in elementary grades: Secondary-tier intervention. Learning Disability Quarterly, 30(3), 153–168.CrossRefGoogle Scholar
  22. Kieffer, M. J. (2008). Catching up or falling behind? Initial English proficiency, concentrated poverty, and the reading growth of language minority learners in the United States. Journal of Educational Psychology, 100, 851–868.CrossRefGoogle Scholar
  23. Kim, H. (2014). Effects of using mobile devices in blended learning for English reading comprehension. Multimedia-Assisted Language Learning, 17, 64–85.Google Scholar
  24. Lesaux, N. K., & Geva, E. (2006). Synthesis: Development of literacy in language-minority students. In D. August & T. Shanahan (Eds.), Developing literacy in second-language learners: Report of the National Literacy Panel on language-minority children and youth (pp. 53–74). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.Google Scholar
  25. Lesaux, N. K., Koda, K., Siegel, L. S., & Shanahan, T. (2006). Development of literacy of language minority learners. In D. August & T. Shanahan (Eds.), Developing literacy in second-language learners: Report of the National Literacy Panel on language-minority children and youth (pp. 75–122). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.Google Scholar
  26. Lexia Learning. (2014). Lexia reading Core5. Concord, MA: Lexia Learning. Retrieved from: http://lexialearning.com/product/core5.
  27. Lim, D. H., Morris, M. L., & Kupritz, V. W. (2014). Online vs. blended learning: Differences in instructional outcomes and learner satisfaction. Journal of Asynchronous Learning Networks, 11(2), 27–42.Google Scholar
  28. Macaruso, P., Hook, P. E., & McCabe, R. (2006). The efficacy of computer-based supplementary phonics programs for advancing reading skills in at-risk elementary students. Journal of Research in Reading, 29, 162–172.CrossRefGoogle Scholar
  29. Macaruso, P., & Rodman, A. (2011a). Benefits of computer-assisted instruction to support reading acquisition in English Language Learners. Bilingual Research Journal, 34, 301–315.CrossRefGoogle Scholar
  30. Macaruso, P., & Rodman, A. (2011b). Efficacy of computer-assisted instruction for the development of early literacy skills in young children. Reading Psychology, 32, 172–196.CrossRefGoogle Scholar
  31. Macaruso, P., & Walker, A. (2008). The efficacy of computer-assisted instruction for advancing literacy skills in kindergarten children. Reading Psychology, 29, 266–287.CrossRefGoogle Scholar
  32. Mitchell, M. J., & Fox, B. J. (2001). The effects of computer software for developing phonological awareness in low-progress readers. Literacy Research and Instruction, 40(4), 315–332.Google Scholar
  33. National Education Association. (2008). English language learners face unique challenges. Washington, DC: NEA Education Policy and Practice Department.Google Scholar
  34. NCS Pearson. (2012). aimsweb: Technical Manual. Bloomington, MN: NCS Pearson. Retrieved from http://www.aimsweb.com/wp-content/uploads/aimsweb-Technical-Manual.pdf.
  35. O’Callaghan, P., McIvor, A., McVeigh, C., & Rushe, T. (2016). A randomized controlled trial of an early-intervention, computer-based literacy program to boost phonological skills in 4-to-6-year-old children. British Journal of Educational Psychology, 86(4), 546–558.CrossRefGoogle Scholar
  36. Pearson Education. (2011). aimsweb default cut scores explained. Bloomington, MN: Pearson. Retrieved from: http://www.aimsweb.com/wpcontent/uploads/AIMSweb_Default_Cut_Score_Guide.pdf.
  37. Picciano, A. G., Seaman, J., Shea, P, & Swan, K. (2012). Examining the extent and nature of online learning in American K–12 education: The research initiatives of the Alfred P. Sloan foundation. Internet and Higher Education, 15, 127–135.CrossRefGoogle Scholar
  38. Powell, A., Watson, J., Staley, P., Patrick, S., Horn, M., Fetzer, L., Hibbard, L., Oglesby, J., & Verma, S. (2015). Blending Learning: The evolution of online and face-to-face education from 2008–2015. Vienna, VA: International Association for K–12 Online Learning. Retrieved from http://www.inacol.org/wp-content/uploads/2015/07/iNACOL_Blended-Learning-The-Evolution-of-Online-And-Face-to-Face-Education-from-2008-2015.pdf.
  39. Rodríguez, C. D., Filler, J., & Higgins, K. (2012). Using primary language support via computer to improve reading comprehension skills of first-grade English language learners. Computers in the Schools, 29(3), 253–267.CrossRefGoogle Scholar
  40. Saine, N. L., Lerkkanen, M. K., Ahonen, T., Tolvanen, A., & Lyytinen, H. (2011). Computer-assisted remedial reading intervention for school beginners at risk for reading disability. Child Development, 82(3), 1013–1028.CrossRefGoogle Scholar
  41. Salkind, N. J. (2010). The greenhouse–geisser correction. In H. Abdi (Ed.), Encyclopedia of research design. Thousand Oaks, CA: Sage.CrossRefGoogle Scholar
  42. Schechter, R., Kazakoff, E. R., Bundschuh, K., Prescott, J. E., & Macaruso, P. (2017). Exploring the impact of engaged teachers on implementation fidelity and reading skill gains in a blended learning reading program. Reading Psychology, 38(6), 553–579.CrossRefGoogle Scholar
  43. Schechter, R., Macaruso, P., Kazakoff, E. R., & Brooke, E. (2015). Exploration of a blended learning approach to reading instruction for low SES students in early elementary grades. Computers in the Schools, 32, 183–200.CrossRefGoogle Scholar
  44. Staker, H., & Horn, M. B. (2012, May). Classifying K-12 blended learning. Mountain View, CA: Innosight Institute. Retrieved from: http://files.eric.ed.gov/fulltext/ED535180.pdf.
  45. Torgesen, J. K., Rashotte, C. A., & Alexander, A. (2001). Principles of fluency instruction in reading: Relationships with established empirical outcomes. In M. Wolf (Ed.), Dyslexia, fluency, and the brain (pp. 333–355). Parkton, MD: York Press.Google Scholar
  46. Troia, G. A. (2004). Migrant students with limited english proficiency can fast ForWord Language™ make a difference in their language skills and academic achievement? Remedial and Special Education, 25(6), 353–366.CrossRefGoogle Scholar
  47. U.S. Department of Education. (2011). The Nation’s Report Card, Reading 2011: National Assessment of Educational Progress at Grades 4 and 8. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, US Department of Education. Retrieved from http://nces.ed.gov/nationsreportcard/pdf/main2011/2012457.pdf.
  48. U.S. Department of Education. (2015). National Center for Education Statistics, National Assessment of Educational Progress (NAEP). Washington, DC: US Department of Education. Retrieved from http://nces.ed.gov/nationsreportcard/.
  49. Vadasy, P. F., & Sanders, E. A. (2011). Efficacy of supplemental phonics-based instruction for low-skilled first graders: How language minority status and pretest characteristics moderate treatment response. Scientific Studies of Reading, 15(6), 471–497.CrossRefGoogle Scholar
  50. Vaughan, N. (2014). Student engagement and blended learning: Making the assessment connection. Education Sciences, 4, 247–264.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Lexia LearningConcordUSA
  2. 2.Community College of Rhode IslandWarwickUSA

Personalised recommendations