Keywords

1 Introduction

Science education is changing. Increasing emphasis is being placed on students in science-as-practice [10] that is helping students understand science not only as a body of knowledge but also as a set of practices for developing and warranting knowledge claims about the natural world. One of the most important aspects of this process is reading scientific texts. The recent report A Framework for K-12 Science Education [10] cites “Obtaining, Evaluating, and Communicating Information” (pg 74) as one of the key science and engineering practices. This practice requires students not only to interpret text, but also to compare and evaluate texts in the context of explaining phenomena and solving problems. By twelfth grade, students are expected to be able to read and critique published scientific texts.

In this study, we present the analysis of student engagement with eBRAVO, an ITS designed to support students’ reading comprehension in strategies by leveraging the Construction-Integration (CI) model of reading comprehension [21, 22]. We use a learning analytics [29] approach to understand student activity within eBRAVO using a feature driven sequential analysis approach [27]. We explore the distribution of engagement variables to understand different patterns of use and how they map to in-system measures of student comprehension and unit-level measures of student reading comprehension abilities.

2 Background

2.1 Reading Comprehension and Science Education

Reading to learn [6] is a critical skill to be supported in late primary and secondary education. This represents a significant challenge, since at present many students do not reach basic proficiency in reading comprehension for their grade level, including one third of fourth grade students and one quarter of eighth grade students [9]. Additionally, while great national and international effort has gone into supporting students’ reading skills, a significant gap exists in supporting the reading of domain-specific text, especially in science. Reading scientific texts requires skills above and beyond general reading comprehension [13], in part due to the complex nature of the writing style and the importance of synthesizing content across multiple texts, figures, charts, and other sources of information [10]. To be successful in school, students need to be able to read content-area texts well enough to learn and remember new content, and they must be able to apply what they know in future reading and learning situations, as well as in novel contexts. When students begin upper elementary school, they are expected to read content-area texts (often in the form of a textbook); however, they are rarely provided with instruction on how to read informational and expository texts.

Compounding the problem, educators are often ill-equipped to address these domain-specific reading issues. Science teachers are not language arts experts. Their training and expertise focuses on scientific concepts for students to learn to independently examine scientific information. This is especially problematic when students read expository texts, not only because of the unfamiliar subject matter, but also due to the higher density of ideas, lack of coherence, and more complicated reference and structure employed in such texts [23]. There is a sudden jump in the complexity of content area readings in middle and high school, such that many of the popular textbooks are in fact as complex as college-level material in their syntax, vocabulary, and presentation as well as in content [16]. Moreover, there are important differences across disciplines in the way information is presented and represented and in their styles of reasoning, argumentation and inquiry that are best conveyed by teachers within the particular disciplines [14]. The problem that many readers share is not their inability to read the words on a page; instead, their most common problem is their inability to construct a coherent representation of the text meaning that is well integrated into their knowledge base. That is, they fail to deeply comprehend what they read [3].

These difficulties have an impact on science learning beyond the traditional reading activity. The landscape of science education is changing to incorporate disciplinary core ideas, cross-cutting concepts, and scientific practices as three different dimensions of learning [10]. The ability to read for understanding affects one’s ability to learn science ideas through the practices that rely heavily on comprehension of text, such as argumentation and modeling, as well as for theory development and dissemination, and even course communication. Thus, it is critical to address these issues for the future of science learning.

2.2 Computer Interventions in Reading Comprehension

To date, the What Works Clearinghouse has evaluated four computer-based literacy programs that had positive effects on comprehension at the middle- or high-school levels: AcceleratedReader [28], Fast ForWord [12], Read180 [20], ReadingPlus [8]. All four programs focus primarily on developing fluency and vocabulary in order to improve comprehension, and are designed to be embedded in language arts or special education classrooms. In contrast, eBRAVO targets deep levels of comprehension in students who are reasonably fluent readers in their science domain classroom. Decoding at a fifth- grade level or higher, these students lack the skills to deeply or even adequately comprehend informational texts.

Recent research supports the development of automated, computer-based interventions that teach students to utilize reading strategies to learn from challenging, complex texts [4, 17, 25, 26, 31]. A leading example developed by McNamara and colleagues called interactive Strategy Training for Active Reading and Thinking (iSTART) utilizes natural language processing algorithms to automatically evaluate and provide feedback on students’ written self-explanations of text (e.g., [25]). Two newer versions, iSTART extended practice and iSTART-ME, also allow teachers to upload texts [17, 18]. However, eBRAVO differs from other online and computer-based programs, such as iSTART, in that it offers a more comprehensive approach to comprehension instruction embedded in the progressive study of content course texts. eBRAVO addresses all levels of comprehension, targeting local and global coherence building, gistmaking and inferencing, as well as metacognitive and problem-solving strategies (e.g., self-explanation) that proficient readers employ to identify and remedy comprehension breakdowns. This comprehensive approach to instructing deep reading comprehension assists students in constructing a coherent textbase and a situation model with multiple connections to the personal knowledge base, thereby ensuring durable and accessible memory for the course content. Furthermore, The pedagogical components of eBRAVO are designed following the CI comprehension model [21, 22], a thoroughly researched theory that describes the cognitive processes that contribute to reading comprehension. eBRAVO focuses on instruction of literacy skills together with knowledge-building, using a gradated series of texts contextualized within a content area.

2.3 Intelligent Tutoring Systems and Video Engagement

Intelligent Tutoring Systems (ITS, [1, 15]) are a promising approach to supporting learning. These tools provide students with a cohesive experience in a digital learning environment, using their engagement with the problems or tasks in the system to develop an underlying learner model that represents their current understanding of the content as well as relevant situational factors such as engagement or affect [7, 11]. ITS then use this model of the learner’s mental state to determine the appropriate next action for developing deeper understanding. This approach has been shown to be as effective as one-on-one time with a tutor, providing students with individualized support that can outpace time on task in a larger classroom activity [30].

Videos are increasingly prevalent as a learning tool, but these are primarily studied at a postsecondary level for MOOC and Blended/Flipped Learning environments (see, e.g. [32]). That said, it is important in supporting reading comprehension as a skill in a computer & learner system; when you cannot rely on the learner’s deep comprehension of the written word, you have to rely on connections between written words, visual diagramming, and narration. This design consideration inspires the demonstration found in the example screen from the video lesson in Fig. 1.

Fig. 1.
figure 1

The video view of a linking ideas lesson.

Full size image

However, when working with these ITS systems at the secondary level, many different paradigms of motivation are important. Many researches have cited issues with motivation in the use of ITS (e.g., [2]), and different approaches to improving engagement have been explored, including leveraging aspects of gaming directly in the interaction (e.g., [24]).

3 Research Context

Our eBRAVO project builds on BRAVO [5], a connected ecology curriculum with reading comprehension lessons originally designed for deployment in language arts classrooms. BRAVO operationalized the CI model of reading comprehension discussed in the background section above by incorporating five central lessons on the increasingly complex layers of the model, flowing from issues of local cohesion (“Linking Ideas” and “Getting the Gist”), global cohesion (“Summary” and “Graphic Organizers”) and situation modeling (“Inference”). These lessons were interspersed among the curriculum chapters, taught by the teacher to the whole class at once.

The eBRAVO curriculum takes the linear process of BRAVO and creates an adaptive reading experience on their web-enabled devices diagramed in Fig. 2 to work through an improved twelve chapter ecology curriculum. This is designed to allow students to encounter lessons as needed, rather than as a class. To accomodate this personalized learning experience in an intelligent tutoring approach, we adapted the five class-level lessons from BRAVO into videos and a practice problem personalized for each chapter, creating 119 lessons (chapter one does not have an inference lesson, since there is no assumed prior experience to draw upon).

Fig. 2.
figure 2

A diagram of the flow of possible trajectories through chapter content in eBRAVO.

Full size image

As students work within eBRAVO in their browser, they first see a short chapter of 3 to 11 paragraphs of ecology reading content connected to what they have read so far in the curriculum. These readings incorporate in-text highlighting and clickable definitions for key vocabulary, along with embedded figures including images of important components or diagrams of certain systems. Once students have completed their readings, they proceed directly to an open-ended log asking them to record the big ideas found in the text. eBRAVO then probes their deep comprehension with multiple choice questions that target both issues brought up directly in the text as well as inferences and connections to greater ideas from previous texts. If the system determines a student should see a reading lesson based on their comprehension scores, it then gauges which type of reading support (linking ideas, gist, summary, graphic organizer, or inference) with which the student needs help and sends them to the appropriate lesson for that chapter and content, as seen in Fig. 1 above. These lessons incorporate two to five minute videos that outline the key comprehension strategy that is targeted and demonstrate how to apply this comprehension strategy to parts of the chapter that the student has just completed.

3.1 Data and Analysis

In this paper, we present results from 27 middle school students who provided family consent and used eBRAVO in a four-week summer intensive program. These students used eBRAVO in their 60-min class period four days a week over 12 class periods. We gave them the TORC paragraph reading activity [19] as a pre and post assessment activity in order to independently gauge reading comprehension ability. During this program, students were not assigned grades or other curriculum outcomes, so external motivation for performance was lower than usual for a typical classroom deployment.

We use a learning analytics [29] approach to understand student activity within eBRAVO. We focus our efforts on relating student engagement with the reading lessons as measured by their clickstream with the video content to the resulting outcomes on practice problems in the lesson as well as future questions in the curriculum. We use a feature driven sequential analysis approach [27], creating independent variables of video and lesson engagement. We explore the distribution of these engagement variables to understand different patterns of use.

4 Results

Overall, our student sample demonstrated a wide variety of lesson personalization within eBRAVO, seeing on average 5.44 (SD 3.59) lessons, with two students seeing 11 lessons (almost one lesson every chapter) and four students seeing no lessons.

Our primary division of students is based on patterns of engagement across videos, as seen in Fig. 3. Overall, we see very few students did not reach at least one lesson (No Lesson), a group of students maintained engagement with lessons (Engaged), and a small group of students simply did not engage (Unengaged). However, the largest group of students began the curriculum engaged, but did not maintain their video engagement by their final lesson (Dropoff). Though “Dropoff” students continued to struggle with the deep comprehension questions, they began to show attrition when addressing their issues with the lesson content.

Fig. 3.
figure 3

The distribution of engagement type across students.

Full size image

These distinctions demonstrate interesting correlations with a variety of student outcomes. The first outcome of note is the student experience in terms of the average number of lessons encountered within each group, as seen in Fig. 4. The three groups that encountered lessons all demonstrated patterns that trend towards significance on a Kruskal Wallis H Test (p = 0.099), with students who demonstrate patterns of dropoff and unengagement trending towards increasing numbers of lessons seen.

Fig. 4.
figure 4

The distribution of lessons encountered by engagement type.

Full size image

These groupings also demonstrate interesting implications for demonstrated learning outcomes. In a pairwise t-test analysis of students’ comprehension scores within eBRAVO, a significant difference appeared when comparing “No Lesson” students to the “Dropoff” group (p = 0.0405), with the No Lesson students doing significantly better on in-system assessments. However, these results did not carry over to the TORC test at significance (p = 0.15).

5 Discussion

We see significant differences among each group in our breakdown.

5.1 No Lessons

For our “No Lessons” group, they clearly worked to ensure that they succeeded at the comprehension checks at the end of each chapter. This group demonstrated a mastery of the content within the system, but did not demonstrate a significant difference in the reading outcomes section, which may be due to causes discussed in the limitations section below.

5.2 Engaged

While our “Engaged” group demonstrated a willingness to keep with the program, and an ability to eventually master the abilities needed to complete the comprehension checks, they also did not demonstrate an improvement.

5.3 Dropoff

Our “Dropoff” group showed continued troubles with the comprehension checks within the curriculum - they are consistently diverted to lessons, which could be the source of their frustration and the reasoning behind their transition from engaged to unengaged.

5.4 Unengaged

The “Unengaged” group acted as you might expect - they never watched lesson videos, and they subsequently also never appeared to succeed on comprehension checks which meant they continued to be diverted to lessons. This did not connect to significant differences in in-system or pre/post test measures, which may be attributable to sample size as discussed in the limitations section below.

6 Conclusions

First and foremost, these results have implications for our design, suggesting the need to revisit our approach of repeated presentation of concepts. We found that students quickly found themselves breaking into different patterns of engagement that were easily distinguishable within the clickstreams and formed a distinct set of outcomes.

Moreover, these results should be addressed in other ITS tools that rely on a repeated sampling approach to concept understanding, demonstrating the need for engagement with lesson content to be motivated further. Students in this setting without external motivation from grades or other factors often do not continue to engage with the same content just because it is presented in a new context – they quickly grow tired of the approach.

6.1 Limitations

One important note to consider when discussing this work is the limited scope of this deployment. The first element of scope to consider is the use of 27 students under one teacher, which does not allow us to generalize much beyond the scope of that program. Many of our usage patters also resolved to very small numbers, which impacted our ability to look for significant differences, particularly in course outcome measures among groups such as the “No Lessons” (4 students) and “Unengaged” (3 students) groups. It will require a larger deployment to hone in on differences between these groups and gain further insight. Furthermore, it is important to remember that this was a deployment over a three-week summer program with no formal assessment policies, which did not allow for the teacher to implement any level of external motivation to participate with true fidelity. Many participants may have seen this as an opportunity to “slack off” when faced with learning from video lessons.

7 Future Work

These results and conclusions have motivated some significant changes in the workflow of the eBRAVO system. Some of our changes have already been implemented - we have focused on upgrading our teacher-facing dashboard to better provide a view for how the students are progressing through the materials within eBRAVO and what measures of success they are showing, in order for the teacher to provide just-in-time intervention to a student that may be struggling or showing signs of disengagement. These results have also prompted discussions of more significant redesign of the eBRAVO platform, incorporating intrinsic motivation patterns within the system, such as automated feedback and potential gamification [24] aspects.