Keywords

1 Introduction

Holding all students to high standards is critical, but as reforms based on standardization have continued to experience little appreciable success, it is important that we ask ourselves if we are pursuing the right solutions. In the United States, stakeholders have argued that we must share a commitment to high quality, standardized learning expectations for all children. However, attempts to standardize education, or provide the same education (i.e., the same content, at the same time, for the same duration, etc.) to all students is likely to create challenges, no matter how high the expectations are. A key factor driving these challenges is that students vary in their needs for instruction, support, and enrichment (Pape 2018). And while some students move quickly through content and require enrichment to keep from growing bored, others need to be taught in multiple ways, require plenty of time to practice and ask questions, need more scaffolding, and more support in general to reach learning goals (George 2005). In other words, perhaps it is not standardization that should be pursued, but rather ways to address learner variability.

The United States is an incredibly diverse country, with a rich tapestry of individuals from all walks of life, cultures, and creeds. This diversity has always been key to our success, as diversity is known to lead to more creativity, innovation, persistence, and better problem-solving in general (Phillips 2014). Research indicates that the variability present in the U.S. student population is due largely to the naturally differentiated development of young children (NAEYC & NCTM 2002), as well as children’s background factors, including socio-economic status (SES), ethnicity, mother’s education level, gender, adverse experiences, level of social support, and physical health (e.g., Abedi et al. 2006; Entwisle and Alexander 1990; Siegler 2009; McKown and Weinstein 2003; Pfefferbaum et al. 2016; Kalil 2013; Kim and Cicchetti 2010). For example, children’s differences in motor skills and executive functioning development (attention, inhibition, working memory) impact how they process information, interact with digital media, and learn early math skills (e.g., Dulaney et al. 2015; Stipek and Valentino 2015; Andersson 2008; Yeniad et al. 2013; Blair and Razza 2007).

In other cases, differences in the home learning environments of children, such as the work habits of the family (e.g., routines, stability), academic guidance and support provided by parents, cognitive stimulation (e.g., cultivating curiosity, etc.), language development (e.g., frequency and quality of parent-child conversation), and the academic expectations parents have for their children, contribute to this diversity (Bloom 1984). As a result, there is extraordinary learner diversity and learner variability present in the student population of the United States, in the form of different cultural, socio-economic and linguistic backgrounds, learner prior knowledge, skills, and aptitudes (Rose et al. 2013; Rose 2016), as well as other differences related to learning difficulties and giftedness (Pape 2018). This learner variability presents both challenges and opportunities.

1.1 The Challenge

Unaddressed learner variability may be the biggest factor in students’ underachievement, from the onset of schooling in preschool, and throughout the K-12 years (Digital Promise 2016), as many children are not achieving the established minimum proficiency standards for mathematics and reading, as well as other content areas (de Brey et al. 2019; Schleicher 2019). This is of particular concern in the United States, which currently ranks 13th internationally in reading achievement, and 37th in mathematics achievement (Schleicher 2019). A closer examination of the performance of students from the United States reveals a more dire problem, with nearly two out of every three 4th grade students not proficient in their grade level standards in mathematics and reading – a number that grows to three out of four in mathematics by 12th grade (de Brey et al. 2019). Furthermore, when the data is disaggregated by socio-economic status, ethnicity, cultural background, home language and more, students from historically disadvantaged groups perform much worse than their more advantaged peers (de Brey et al. 2019). More concerning is the fact that the achievement of students has not significantly improved over the past two decades, despite numerous education reforms aimed at solving the problem (Keieleber 2019; Rebarber 2020).

The problem of underachievement often begins before children start formal schooling. Many children begin school with gaps in their learning foundation – gaps that only widen as children move on to successive grades (Duncan et al. 2007). These gaps are thought to form as the result of early experiences in the home. While some children may spend their early years in an enriching home learning environment filled with a wide array of literacy and numeracy experiences, other children may receive very little (if any) exposure prior to beginning kindergarten (Booth and Crouter 2008; Hart and Risley 1995). This creates a learning opportunity gap (Cameron 2018), as those children who have had early exposure to math and literacy enter school better prepared to learn, while those who have not benefitted from such exposure enter school unready to take advantage of the learning school has to offer (Betts et al. 2020).

Learning opportunity gaps mean that some children have more advantages, and other less, when it comes to learning. Not all children have equal opportunity to learn, because not all children are ready to learn the same content at the same time. To mediate the problem of learning readiness, teachers are asked to differentiate their instruction to meet each individual student’s needs, but the task is herculean. At the elementary school level alone, teachers may be working with thirty or more students, all with vastly different skills and prior knowledge; at the secondary level teachers may be responsible for a hundred or more students. In any given classroom, a teacher may be required to provide instruction at three or four different grade levels, sometimes more. Consider that the average third grade teacher may have precocious students reading at a 5th grade level or higher, even while other students are struggling to master basic sound-symbol correspondences typically learned in 1st grade. It is nearly impossible for teachers to identify all of students’ individual needs, must less address them—even for highly skilled teachers putting forth tremendous effort.

The goal of a one-size-fits all approach to education that standardizes the curriculum and learning expectations by grade level is to deliver instructional content that the typical or average student can learn within a school year. The problem is that there is, in fact, no such thing as an average student. In his book, “The End of Average,” Rose (2016) debunks more than a century of thinking devoted to design of products and processes for the average human being. Whether you’re designing a cockpit for the average pilot, or an educational system to serve millions of students, designing for the average is useless, or “worse than useless, in fact, because it creates the illusion of knowledge” (Rose 2016, p. 11). Rose points out that “individuals behave, learn, and develop in distinctive ways, showing patterns of variability that are not captured by models based on statistical averages” (Rose et al. 2013, p. 152). In other words, a system that standardizes approaches and processes to meet the needs of average students is likely to fail. Because no average student exists, processes designed to meet the needs of an average student end up serving, at best, the needs of the very few, or at worst, the needs of no one. It would seem that decades of data on the underachievement of students would support this notion.

1.2 The Opportunity

Unprecedented learner variability within the U.S. student population is a known challenge (Pape 2018), which may be why the one-size-fits-all approach of standardization has not proven effective in increasing student achievement. Learner variability is also not a new challenge. When considering the diverse learning needs of individual students, Bloom (1984) advocated for a mastery-based personalized learning approach that sought to ensure progression for all students regardless of learner variability. In a landmark study comparing three separate learning conditions—(1) one-to-one personalized Mastery Learning, (2) whole-group Mastery Learning, and (3) conventional classroom instruction. Unsurprisingly, students in the one-to-one Mastery Learning condition achieved at rates of two standard deviations above those in the conventional classroom (also known as the 2 sigma problem). However, Bloom’s experiment also showed that students in the whole-group Mastery Learning condition achieved one standard deviation above conventional classroom instruction, indicating that the Mastery Learning model could significantly improve student learning over conventional classroom instruction.

Mastery Learning works because it requires the accumulation of knowledge and the mastery of new skills before moving onto successive ones. To ensure that all students master content, learner variability must be accounted for. Mastery Leaning accounts for learning variability by personalizing the instruction through appropriately individualized scaffolds, feedback, and enrichments. The biggest challenge in Bloom’s model, however, is meeting the varied needs of so many students simultaneously. Recent advancements in technology, data science, and adaptive instructional systems (AISs) may provide the solution for learner variability at scale (Ma et al. 2014; Steenbergen-Hu and Cooper 2014; U.S. Department of Education, Institute of Education Sciences, What Works Clearinghouse 2009; VanLehn 2011; Kulik and Fletcher 2016). For example, the work of VanLehn (2011) has shown that when adaptive learning systems are designed to emulate human tutors, recognizing the needs of their tutees and pacing the presentation of new materials accordingly, AISs can come moderately close to Bloom’s level of success with one-to-one tutors.

However, learning variability factors that contribute to learning outcomes are not limited to the learning content alone. These factors also include each child’s cognitive development, social and emotional development, their family background and physical development (Digital Promise 2021; Booth and Crouter 2008; Hart and Risley 1995; Pape 2018). In attempting to develop solutions to the 2-sigma problem, Bloom proposed that targeted modifications to four objects of change: the learner, the instructional materials, teacher quality and methodology, the learner’s environment (at home, school, and socially). Modifications or enhancements to these four objects have the potential to increase student achievement (see Fig. 1). Yet, Bloom showed that making changes to one object was most likely insufficient to substantially increase student achievement, stating “two variables involving different objects of the change process may, in some instances, be additive, whereas two variables involving the same object of the change process are less likely to be additive” (Bloom 1984, p. 6). Meaning, working through only one of these areas, such as teacher quality or the quality of instructional materials is likely not enough to affect change—a more systematic multi-pronged approach is necessary.

The question is, how might we combine what is changeable with personalized Mastery Learning to produce additive learning impact? As technology and internet connectivity have increased in both homes and in schools, it is possible to expand the reach of AISs to more students than ever before. Our ability to use technology to connect home and school in new ways also provides additional opportunities, for learner variability is context specific (Rose 2016; Immordino-Yang 2016). Effective approaches addressing learner variability must take these differences into account, recognizing and leveraging opportunities inherent in the learner’s ecosystem of resources (Betts et al. 2020). We believe the answer can be found in a learning engineering approach toward building of a Personalized Mastery Learning Ecosystem.

Fig. 1.
figure 1

Bloom’s four objects of the change process (Bloom 1984)

Full size image

1.3 Understanding Learning Ecosystems

In the biological sciences, the term ecosystem refers to the complex relationships and interactions between living things and their environment, usually toward the goal of ensuring survival (Merriam Webster 2021). Biological ecosystems are interconnected, interdependent, and context specific. Learning ecosystems are similar in that they include the “dynamic human interactions between people and their environment, relationships, resources and occurring processes” (Väljataga et al. 2020, p. 48). In biological ecosystems, adaptation and responsiveness are key components; everything in the system is impacted by everything else in the system, and failure to adapt is often a recipe for failure. In learning ecosystems, the same is also true. To succeed, the components of a learning ecosystem must adapt to the needs of the learner to effectively foster learning.

Thinking of the learning process as an ecosystem is not a new idea, even if the use of the metaphor is. The interconnectedness and interdependency of learners with their environment has been described or alluded to in the work of many educational theorists, including Lev Vygotsky (1896–1934), Benjamin Bloom (1913–1999), Urie Bronfrenner (1917–2005), and others. Vygotsky (1986) wrote extensively about the importance of “more knowledgeable others” in the child’s learning environment, including parents and other family members, as well as teachers. As previously mentioned, Bloom (1984) described relationships between four objects of change (i.e., child, materials/curriculum, teacher, environment) which could be leveraged to increase the learner’s ability to learn. Bronfrenner’s (1986, 1992, 1999) created more expanded models of the learning ecosystem to describe the complexity of the relationships contained there. This ecological systems theory positioned the child at the center of a complex, ever-expanding layers of influence, from proximal to extremely distal (i.e., the family, parent-teacher, education policy, societal views on education, etc.). Building on Bronfrenner’s work, Neal and Neal (2013) argued that these systems of relationships were less like successive levels of influence radiating outward from the learner, and more like a complex network of influences impacting the learner and interacting with each other through the learner.

While these conceptualizations of the learning ecosystem are helpful, it remains elusive how we can practically support the necessary multileveled and interdependence among systems to foster individual development, and to do so at scale for many learners at the same time. Here we describe our attempt to leverage research, technology, and data to construct such a social-technological infrastructure that empowers individuals and enables their interactions within the learning ecosystem. We call it the Personalized Mastery Learning Ecosystem.

2 Personalized Mastery Learning Ecosystem (PMLE)

Recent work by Betts and colleagues (2020) described an Ambient and Pervasive Personalized Learning Ecosystem (APPLE) that leverages the “people, processes, data, and networked connections” in a learner’s environment to create “new capabilities, richer experiences, and unprecedented educational opportunities” (p. 23). APPLE described a future system where all “smart” things were connected such that data could be shared systematically across networked connections and with the humans in the system, for the benefit of the learner. For example, data on student performance in My Reading Academy, a program designed to help young learners master early literacy skills (Fabienke et al. 2021), could be shared with other programs such as Kindle or Audible to automatically generate “just-right” collections of books targeted and dynamically adapted based on real-time data about the child’s current reading or listening comprehension levels (Betts et al. 2020). Though APPLE describes a possible future for adaptive learning that may yet be years away in terms of development, there are aspects of APPLE that are within reach, even today.

For the past several years, the learning engineering team at EdTech developer Age of Learning, Inc., has been developing and iterating on an AIS called Personalized Mastery Learning System (PMLS; Dohring et al. 2019). AISs are computer-based system that guide learning experiences by tailoring instructions and recommendations based on the learners’ goals, needs, or preferences in the context of the learning domain (Sottilare and Brawner 2018). Similarly, the job of the PMLS is to assess in real-time what the learner already knows or has mastered, what the learner doesn’t yet know, what the learner is most ready to learn next, and deliver appropriate instruction and scaffolding at a granular skill level.

This aligns closely with Bloom’s (1984) Mastery Learning and the work of Vygotsky (1986), who described the learning process in terms of three areas of development: the zone of actual development (ZAD), the zone of proximal development (ZPD), and the zone of insurmountable difficulty (ZID) (see Fig. 2). Comparing the PMLS to Vygotsky’s model, the process of learning is characterized by what the learner already knows and is capable of doing independently (ZAD), what the learner doesn’t yet know and is incapable of doing or understanding (ZID) on their own, and the critical area in between that represents what the learner can do or understand with the help of a more knowledgeable other (ZPD).

Fig. 2.
figure 2

How the PMLS aligns with Vygotsky’s (1986) zones of development

Full size image

2.1 From PMLS to PMLE

It is important to differentiate between the PMLS and PMLE. The Personalized Mastery Learning System is a digital, adaptive instructional system that includes the instructional design, data collection, analytics, and information delivery mechanisms (i.e., through dashboards, etc.). The Personalized Mastery Learning Ecosystem is the PMLS plus all components that exist outside of PMLS, including all the people (e.g., the child, peers, teachers, parents, caregivers, etc.), and offline materials (e.g., worksheets, projects, teacher-led lessons, parent-child math talks, etc.). In other words, the PMLS is one component (albeit a critically large one) of the broader, more inclusive, PMLE.

Just as an ecosystem describes the complex interactions between all the living and non-living parts of an environment, the Personalized Mastery Learning Ecosystem places the learner at the center and describes the complex interactions among the learner, all people, processes, data, and networked connections in the learner’s environment (see Fig. 3). Through these interactions, the automated mechanisms of the system adapt to learners’ individual needs and moves them through an iterative cycle of instruction and application, promoting the mastery of new concepts and skills. The people, processes, data, and networked connections include, in particular, Bloom’s four objects of change: the child (learner), the instructional materials (through the evolving and adaptive PMLS), the teacher, and the environment (parents, caregivers, families, etc.). By connecting all in a systematic way, they are no longer objects of change to be acted upon, but rather agents of change that may be acted through.

Fig. 3.
figure 3

Personalized mastery learning ecosystem.

Full size image

2.2 The PMLE Places the Learner at the Center

In considering the diagram of the Personalized Learning Ecosystem (see Fig. 3), the Learner is positioned at the center, interacting directly with the PMLS for the purposes of ongoing assessment, dynamically adapting the learning Materials to the learner’s individual needs, and providing actionable insights to the other humans in the system (e.g., teachers, parents, caregivers, families, etc.). As learners interact with the PMLS, the system captures key event-stream data as a means of evaluating the learners’ needs in any given moment and dynamically adjusts their learning trajectories (Simon 1995). For example, during an interactive learning activity, the system evaluates everything the learner touches, determines the level of needed to scaffold the learner to success, and so on. In other words, the system evaluates where in the ZPD the learner is by assessing and providing only the scaffolds that are needed to ensure progress from instruction and practice to application. This approach moves the learner efficiently toward independence, or what Vygotsky (1986) called the zone of actual development. As learners’ competencies are assessed during learning activities, the system adjusts the learning path of future activities. The system accomplishes this by determining whether the learner should stay in the current activity for more practice on the present learning objective, move forward to a new activity with a successive learning objective, or revisit a previous activity designed to strengthen and review prior competencies. In this manner, the system creates and adapts individualized learning trajectories through the learning content for each learner.

2.3 The PMLE Describes the Complex Interactions Among the Learner, All People, Processes, Data, and Networked Connections in the learner’s Environment

As seen in Fig. 3, data collected from the PMLS are not only analyzed for the purposes of dynamically adjusting learning trajectories. Data are also analyzed to provide actionable insights and individualized activity recommendations to the Teacher. These recommendations range from small group instruction ideas, to printable individualized offline activities, to targeted whole group lessons, or tailored enrichment projects – all designed expressly for the purposes of teaching the learner in their ZPD. In this manner, the role of the PMLS in the broader PMLE is to act as a vigilant, automated, teaching assistant with its eye constantly evaluating the progress of learners, while providing both detail and evidence of that learning to teachers. This intelligent assistant pays attention to everything each student does, down to the last keystroke, and provides teachers with a comprehensive picture of where each and all students are with respect to their individual levels of understanding. The reporting features empower teachers with critical information that allows for more immediate, tailored, data-driven instruction—no matter how many students are in the classroom. The PMLS provides automatic formative and summative assessments, delivers customized adjustments and interventions, and immediate identification of students who may need special attention or intervention –all while freeing teachers’ time to remediate, challenge, and motivate students to learn more.

In the home Environment, parents, caregivers, and families too receive direct, actionable, communications from the PMLS. Recommendations, based on the learner’s ZPD, including such activities as parent-child math talks, “how-to-help” ideas, and hands-on projects. These activities are dynamically generated at the “just-right” level based on the learner’s performance in the system. Moreover, the system also delivers parent education activities in the form of informational articles, tips, and educational videos on topics timed to coincide with their child’s learning. For example, the system has the ability to detect learners who may be experiencing productive struggle while playing the learning games; in recognizing this, the system might then make the decision to suggest a video to parents about growth mindset (Boaler 2016; Dweck 2008). Educating the parent about key topics and concepts at critical moments in the child’s development not only builds awareness but allows for parents to more readily leverage these “teachable moments,” and to capitalize parents’ role in helping to ripen the child’s learning (Vygotsky 1986).

The PMLS is a powerful system that monitors the learners’ progress in real-time, adapting their needs in every moment. But it is not complete. The PMLS only knows what the learners’ actions reveal as they engage with the system. On its own, the PMLS does not know what the learner may be learning or accomplishing outside of the system. This is one of the reasons that a PMLE approach is more desirable and efficient. For example, if the learner has been away from the PMLS for a while (i.e., not engaging with the app or the games), the PMLS alone would have no way of knowing the progress the learner has made outside the system, say for example, during classroom instruction. By evolving toward an ecosystem and developing mechanisms to collect data on activities and experiences that occur outside of the core system, the PMLS is able to incorporate and leverage additional data to more readily adapt to the learner’s needs.

For the PMLE to be effective, data must flow back into the PMLE from the offline activities that occur between the learner and the other humans in the system. For example, parent and teacher engagement data are collected based interactions in the respective parent and teacher dashboards. Parents and teachers may also enter data into the system (e.g., from offline activities), which is then incorporated into each child’s learning analytics data profile. These complex interactions create “new capabilities, richer experiences, and unprecedented educational opportunities” (Betts et al. 2020, p. 23) by understanding the learners’ ZPD at any given moment and delivering content and activity suggestions that are at the learners’ “just-right” level.

The research literature has shown that parents lack confidence in supporting the early literacy and numeracy development of their children (Betts 2021; Sonnenschein et al. 2005), and that they look to the child’s teacher for guidance. However, the research also shows that early childhood teachers also often have limited expertise in developing these early competencies, especially when it comes to early mathematics (Clements and Sarama 2014; Early et al. 2010; Li 2020). Given this finding, parents are often not receiving the appropriate guidance for the ways in which they can best support the early learning of their children. As such, the PMLE works to empower those individuals who are most well positioned to directly impact the learning and growth of the child, by providing them with learning and growth opportunities of their own.

Building the knowledge and competencies of both teachers and parents is a critical aim of the PMLE, as doing so ensures that the adults in the child’s environment are able to provide the appropriate support at the moment it is needed. The data collected by the PMLE on student interactions with the system are used to drive these educational experiences for the adults. For example, the PMLS may conclude from the child’s data that the child is engaged in productive struggle in one or more activities – meaning that while the child might appear to be “stuck,” the adaptive algorithms of the PMLS recognize this particular kind of “stuckness” as productive (i.e., moving the child forward and building the child’s persistence). In such a moment, the system recognizes that the parent may benefit from receiving information on growth mindset (Dweck 2008), and recommendations for how best to encourage and support their child’s development of persistence. In response to this, the PMLE triggers the delivery of a short video to the parent that teaches them about productive struggle, persistence, the development of growth mindset, and provides actionable strategies for the parent to use to support their child in developing these positive learner characteristics. The PMLE then is able to digitally track whether or not the parent watched the video. Over many thousands of learners and parents, the PMLE uses data to determine whether there are relationships between parents who did or did not watch the video, and the impact on the child’s performance. Based on this data, new algorithms are developed to anticipate the type of impact this might have on the child and learning trajectories can be adapted based on this added information. Similar opportunities for just-in-time learning and professional development are provided to teachers as well.

In sum, the PMLE empowers the adults in the child’s environment to support the child through their ZPD, with the confidence that their efforts are the best match for what the child needs at that moment. This approach has the potential to not only address the unique needs of an individual learner, but for all learners at scale.

2.4 PMLE Requires Learning Engineering

Given many factors contributing to learner variability and variability in the agents of change, how do we build an effective PMLE that works for all learners?

There are decades of research on learning and instruction (e.g. Clark and Mayer 2003; Bransford et al. 2000; Bransford et al. 2005) and on distilling guidelines for practice and design (Bjork and Yan 2014; Pashler et al. 2007). While useful, when it comes to the design of specific instructional experiences, designers often deal with enormous complexities, trade-offs, and uncertainties associated with learning in real-world contexts. As a result, these guidelines alone prove to be insufficient (Koedinger et al. 2013). Particular complexities exist when designing for young children, where it is critical that learning experiences are appropriate to their developmental stages and cognitive growth (Gelman 2014; Fisher 2014). Existing literature does not yet provide comprehensive and detailed guidance on how to design learning experiences that address the needs of young children engaging with real learning in real contexts.

This issue is magnified with scale, not just with issues of learner variability, but also in the variability of time and space for learning opportunities, in the resulting rich data about learner engagement and performance, in the mass personalization (Schuwer and Kusters 2014) of learners and learner groups, and in the ways in which our pedagogy must adapt to these needs (Roll et al. 2018). All this must be accounted for as we think about how to combine technologies, pedagogies, research and analyses, and theories of learning and teaching to design effective learning interactions and experiences.

Growing efforts on learning engineering are beginning to shed light on processes that help define what works, why it works, and how to scale what works. “Learning engineering,” a concept originally introduced by Herbert Simon (1967), has been formalized recently as “a process and practice that applies the learning sciences using human-centered engineering design methodologies and data-informed decision making to support learners and their development” (ICICLE 2019). Learning engineering applies the learning sciences – informed by cognitive psychology, neuroscience, and education research (Wilcox et al. 2016) – and engineering principles to create and iteratively improve learning experiences for learners. It leverages human-centered design to guide design choices that promote robust student learning, but also emphasizes the use of data to inform iterative design, development and the improvement process. The Knowledge-Learning Instruction (KLI) Framework (Koedinger et al. 2010) and similar efforts such as ASSISTments as an open platform for research (Heffernan and Heffernan 2014) are excellent examples of learning engineering in practice. They bundle the platform, the instructor role, and the content, in which affordances match content and enable them to provide rich and relevant interactions. Like the PMLS, these focus on the student-facing instructional system. However, the creation of a PMLE must also incorporate the home environment and the school-home connection in a child’s learning.

The learning engineering approach for PMLE development must continue to leverage advances from different fields including learning sciences, design research, curriculum research, game design, data sciences, and computer science. It requires deep integration of research and practices across these different fields in the implementation of research-based and data-informed design cycles, all while being quick and lean enough to be sustainable in a resource-limited industry production environment. This often calls for agile development methodologies (Rubin 2012) to allow teams to nimbly iterate to explore concepts, test prototypes, and validate design decisions. The result is a social-technical infrastructure to support iterative learning engineering for scaling learning sciences through design research, deep content analytics, and iterative product improvements (more on this in Sect. 3.2). In this next section, we describe how Age of Learning applies this learning engineering approach toward a PMLE called My Math Academy.

3 My Math Academy PMLE: A Learning Engineering Approach

3.1 My Math Academy and Bloom’s Four Objects of Change

My Math Academy targets three complimentary avenues for child learning: self-directed learning supported by adaptive algorithms using child performance data, parent-supported learning, and teacher-supported learning. All three avenues work together, leading to increases in children’s math skills and knowledge, as well as their motivation, confidence, and persistence in learning math.

At the time of writing, we have a fully functional version of the child-facing My Math Academy games with over 2 million users. The child-facing app features 98 games consisting of 300+ activities, covering number sense and operations concepts and skills for pre-kindergarten through second grade. The parent-facing and teacher-facing dashboards and resources are publicly available, with improvements currently in progress. The PMLE of My Math Academy is actualized through a framework that accounts for Bloom’s four objects of change: the child (learner) and the learning materials, the teachers, and the parents or caregivers in the home environment.

The Child and the Learning Materials.

The PMLE places the child (learner) at the center of the system. It is through the child’s interactions with the digital learning materials that data is collected, analyzed, and used to determine the child’s learning needs at any given moment. The learning materials for My Math Academy are delivered primarily through a digital child-facing app, as well as targeted offline activities that designed to extend the child’s learning from the app to the real world. Both the digital and offline materials contained in My Math Academy cover number sense and operations concepts and skills for pre-kindergarten through second grade. Specific skills covered range from counting to 10, to adding and subtracting three-digit numbers using the standard algorithm, skills that are foundational for later mathematical skill development. These activities were developed based on extensive research into early numeracy development, intervention programs, and state, national, and international standards frameworks, and are aligned with Bloom’s Mastery Learning theory. This research helped us define granular, measurable learning objectives toward number sense development and build an extensive knowledge map representing the precursor, successor, and parallel relationships between those objectives.

In each game, learners progress through a narrative world, playing and interacting with “Shapeys,” which serve as both characters and manipulatives in the game (see Fig. 4). Consistent with game-based assessment practices (e.g., Shute and Kim 2014), every game in My Math Academy is associated with a clear learning objective, learning tasks, and evidence of learning. Moreover, each learning objective is supported by an interactive instruction level that introduces skills, along with several layers of scaffolding and learning-specific feedback. Based on each learner’s performance, the adaptive system decides what games to recommend and at which level of difficulty, using a predetermined network map of learning objectives and their prerequisite relationships (i.e., a knowledge map, where each node is a discrete learning objective). For each individual game, adaptivity functions provide scaffolding within each skill level, connect games to adjust to difficulty needs, and guide learners through a customized pathway between performance-based skills. Game-based pretests and final assessment tasks serve as embedded assessments that check for understanding at a granular skill level.

Fig. 4.
figure 4

A snapshot of four different games within the My Math Academy system

Full size image

The Teacher.

The PMLE empowers the teacher to be an agent of change in the child’s learning, by providing key information, actionable insights, and recommendations for personalizing the learning for each student in the system. These data and recommendations are delivered to the teach via a teacher dashboard that provides real-time data about children’s usage and progress within the app. It provides an overview of the entire class which can be filtered into teacher-created groups. It also contains activity recommendations for each child according to their level of mastery.

Figure 5 provides a sample teacher dashboard, in which children’s’ progress is color-coded for each granular learning objective. Blue denotes regular or quick progression (i.e., ready to learn). Yellow indicates a child is engaged in productive struggle, a relatively slower but continued progress that can benefit from review or teacher reinforcement (i.e., need for review or reinforcement; Hiebert and Grouws 2007). Red indicates a child who is stuck, having made no progress after multiple attempts at the same activity level; such a child requires teacher support (i.e., intervention). Finally, gray indicates that the child has not yet reached that learning objective. In this example, the “Grab and Count 11–15” activity is suggested for two specific children who are stuck and/or have not reached the Count 11–15 learning objective. In sum, the dashboard provides teachers with objective data about children’s current proficiency and learning trajectories and aims to help them better tailor classroom instruction to accelerate learning (Gersten et al. 2009).

Fig. 5.
figure 5

Sample views of the teacher dashboard.

Full size image

We have taken a phased approach to the development of the My Math Academy PMLE. Previous efforts have focused on developing features of the system that deliver information, insights, and actionable recommendations to the teacher. In other words, the information has flowed in one direction, outward from the PMLS to the teach via the teacher dashboard. Present efforts focus on developing ways for information to flow from the external environment (i.e., the broader PMLE) back into the PMLS. A range of possibilities is being explored, from more indirect methods of evaluating clickstream data collected through teachers’ interactions directly with the digital dashboard (e.g., what tools, features, downloads, the teacher clicks on), to more direct methods of data collection such as data entered into the system by the teacher. Examples include the teacher entering the child’s score in an offline activity, or indicating that specific students participated in teacher-led intervention lessons, etc. In this manner, information about each child can flow both in and out of the system, allowing for a fuller examination of the child’s learning activities related to their progress through the digital learning materials.

Parents and/or Caregivers in the Home Environment.

Parents and caregivers are often a child’s first teacher, and as such Bloom considered them an essential object of change impacting a child’s learning. While the child is the central user of My Math Academy, we consider the impact of parents (including caregivers), educators, and instructional materials as important variables in an effective system for learning (Bloom 1984). Children tend to do better in school and enjoy learning more when schools, families, and community groups work together to support learning (e.g., Henderson and Mapp 2002). As product developers, we have a responsibility to educators, parents, and families to model appropriate, effective uses of technology, social media, and methods of communication that are safe, healthy, acceptable, responsible, and ethical (Fred Rogers Center 2012). Moreover, well-designed technology can be used effectively for learning and for strengthening parent-child interactions and home-school connections. Effective technology tools connect on- and off-screen activities with an emphasis on co-participation between adults and children and children and their peers (Stevens and Penuel 2010; Takeuchi 2011; Takeuchi and Stevens 2011).

Fig. 6.
figure 6

Sample views of the parent dashboard. From left to right: (a) child usage overview, (b) math talk prompts & instructions, (c) offline enrichment activity, (d) parent education video.

Full size image

In line with this thinking, the PMLE provides information and recommendations directly to parents and families through the parent dashboard as means of encouraging and enhancing learning interactions between the child and the important adults in the home. The parent dashboard (Fig. 6) offers ideas for activities that families can engage in to provide additional learning opportunities for the child, based on his or her progress in the app. This is essential because the home learning environment and parental engagement are critical for children’s development of early math skills (Epstein and Sanders 2002; Fantuzzo et al. 2004). Home-based family engagement practices also encourage family members to communicate high expectations for their child’s learning, which is important for academic success (Thompson et al. 2014).

3.2 Applying Learning Engineering to My Math Academy PMLE

The learning engineering team at Age of Learning is interdisciplinary, consisting of curriculum experts, learning scientists, data scientists, design researchers, efficacy researchers, and professional game developers. Together, this team produced the game-based learning solution called My Math Academy. It was built upon rigorous academic curriculum, developed with an emphasis on engagement, and grounded in theoretical foundations of learning sciences. The team also paid special attention to data, ensuring quality data for later applications of quantitative methods to inform ongoing improvements. True to the definition of learning engineering (ICICLE 2019) - “a process and practice that applies the learning sciences using human-centered engineering design methodologies and data-informed decision making to support learners and their development” - key to our learning engineering approach (Goodell and Thai 2020) are:

Learning Sciences Research.

Applications of learning sciences research informs the initial design of My Math Academy, including applications of mastery-based learning (Bloom 1971; Guskey 1997), mathematics learning trajectories (Clements and Sarama 2014; Simon 1995), game-based learning and engagement (Barab et al. 2005; Bransford et al. 2000; Gee 2003; Shute 2008; Rupp et al. 2010), design strategies for long-term learning and transfer (Bjork 1994; Roediger and Karpicke 2006; Taylor and Rohrer 2010; Bransford and Schwartz 1999; Kellman and Massey 2013; Anderson et al. 1996), and game-based assessment and structured data for evidence (Mislevy et al. 2003; Owen et al. 2012; Shute and Kim 2014).

Human-Centered Design Methodologies.

Human-centered design starts with understanding the challenge from the learners’ perspective (IDEO 2015). Goodell and Thai (2020) proposed an AIS model that considers the learner as a key component at the heart of a distributed learning (eco)system in which the learner, along with other adults and peers, interact with technology components in varying environmental conditions. Such an AIS model requires the learning engineering team to be grounded in empathy (IDEO 2015), beginning with the needs and perspectives of the people we are designing for. This includes who they are, what they need to learn, how they learn, when and where they learn, why they want to learn, etc. In understanding how and why people behave the way they do, we can design for meaningful interactions and uncover opportunities for new innovation.

To do so, the My Math Academy team regularly recruit learners, parents, and teachers to playtest early production prototypes. This process is critical in understanding how children make sense of and solve problems through our proposed playful interactions, and how teachers and parents can be best supported in helping their children learn and grow. Data from such design testing sessions drive concrete interactions, user interface, and user experience design for each learning interactions, that are sensitive to children’s developmental stages and parents’ and teachers’ needs and perspectives.

Data-Informed Decision Making.

Beyond data from design testing sessions, My Math Academy was designed with a game-based learning data framework for event-stream data collection (Owen and Hughes 2019). This captures event-stream interactions from the child (e.g., keystrokes, clicks, taps, drags) within the context of learning mechanics and game progress. As players move through the system, My Math Academy games react to player performance on core game mechanics (i.e., basic actions that players perform), translating main game interactions into learning performance data. This this approach, we can generate quality in-game learning evidence because we took into account early in the design process what learning goals are to be assessed, how they will be assessed through game interaction design, and what evidence these designed interactions will provide. Such data captures a context-rich data stream of player interactions while enabling learning analytics and educational data mining investigations into emergent patterns in learner behaviors and performance (Baker and Yacef 2009; Siemens 2010; Romero and Ventura 2010; Ferguson 2012). Such data can also be interpreted in combination with other interactions and features, such as event-stream interactions and manual inputs from offline activities from teachers and parents.

All of this wide variety of collected data – and copious amounts of it, collected from the increasing number of learners and all agents of change engaging in the ecosystem – allow us to better calibrate and adapt our AIS and to develop new and better adaptive technology. Fundamentally this not only changes how content is delivered, but also change how learning materials are created and improved over time.

Figure 7 illustrates Age of Learning’s learning engineering framework. Learning sciences research informed our initial design for learning and engagement. The My Math Academy learning engineering team iteratively released new content, which meant that curriculum research and design, game design, design research, production, learning analytics, and efficacy research were often taking place simultaneously. With the initial design, prototypes were built and tested with learners (children, parents, teachers), and data were collected (via design research sessions with prototypes, or via interaction logs and efficacy studies from live games) to draw insights into how well the games were engaging players in learning. Over time, and across approaches to research design and analysis, findings were layered and triangulated for deeper insights to inform further improvements and for contribution to a corpus of institutional knowledge as well as the knowledge base at large.

Fig. 7.
figure 7

Age of learning, Inc.’s learning engineering framework

Full size image

Agile Development Process.

For this learning engineering approach to work in an industry setting, we have found that the agile development process acts as a practical manifestation of this learning engineering framework. At Age of Learning, we formalized the learning engineering tenets into tools and processes embedded throughout the Scrum agile development process (Rubin 2012). Agile methodologies allow us to nimbly and quickly iterate to explore concepts, test prototypes, and validate design decisions.

By collaborating closely throughout this process, the learning engineering team strengthened our understanding of how learning works within the learning ecosystem and used those insights to improve the design of effective learning experiences. In effect, the learning engineering team members are learners too, acting as a fifth “agent” of the change process toward building effective education at scale.

4 Conclusion

Given the enormity of the student underachievement problem, the need for solutions that account for and address learner variability has never been more critical. As the population of the United States grows more diverse, resulting in even more learner variability, we can no longer rely on traditional methods of educating our children. And while efforts at personalization are encouraging and ongoing in many arenas, the ability to truly personalize for learners at scale has yet to be achieved. However, evolving technologies, processes, and approaches provide new opportunities and potential solutions. The evolution of the PMLE is just one such potential solution that leverages Bloom’s four agents of change as part of a broader ecosystem of learner support.

The PMLE is an example of how AISs can move beyond working solely with the child to create more fully formed ecosystems that account for all of the “agents of change” that influence a child’s learning, including the learners themselves, parents and caregivers, teachers, and the learning engineering teams. As all of the “learners” in the ecosystem increase their knowledge and understanding, and as the AIS ecosystem captures the outcomes of this learning, we are better able to optimize personalization for and ensure the long-term development and success of all children at scale.