Skip to main content
SpringerLink
Log in

Implementing an adaptive intelligent tutoring system as an instructional supplement

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Evidence is emerging that technology-based curricula and adaptive learning systems can personalize students' learning experiences and facilitate development of mathematical skills. Yet, evidence of efficacy in rigorous studies for these blended instructional models is mixed. These studies highlight challenges implementing the systems in classrooms, which may contribute to a lack of consistently positive effects on student learning. This article extends the literature by closely examining implementation models and dosage levels for a supplemental software, two gaps in existing research. It also investigates adherence to the core components of the software, and extent to which the supplement enabled personalized instruction. The study was conducted in 40 algebra I classes in an urban school district. Sixty-two percent of classes implemented models that integrated instructional modalities. There was mixed adherence to core components of the software in classes that used it. In the vast majority of classes (94%), software did not enable personalized instruction. Software and the existing curricula were largely independent and did not inform each other. Only one class implemented an integrated instructional model, adhered to the core design components of the software, and demonstrated high levels of personalized instruction. Findings identify implementation barriers and offer suggestions for future implementations and studies of technology-enabled personalization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ahlgren, A., & Harper, M. (2009). Assessment and placement through Calculus I at University of Illinois. Notices of the AMS,58, 1460–1461.

    Google Scholar 

  • ALEKS Corporation. (2011). ALEKS at big bear middle school: Report on efficacy study results for the 2010–2011 school year. Retrieved March 20, 2016, from https://www.aleks.com/k12/success_stories.

  • Aladjem, D. K., & Borman, K. M. (2006). Examining comprehensive school reform. Washington, DC: The Urban Institute Press.

    Google Scholar 

  • Augustine, C. H., McCombs, J. S., Pane, J. F., Schwartz, H. L., Schweig, J., McEachin, A., et al. (2016). Learning from Summer. Santa Monica, CA: Rand Corporation.

    Google Scholar 

  • Bailey, J., Martin, N., Schneider, C., Vander Ark, T., Duty, L., Ellis, S., Owens, D., Rabbitt, B., & Terman, A. (2013). Blended Learning Implementation guide: Version 2.0. Digital Learning Now. retrieved from https://digitallearningnow.com/site/uploads/2013/10/BLIg-2.0-Final-Paper.pdf.

  • Baxter, R. J., & Thibodeau, J.C. (2011) Does the use of intelligent learning and assessment software enhance the acquisition of financial accounting knowledge? Issues in Accounting Education, 26(4), 647–656.

    Article  Google Scholar 

  • Bertram, R. M., Blase, K. A., & Fixsen, D. L. (2015). Improving programs and outcomes: Implementation frameworks and organization change. Research on Social Work Practice,25(4), 477–487. https://doi.org/10.1177/1049731514537687.

    Article  Google Scholar 

  • Boddy, D., & Macbeth, D. (2000). Prescriptions for managing change: A survey of their effects in projects to implement collaborative working between organizations. International Journal of Project Management,18(5), 297–306.

    Article  Google Scholar 

  • Bowyer, J., & Chambers, L. (2017). Evaluating blended learning: Bringing the elements together. Research Matters: A Cambridge Assessment publication,23, 17–26.

    Google Scholar 

  • Brenner, A. M., & Brill, J. M. (2016). Investigating practices in teacher education that promote and inhibit technology integration transfer in early career teachers. TechTrends, 60(2), 136–144. https://doi.org/10.1007/s11528-016-0025-8.

    Article  Google Scholar 

  • Bryk, A. S., & Schneider, B. (2003). Trust in schools: A core resource for school reform. Educational Leadership, 60(6), 40–45.

    Google Scholar 

  • Carpenter, J., & Hanna, R. (2006). Predicting student preparedness in calculus. In Proceedings of the 2006 annual conference of the American Society for Engineering Education.

  • Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science,2, 40. https://doi.org/10.1186/1748-5908-2-40.

    Article  Google Scholar 

  • Craig, S. D., Hu, X., Graesser, A. C., Bargagliotti, A. E., Sterbinsky, A., Cheney, K. R., et al. (2013). The impact of a technology-based mathematics after-school program using ALEKS on student's knowledge and behaviors. Computers and Education,68, 495–504. https://doi.org/10.1016/j.compedu.2013.06.010.

    Article  Google Scholar 

  • Craig, S. D., Xie, J., Huang, X., Graesser, C., & Hu, X. (2014, June). The impact of epistemological beliefs on student interactions with an intelligent tutoring system. In International conference on intelligent tutoring systems (pp. 660–661).Cham: Springer.

  • Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science,4, 1–15.

    Article  Google Scholar 

  • Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effectis out of control. Clinical Psychology Review,18, 23–45.

    Article  Google Scholar 

  • Datnow, A., & Stringfield, S. (2000). Working together for reliable school reform. Journal of Education for Students Placed at Risk, 5(1–2), 183–204. https://doi.org/10.1080/10824669.2000.9671386.

    Article  Google Scholar 

  • Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research,18, 237–256. https://doi.org/10.1093/her/18.2.237.

    Article  Google Scholar 

  • Ebersole, S., & Vordan, M. (2003). Adoption of computer based Instructional Methodologies: A case study. International Journal of e-Learning,2(2), 15–20.

    Google Scholar 

  • Ely, D. P. (1990). Conditions that facilitate the implementation of educational technology innovations. Journal of Research on Computing in Education,23(2), 298–306.

    Article  Google Scholar 

  • Ely, D. P. (1999). New perspectives on the implementation of educational technology innovations. East Lansing, MI: National Center for Research on Teacher Learning.

    Google Scholar 

  • Ensminger, D. C., Surry, D. W., Porter, B. E., & Wright, D. (2004). Factors contributing to the successful implementation of technology innovations. Journal of Educational Technology & Society,7(3), 61–72.

    Google Scholar 

  • Ferdig, R. E., & Kennedy, K. (2014). Handbook of research on K-12 online and blended learning. Pittsburgh, PA: ETC Press.

    Google Scholar 

  • Fixsen D. L., Naoom S. F., Blase K. A., Friedman R. M., & Wallace F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

  • Fixsen, D. L., Blase, K., Metz, A., & Van Dyke, M. (2013). Statewide implementation of evidence-based programs. Exceptional Children (Special Issue),79(2), 213–230.

    Article  Google Scholar 

  • Hew, K. F. & Brush, T. (2007). Integrting technology into K-12 teachign and learning: Current knowledge gaps and recommendations for future research. Educational Technology Research and Development, 55(3), 223–252.

    Article  Google Scholar 

  • Honig, M. I., & Hatch, T. C. (2004). Crafting coherence: How schools strategically manage multiple, external demands. Educational Researcher, 33(8), 16–30. https://doi.org/10.3102/0013189X033008016.

    Article  Google Scholar 

  • Hu, X., Craig, S. D., Bargagliotti, A. E., Graesser, A. C., Okwumabua, T., Anderson, C., et al. (2012). The effects of a traditional and technology-based after-school setting on 6th grade student’s mathematics skills. Journal of Computers in Mathematics and Science Teaching,31(1), 17–38.

    Google Scholar 

  • Hu, X., Xu, Y. J., Hall, C., Walker, K., & Okwumabua, T. (2013). A potential technological solution for reducing the achievement gap between White and Black students. In Knowledge spaces (pp. 79–91). Berlin: Springer.

  • Huang, X., Craig, S. D., Xie, J., Graesser, A., & Hu, X. (2016). Intelligent tutoring systems work as a math gap reducer in 6th grade after-school program. Learning and Individual Differences,47, 258–265. https://doi.org/10.1016/j.lindif.2016.01.012.

    Article  Google Scholar 

  • Joyce, B. R., & Showers, B. (2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.

    Google Scholar 

  • Karam, R., Pane, J. F., Griffin, B. A., Robyn, A., Phillips, A., & Daugherty, L. (2016). Examining the implementation of technology-based blended algebra I curriculum at scale. Education Technology, Research & Development,65(2), 399–425. https://doi.org/10.1007/s11423-016-9498-6.

    Article  Google Scholar 

  • Koedinger, K. R., Corbett, A. T., Ritter, S., & Shapiro, L. J. (2000). Carnegie learning's cognitive tutor: Summary research results. Pittsburgh, PA: Carnegie Learning.

    Google Scholar 

  • Malbert, V. A., Soni, A., & Venkataramanan, M. A. (2003). Enterprise resource planning: Managing the implementation process. European Journal of Operational Research,146(2), 302–314.

    Article  Google Scholar 

  • Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: Center for Technology. in Learning.

    Google Scholar 

  • Morgan, P., & Ritter, S. (2002). An experimental study of the effects of cognitive tutor algebra I on student knowledge and attitude. Pittsburgh, PA: Carnegie Learning Inc.

    Google Scholar 

  • Newmann, F. M., Smith, B., Allensworth, E., & Bryk, A. S. (2001). Instructional program coherence: What it is and why it should guide school improvement policy. Educational Evaluation and Policy Analysis, 23(4), 297–321. https://doi.org/10.3102/01623737023004297.

    Article  Google Scholar 

  • Pane, J. F., McCaffrey, D. F., Slaughter, M. E., Steele, J. L., & Ikemoto, G. S. (2010). An experiment to evaluate the efficacy of cognitive tutor geometry. Journal of Research on Educational Effectiveness,3(3), 254–281.

    Article  Google Scholar 

  • Pane, J. F., Griffin, B. A., McCaffrey, D. F., & Karam, R. (2014). Effectiveness of cognitive tutor algebra I at Scale. Educational Evaluation and Policy Analysis,36(2), 127–144. https://doi.org/10.3102/0162373713507480.

    Article  Google Scholar 

  • Pane, J. F., Steiner, E. D., Baird, M. D., Hamilton, L. S., & Pane, J. D. (2017). Informing progress: Insights on personalized learning implementation and effects. Santa Monica, CA: RAND Corporation.

    Book  Google Scholar 

  • Patrick, S., Kennedy, K., & Powell, A. (2013). Mean what you say: Defining and integrating personalized, blended, and competency education. International Association for K-12 Online Learning. Retrieved from https://www.inacol.org/resource/mean-whatyou-say-defining-and-integrating-personalized-blended-and-competency-education/

  • Ritter, S., Kulikowich, J., Lei, P., McGuire, C. L., & Morgan, P. (2007). What evidence matters? A randomized field trial of cognitive tutor algebra I. Supporting Learning Flow through Integrative Technologies,162, 13–20.

    Google Scholar 

  • Roehrig, G. H., & Kruse, R. A. (2005). The role of teachers’ beliefs and knowledge in the adoption of a reform‐based curriculum. School Science and Mathematics, 105(8), 412–422. https://doi.org/10.1111/j.1949-8594.2005.tb18061.x.

    Article  Google Scholar 

  • Rogers, E. M. (1995). Diffusion of Innovations (4th Ed.), New York: The Free Press.

    Google Scholar 

  • Rogers, P. L. (2000). Barriers to adopting emerging technologies in education. Journal of Computing Research,22(4), 455–472.

    Article  Google Scholar 

  • Schacter, J. (1999). The impact of education technology on student achievement: What the most current research has to say. Santa Monica, CA: Milken Exchange on Educational Technology, Milken Family Foundation.

    Google Scholar 

  • Schofield, J. (2004). A model of learned implementation. Public Administration,82(2), 283–308.

    Article  Google Scholar 

  • Scott, G. (2000). ‘Seeing the forest for the trees’: Towards a framework for effective change research in education. Innovations in Education and Teaching International,38(1), 86–92.

    Article  Google Scholar 

  • Slavin, R. E. (1987). Mastery learning reconsidered. Review of Educational Research,57(2), 175–213.

    Article  Google Scholar 

  • Snodgrass Rangel, V., Bell, E. R., Monroy, C., & Whitaker, J. R. (2015). Toward a new approach to the evaluation of a digital curriculum using learning analytics. Journal of Research on Technology in Education,47(2), 89–104.

    Article  Google Scholar 

  • Sullins, J., Meister, R., Craig, S. D., Wilson, W. M., Bargagliotti, A., & Hu, X. (2013). Is there a relationship between interacting with a mathematical intelligent tutoring system and students performance on standardized high-stake tests (pp. 69–78). Knowledge Spaces: Applications to education, Springer, Berlin Heidelberg.

    Google Scholar 

  • Surry, D. W., & Ely, D. P. (2002). Adoption, diffusion, implementation, and institutionalization of instructional design and technology. In R. A. Reiser & J. V. Dempsey (Eds.), Trends and issues in instructional design and technology (pp. 183–193). Upper Saddle River, NJ: Pearson Education Inc.

    Google Scholar 

  • Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes (M. Cole, V. John-Steiner, S. Scribner & E. Souberman., Eds., A. R. Luria, M. Lopez-Morillas & M. Cole [with J. V. Wertsch], Trans.) Cambridge, MA: Harvard University Press. (Original manuscripts [ca. 1930–1934])

  • Watson, J., Murin, A., Vashaw, L., Gemin, B., & Rapp, C. (2013). Keeping pace with K-12 online & blended learning: An annual review of policy and practice. 10 Year Anniversary Issue. Evergreen Education Group.

  • Wenglinsky, H. (1998). Does it compute? The relationship between educational technology and student achievement in mathematics. Princeton, NJ: Educational Testing Service Policy Information Center.

    Google Scholar 

Download references

Acknowledgement

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A1400221 to RAND Corporation. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

Author information

Authors and Affiliations

  1. RAND Corporation, Santa Monica, CA, USA

    Andrea Phillips & John F. Pane

  2. Research for Action, Philadelphia, PA, USA

    Rebecca Reumann-Moore & Oluwatosin Shenbanjo

Authors
  1. Andrea Phillips
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John F. Pane
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rebecca Reumann-Moore
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oluwatosin Shenbanjo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrea Phillips.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Phillips, A., Pane, J.F., Reumann-Moore, R. et al. Implementing an adaptive intelligent tutoring system as an instructional supplement. Education Tech Research Dev 68, 1409–1437 (2020). https://doi.org/10.1007/s11423-020-09745-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-020-09745-w

Keywords

Search

Navigation