Skip to main content

Analysis of patterns in time for evaluating effectiveness of first principles of instruction

Educational technology research and development volume 70, pages 1–29 (2022)Cite this article

This article has been updated

Abstract

In this naturalistic design-research study, we tracked 172,417 learning journeys of students who were interacting with an online resource, the Indiana University Plagiarism Tutorials and Tests (IPTAT) at https://plagiarism.iu.edu. IPTAT was designed using First Principles of Instruction (FPI; Merrill in Educ Technol Res Dev 50:43–59, 2002, https://doi.org/10.1007/BF02505024; First principles of instruction: identifying and designing effective, efficient, and engaging instruction, Wiley, New York, 2013; M. David Merrill’s First Principles of Instruction, Association for Educational Communications and Technology, Washington, 2020). Students who used IPTAT were mostly from university and advanced high school courses taught in 186 countries and territories. Instructors expected their students to pass one of trillions of difficult Certification Tests (CT) provided through IPTAT. Each CT assessed student ability to classify samples of writing as word-for-word plagiarism, paraphrasing plagiarism, or no plagiarism—when given original source materials. In 51,646 successful learning journeys, students who were initially nonmasters and who later achieved mastery had viewed, on average, 89 IPTAT tutorial webpages designed with FPI. In the 23,307 unsuccessful learning journeys, students who were nonmasters and who had not (yet) achieved mastery had viewed an average of 52 tutorial webpages designed with FPI. Analysis of Patterns in Time (Frick in American Educational Research Journal 27:180–204, 1990) and Bayesian analysis revealed that students were nearly 4 times more likely to pass a CT when they selected one or more parts of IPTAT instruction designed with FPI. These results support the instrumental value of First Principles of Instruction for design of online learning in a massive, open, online course (MOOC). These findings further demonstrate the value of an innovative approach to modern learning analytics, Analysis of Patterns in Time, when coupled with Bayesian reasoning.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Change history

  • 21 January 2022

    The journal name presented at the head of the title page of the PDF version of this article has been corrected from a previous version.

References

  • Bertolero, M., & Bassett, D. (2019). How the mind emerges from the brain’s complex networks. Scientific American, 321(1), 26–33. Original title: How matter becomes mind, https://doi.org/10.1038/scientificamerican0719-26. https://www.scientificamerican.com/article/how-the-mind-emerges-from-the-brains-complex-networks/

  • DBRC: Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32, 5–8.

    Article  Google Scholar 

  • Eagleman, D. (2020). Livewired: The inside story of the ever-changing brain. Pantheon Books.

  • Educology Website (n.d.). Knowledge of education. https://educology.iu.edu

  • Frick, T. W. (1983). Nonmetric temporal path analysis: An alternative to the linear models approach for verification of stochastic educational relations [Unpublished doctoral dissertation]. Indiana University Graduate School.

  • Frick, T. W., & Reigeluth, C. M. (1992). Verifying prescriptive instructional theory by Analysis of Patterns in Time [Paper presentation]. The annual meeting of the American Educational Research Association (AERA) San Francisco, CA, United States.

  • Frick, T., Myers, R., Thompson, K., & York, S. (2008). New ways to measure systemic change: Map & Analyze Patterns & Structures Across Time (MAPSAT). Featured research paper presented at the annual conference of the Association for Educational Communications & Technology, Orlando, FL, United States. https://www.indiana.edu/~tedfrick/MAPSATAECTOrlando2008.pdf

  • Frick, T. W., Dagli, C., Kwon, K., & Tomita, K. (2018). Indiana University plagiarism tutorials and tests: 14 years of worldwide learning online. In B. Hokanson, B. Clinton, & K. Kaminski (Eds.), Educational technology and narrative: Story and instructional design, (Chapter 16, pp. 191–205). Springer.

  • Frick, T. W. (2021). Educology is interdisciplinary: What is it? Why do we need it? Why should we care? In B. Hokanson, M. Exter, A. Grincewicz, M. Schmidt, & A. Tawfik (Eds.), Intersections across disciplines: Interdisciplinarity and learning, (Chapter 3, pp. 27–42). Springer. https://doi.org/10.1007/978-3-030-53875-0_3

  • Frick, T. W., Myers, R.D., & Dagli, C. (2021). Analysis of patterns in time: Innovative learning analytics with GA4. Featured Research Paper Presented at the Annual Conference of the Association for Educational Communications and Technology, Chicago, IL.

  • Frick, T. W., Myers, R. D., Dagli, C., & Barrett, A. F. (2022). Innovative learning analytics for evaluating instruction: A big data roadmap to effective online learning. Routledge.

  • Frick, T. W. (1990). Analysis of patterns in time (APT): A method of recording and quantifying temporal relations in education. American Educational Research Journal, 27(1), 180–204.

    Article  Google Scholar 

  • Frick, T. W., & Dagli, C. (2016). MOOCs for research: The case of the Indiana University Plagiarism Tutorials and Tests. Technology, Knowledge and Learning, 21, 255–276. https://doi.org/10.1007/s10758-016-9288-6

    Article  Google Scholar 

  • Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x

    Article  Google Scholar 

  • Google Analytics (2020).  Set up Analytics for a website (Universal Analytics).  https://support.google.com/analytics/answer/10269537?hl=en

  • Google Analytics 4 (2020). Meet the next generation of Google Analytics. https://support.google.com/analytics/answer/10089681?hl=en

  • Greenwood, D. J., & Levin, M. (1998). Introduction to action research: Social research for social change. SAGE.

  • Ifenthaler, D. (2017). Learning analytics design. In L. Lin & M. Spector (Eds.), The sciences of learning and instructional design: Constructive articulation between communities (pp. 202–211). Routledge.

    Chapter  Google Scholar 

  • Kant, I. (1785). Groundwork for the metaphysics of morals (translated by J. Bennett, 2017). http://www.earlymoderntexts.com/assets/pdfs/kant1785.pdf

  • Kirk, R. E. (1999). Statistics: An introduction (4th ed.). Harcourt Brace.

  • Kirk, R. E. (2013). Experimental design: Procedures for the behavioral sciences (4th ed.). SAGE.

  • Klein, C., & Hess, R. M. (2019). Using learning analytics to improve student learning outcomes assessment: Benefits, constraints, & possibilities. In J. Lester, C. Klein, A. Johri, & H. Rangwala (Eds.), Learning analytics in higher education: Current innovations, future potential, and practical applications (pp. 140–159). Routledge.

    Google Scholar 

  • Mangaroska, K., & Giannakos, M. (2019). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://doi.org/10.1109/TLT.2018.2868673

    Article  Google Scholar 

  • Merrill, M. D. (2002). First principles of instruction. Educational Technology Research & Development, 50(3), 43–59. https://doi.org/10.1007/BF02505024

    Article  Google Scholar 

  • Merrill, M. D. (2013). First principles of instruction: Identifying and designing effective, efficient, and engaging instruction. Pfeiffer.

    Google Scholar 

  • Merrill, M. D. (2020). M. David Merrill’s First Principles of Instruction. Association for Educational Communications and Technology.

  • Myers, R. D., & Frick, T. W. (2015). Using pattern matching to assess gameplay. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement, (Chapter 19, pp. 435–458). Springer. https://doi.org/10.1007/978-3-319-05834-4_19

  • Phillips, T., & Ozogul, G. (2020). Learning analytics research in relation to educational technology: Capturing learning analytics contributions with bibliometric analysis. TechTrends, 64(6), 878–886. https://doi.org/10.1007/s11528-020-00519-y

    Article  Google Scholar 

  • Reigeluth, C. M., & Frick, T. W. (1999). Formative research: a methodology for improving design theories. In C. M. Reigeluth (Ed.), Instructional-Design Theories and Models: A New Paradigm of Instructional Theories (Vol. II, pp. 633–651). Lawrence-Erlbaum.

  • Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1355. https://doi.org/10.1002/widm.1355

    Article  Google Scholar 

  • Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagné, & M. Scriven (Eds.). Perspectives on curriculum evaluation (AERA monograph series on curriculum evaluation no. 1, pp. 39–83). Rand McNally.

  • Smialek, J. (2021). The Nobel in economics goes to three who find experiments in real life. The New York Times. https://www.nytimes.com/2021/10/11/business/nobel-economics-prize-david-card-joshua-angrist-guido-imbens.html

  • Steiner, E. (1981). Educology of the free. Philosophical Library.

  • Steiner, E. (1988). Methodology of theory building. Educology Research Associates.

  • Steiner, E. (2009). Ethical theory (Lecture notes). https://tedfrick.sitehost.iu.edu/steiner/Ethical%20Theory.pdf

  • Tabachnick, B. G., & Fidell, L. S. (2018). Using multivariate statistics (7th ed.). Pearson.

  • Wong, J., Baars, M., de Koning, B. B., van der Zee, T., Davis, D., Khalil, M., & Paas, F. (2019). Educational theories and learning analytics: From data to knowledge. In D. Ifenthaler, D.-K. Mah, & J. Y.-K. Yau (Eds.), Utilizing learning analytics to support study success (pp. 3–25). Springer, Cham. https://doi.org/10.1007/978-3-319-64792-0_1

  • Worthen, B., & Sanders, J. (1987). Educational evaluation: Alternative approaches and practical guidelines. Longman.

Download references

Author information

Authors and Affiliations

  1. Department of Instructional Systems Technology, School of Education, Indiana University, Bloomington, USA

    Theodore W. Frick

  2. School of Education, Indiana University, Bloomington, USA

    Rodney D. Myers

  3. Office of Analytics & Institutional Effectiveness, Virginia Polytechnic Institute and State University, Blacksburg, USA

    Cesur Dagli

Authors
  1. Theodore W. Frick
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rodney D. Myers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cesur Dagli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Theodore W. Frick.

Ethics declarations

Conflict of interest

The authors have no conflicts of interest to declare that are relevant to the content of this article. This study was not funded. The authors did not receive support from any organization for conducting this study.

Research involving human participants and/or animals

This study has been approved and granted exemption for human subjects research by the Indiana University Institutional Review Board, Protocol No. 1304011238.

Informed consent

No informed consent was required for this study by the Indiana University Institutional Review Board, Protocol No. 1304011238. The Privacy Policy for the Indiana University Plagiarism Tutorials and Tests is stated at https://plagiarism.iu.edu/privacy.html. In compliance with the Privacy Policy, we share only aggregate, non-personally identifiable information about participants in this study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Frick, T.W., Myers, R.D. & Dagli, C. Analysis of patterns in time for evaluating effectiveness of first principles of instruction. Education Tech Research Dev 70, 1–29 (2022). https://doi.org/10.1007/s11423-021-10077-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-021-10077-6

Keywords

  • Online learning
  • Learning journeys
  • Innovative learning analytics
  • MOOC
  • First principles of instruction
  • Analysis of patterns in time
  • Instructional effectiveness
  • Recognizing plagiarism
  • Certification tests
  • Mastery learning
  • Web-based instruction