, Volume 62, Issue 1, pp 58–70 | Cite as

Using the “Indicators of Engaged Learning Online” Framework to Evaluate Online Course Quality

  • Paula M. BigatelEmail author
  • Stephanie Edel-Malizia
Original Paper


This article is a case study of the use of the Indicators of Engaged Learning Online (IELO) framework (See Appendix 1) as a guide to evaluate the quality of online courses. The framework lends itself well to measures of engagement, particularly, in terms of online course design because of its comprehensiveness. Six online courses were evaluated for quality in terms of engaged learning based on thirty indicators contained within the framework. Results ranged from a score of 21 to 71 out of a potential total score of 90. This 0–90 scale represented a continuum of passive to engaged learning. The purpose of the pilot study was to explore how the Indicators of Engaged Learning Online (IELO) framework could be used as a tool for evaluating the quality of online courses by instructional designers (IDs) and instructors. Insights into the practical use of the IELO framework and the need for improved guidelines for IDs and instructors as they assess the amount of student engagement designed in a course are provided. Recommendations for practice have implications for both aspects of engagement: how a course is designed and how it is delivered.


Instructional design Online learning Quality online instruction Student engagement 


Compliance with Ethical Standards

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

Paula M Bigatel declares that she has no conflict of interest. Stephanie Edel-Malizia declares that she has no conflict of interest.


  1. Allen, I., & Seaman, J. (2015). Grade level: Tracking online education in the United States. Babson Survey Research Group.Google Scholar
  2. ASHE Higher Education Report (2014). Effects from student engagement online, 40(6), 67–73.
  3. Association of American Colleges and Universities (AAC&U). (2007). College learning for the new global century: A report from the national leadership council for liberal education & America's promise. Washington, DC: Author.Google Scholar
  4. Bigatel, P., & Williams, V. (2015). Measuring student engagement in an online program. Online journal of distance learning and administration. Retrieved from
  5. Blackmon, S. J., & Major, C. (2012). Student experiences in online courses: A qualitative research synthesis. The Quarterly Review of Distance Education, 13(2), 77–85.Google Scholar
  6. Boston, W. E., & Ice, P. (2011). Assessing retention in online learning: An administrative perspective. Online Journal of Distance Learning Administration. Retrieved from,
  7. Boston, W., Diaz, S. R., Gibson, A. M., Ice, P., Richardson, J., & Swan, K. (2010). An exploration of the relationship between indicators of the Community of Inquiry framework and retention in online programs. Journal of Asynchronous Learning Networks, 14(1), 3–19.Google Scholar
  8. Chickering, A. W., & Gamson, Z. F. (1987). Seven principles for good practice in undergraduate education. AAHE Bulletin, 3, 7.Google Scholar
  9. Czerkawski, B. C., & Lyman III, E. W. (2016). An instructional design framework for fostering student engagement in online learning environments. TechTrends, 60, 532–539. Scholar
  10. Dayton, D., & Vaughn, M. M. (2007). Developing a quality assurance process to guide the design and assessment of online courses. Technical Communication, 54(4), 475–489.Google Scholar
  11. Edel-Malizia, S., & Brautigam, K. (2014). Gauging the quality of online learning by measuring 21st century engagement, in Proceedings of the 13th European Conference on E-Learning, Copenhagen (pp. 700–703). Copenhagen: Aalborg University.Google Scholar
  12. Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109.CrossRefGoogle Scholar
  13. Garrison, D. R. (2007). Online community of inquiry review: Social, cognitive, and teaching presence issues. Journal of Asynchronous Learning Networks, 7(1), 61–72.Google Scholar
  14. Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical thinking, cognitive presence, and computer conferencing in distance education. American Journal of Distance Education, 15(1), 7–23.CrossRefGoogle Scholar
  15. Gray, J. A., & DiLoreto, M. (2016). The effects of student engagement, student satisfaction, and perceived learning in online learning environments. International Journal of Educational Leadership Preparation, 11(1), n1.Google Scholar
  16. Jones, B. F., Valdez, G., Nowakowski, J., & Rasmussen, C. (1995). Plugging in: Choosing and using educational technology. Washington, DC: North central regional educational lab., oak brook, IL; Council for Educational Development and Research.Google Scholar
  17. Jung, I. (2011). The dimensions of e-learning quality: From the learner’s perspective. Educational Technology Research and Development, 59(4), 445–464.CrossRefGoogle Scholar
  18. Kahn, P. E. (2014). Theorising student engagement in higher education. British Educational Research Journal, 40(6), 1005–1018.CrossRefGoogle Scholar
  19. Kuh, G. D. (2001). Assessing what really matters to student learning inside the national survey of student engagement. Change: The Magazine of Higher Learning, 33(3), 10–17.CrossRefGoogle Scholar
  20. Kuh, G. D. (2003). What we’re learning about student engagement from NSSE: Benchmarks for effective educational practices. Change, 35(2), 24–32.CrossRefGoogle Scholar
  21. Kuh, G. D. (2008). Excerpt from high-impact educational practices: What they are, who has access to them, and why they matter. Association of American Colleges and Universities. Retrieved from
  22. Lehman, R. M., & Conceição, S. C. O. (2014). Motivating and retaining online students: Research-based strategies that work. San Francisco: Jossey-Bass/Wiley.Google Scholar
  23. Means, B. (1993). Using technology to support education reform (pp. 20402–29328). US Government Printing Office, Superintendent of Documents, Mail Stop: SSOP, Washington, DC.Google Scholar
  24. Menchaca, M. P., & Bekele, T. A. (2008). Learner and instructor identified success factors in distance education. Distance Education, 29(3), 231–252.CrossRefGoogle Scholar
  25. Morris, L. V., & Finnegan, C. L. (2009). Best practices in predicting and encouraging student persistence and achievement online. Journal of College Student Retention, 10(1), 55–64.CrossRefGoogle Scholar
  26. National Survey of Student Engagement. (2000). Improving the college experience: National benchmarks of effective educational practice. Bloomington: Indiana University Center for Postsecondary Research and Planning.Google Scholar
  27. North Central Regional Educational Laboratory & the Metiri Group. (2003). enGuage 21st century skills: Literacy in the digital age. Retrieved September 30, 2016, from
  28. Phipps, R., & Merisotis, J. (2000). Quality on the line: Benchmarks for success in internet-based distance education. Retrieved from
  29. Quality Matters (QM). (2011). Helping you deliver on your online promise. Retrieved from
  30. Revere, L., & Kovach, J. V. (2011). Online technologies for engaged learning: A meaningful synthesis for educators. The Quarterly Review of Distance Education, 12(2), 113–124.Google Scholar
  31. Robinson, C. C., & Hullinger, H. (2008). New benchmarks in higher education: Student engagement in online learning. Journal of Education for Business, 84(2), 101–109.CrossRefGoogle Scholar
  32. Rovai, A. P., & Wighting, M. J. (2005). Feelings of alienation and community among higher education students in a virtual classroom. The Internet and Higher Education, 8, 97–110.CrossRefGoogle Scholar
  33. Ryan, R. M., & Deci, E. L. (2000). Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary Educational Psychology, 25, 54–67.CrossRefGoogle Scholar
  34. Southard, S., & Mooney, M. (2015). A comparative analysis of distance education quality assurance standards. The Quarterly Review of Distance Education, 16(1), 55–68.Google Scholar
  35. Street, H. D. (2010). Factors influencing a learner’s decision to drop-out or persist in higher education distance learning. Online Journal of Distance Learning Administration, 13(4). Retrieved from
  36. Swaner, L., & Brownell, J. (2009). Outcomes of high impact practices for undeserved students. A review of the literature. Prepared for the Association of American Colleges and Universities.Google Scholar
  37. Wyatt, L. G. (2011). Nontraditional student engagement: Increasing adult student success and retention. The Journal of Continuing Higher Education, 59(1), 10–20.CrossRefGoogle Scholar
  38. Yin, K. (2009). Case study research: Design and method (4th ed.). Thousand Oaks: Sage.Google Scholar

Copyright information

© Association for Educational Communications & Technology 2017

Authors and Affiliations

  1. 1.World Campus, Faculty DevelopmentPenn State UniversityUniversity ParkUSA
  2. 2.Education Technology ServicesPenn State UniversityUniversity ParkUSA

Personalised recommendations