Abstract
As the areas of education and workforce training tend more toward online training, they have emphasized the rapid setup and expansion of online environments. This scaling-up and scaling-out of training programs can be difficult, but there are examples of success available for us to learn from. To capture the lessons learned, we conducted semi-structured interviews across academic, public, private, and non-profit organizations. Using deductive thematic analysis, three major groups of determinants were identified: technological infrastructure, human infrastructure, and governance. A framework consisting of three actionable recommendations was created to facilitate online learning organizational scaling-up including learner (learning experiences and user-friendly design), technical (technical competencies and the use of integrated data), and management (technical competencies for both learners and instructors as well as the use of integrated data) factors.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abouchedid, K., & Eid, G. M. (2004). E-learning challenges in the Arab World: revelations from a case study profile. Quality Assurance in Educational, 12(1), 15–27. https://doi.org/10.1108/09684880410517405
Alrasheedi, M., & Capretz, L. (2015). Determination of Critical Success Factors Affecting Mobile Learning: A Meta-Analysis Approach. Turkish Online Journal of Educational Technology, 14(2), 41–51. Retrieved January 15, 2021, from https://arxiv.org/ftp/arxiv/papers/1801/1801.04288.pdf
Bari, M., Djouab, R., & Hoa, C. P. (2018). Elearning current situation and emerging challenges. PEOPLE: International Journal of Social Sciences, 4(2), https://doi.org/10.20319/pijss.2018.42.97109
Bird, J., & Morgan, C. (2003). Adults contemplating university study at a distance: Issues, themes and concerns. The International Review of Research in Open and Distributed Learning, 4(1). https://doi.org/10.19173/irrodl.v4i1.130
Burnette, D. M. (2015). Negotiating the mine field: Strategies for effective online education administrative leadership in higher education institutions. Quarterly Review of Distance Education, 16(3), 13–25.
Carrillo, C., & Flores, M. A. (2020). Covid-19 and teacher education: A literature review of online teaching and learning practices. European Journal of Teacher Education, 43(4), 466–487. https://doi.org/10.1080/02619768.2020.1821184
Cheok, M. L., & Wong, S. L. (2015). Predictors of e-learning satisfaction in teaching and learning for school teachers: A literature review. International Journal of Instruction, 8(1), 75–90. Retrieved January 15, 2021, https://eric.ed.gov/?id=EJ1085289
Chitura, M. (2015). Making it happen with distance education: Critical concerns for student support services in open and distance learning. Greener Journal of Educational Research, 5(1), 009–016. https://doi.org/10.15580/GJER.2015.1.012315019
Clarke, J., & Dede, C. (2009). Design for scalability: A case study of the River City curriculum. Journal of Science Education and Technology, 18(4), 353–365.
Coburn, C. E. (2003). Rethinking scale: moving beyond numbers to deep and lasting change. Educational Researcher, 32(6), 3–12. https://doi.org/10.3102/0013189X032006003
Collins, C., Buhalis, D., & Peters, M. (2003). Enhancing SMEs business performance through the internet and E-learning platforms. Education + Training, 45(8), 483–494. https://doi.org/10.1108/00400910310508874
Cook, V., & Uranis, J. (2019). Supporting online program quality through online enterprise-level standards. In Ensuring Quality and Integrity in Online Learning Programs (pp. 254–280). IGI Global. Retrieved January 15, 2021, from https://www.igi-global.com/chapter/supporting-online-program-quality-through-online-enterprise-level-standards/225268
Coursetune. (n.d.). Retrieved May 9 2022 from https://coursetune.com/
Craig, S. D., Hu, X., Graesser, A. C., Bargagliotti, A. E., Sterbinsky, A., Cheney, K. R., & Okwumabua, T. (2013). The impact of a technology-based mathematics after-school program using ALEKS on student's knowledge and behaviors. Computers & Education, 68, 495–504. https://doi.org/10.1016/j.compedu.2013.06.010
Craig, S. D., & Douglas, I. (2019). Distributed learning instructional theories. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 43–60). Washington, DC: Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/
Craig, S. D., & Schroeder, N. L. (2020). Science of learning and readiness (SoLaR) recommendation report: science of learning practices for distributed online environments. Arizona State University Tempe United States. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/pdfs/AD1105006.pdf
Craig, S. D., Schroeder, N. L., Roscoe, R. D., Cooke, N. J., Prewitt, D., Siyuan, L., ... & Clark, A. (2020a). Science of learning and readiness state of the art report. Arizona State University Tempe United States. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/pdfs/AD1106815.pdf
Craig, S. D., Li, S., Prewitt, D., Morgan, L. A., & Schroeder, N. L. (2020b). Science of learning and readiness (SoLaR) exemplar report: A path toward learning at scale. Arizona State University. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/citations/AD1104999.pdf
Craig, S. D., Siegel, R. F., & Li, S. (2021). An application of the learning science evaluation checklist: use case report. Arizona State University. Retrieved January 15, 2021, from https://apps.dtic.mil/sti/citations/AD1141190.pdf
Crompton, H., & Burke, D. (2018). The use of mobile learning in higher education: A systematic review. Computers & Education, 123, 53–64. Retrieved May 12, 2022 from https://www.learntechlib.org/p/201270/
Davis, D., Chen, G., Hauff, C., & Houben, G. J. (2018). Activating learning at scale: A review of innovations in online learning strategies. Computers & Education, 125, 327–344. https://doi.org/10.1016/j.compedu.2018.05.019
Dede, C. (2005). Scaling up: evolving innovations beyond ideal settings to challenging contexts of practice. In R. Sawyer (Ed.), The Cambridge Handbook of the Learning Sciences (pp. 551–566). Cambridge University Press. https://doi.org/10.1017/CBO9780511816833.034
Dede, C., Richards, J., & Saxberg, B. (Eds.). (2019). Learning engineering for online education: Theoretical contexts and design-based examples. Routledge. https://doi.org/10.37074/jalt.2019.2.2.17
DeMonte, J. (2013). High-quality professional development for teachers: Supporting teacher training to improve student learning. Center for American Progress. Retrieved May 12, 2022 from https://cdn.americanprogress.org/wp-content/uploads/2013/07/DeMonteLearning4Teachers-1.pdf
Denzin, N. K. (2017). The sociological interview. The Research Act, 122–143. https://doi.org/10.4324/9781315134543-6
D’Mello, S. K., Picard, R., & Graesser, A. C. (2007). Toward an affect-sensitive AutoTutor. IEEE Intelligent Systems, 22(4), 53–61. https://doi.org/10.1109/MIS.2007.79
Dooley, K. E., & Murphrey, T. P. (2000). How the perspectives of administrators, faculty, and support units impact the rate of distance education adoption. Online Journal of Distance Learning Administration, 3(4), 1–12. Retrieved May 12, 2022 from https://www.learntechlib.org/p/92503/
Elmore, R. (1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–27. https://doi.org/10.17763/haer.66.1.g73266758j348t33
Erb, S., & Shaw, R. (2019). Culture Change. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 339–356). Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/
Farid, S., Qadir, M., Ahmed, M. U., & Khattak, M. D. (2018). Critical success factors of E-learning systems: a quality perspective. Pakistan Journal of Distance and Online Learning, 4(1), 1–20. Retrieved January 15, 2021, from https://files.eric.ed.gov/fulltext/EJ1267239.pdf
Finelli, C. J., Ott, M., Gottfried, A. C., Hershock, C., O’neal, C., & Kaplan, M. (2008). Utilizing instructional consultations to enhance the teaching performance of engineering faculty. Journal of Engineering Education, 97(4), 397–411. https://doi.org/10.1002/j.2168-9830.2008.tb00989.x
Fishman, B., Dede, C., & Means, B. (2016). Teaching and technology: New tools for new times. Handbook of research on teaching, 1269–1334. https://doi.org/10.3102/978-0-935302-48-6_21
Floyd, D. L., & Casey-Powell, D. (2004). New roles for student support services in distance learning. New Directions for Community Colleges, 128, 55–64. Retrieved May 12, 2022 from https://www.learntechlib.org/p/100773/
Giattino, T., & Stafford, M. (2019). Governance for learning ecosystems. In J. J. Walcutt & S. Schatz (Eds.), Modernizing learning: Building the future learning ecosystem (pp. 317–338). Government Publishing Office. Retrieved January 15, 2021, from https://adlnet.gov/publications/2019/04/modernizing-learning/
Hafeez, A., Gujjar, A. A., & Noreen, Z. (2014). Demanding need of growing technologies in distance learning system. Turkish Online Journal of Distance Education, 15(4), 170–180. https://doi.org/10.17718/tojde.04872
Hai-Jew, S. (2006). Operationalizing trust: Building the online trust student survey (OTSS). Journal of Interactive Instruction Development, 19(2), 16–30. Retrieved January 15, 2021, from http://www.salt.org/salt.asp?ss=l&pn=jiid
Herzberg, F. I. (1966). Work and the nature of man. World.
Hong, S., Thong, J. Y., & Tam, K. Y. (2006). Understanding continued information technology usage behavior: A comparison of three models in the context of mobile internet. Decision Support Systems, 42(3), 1819–1834. https://doi.org/10.1016/j.dss.2006.03.009
Hunt, T., Carter, R., Zhang, L., & Yang, S. (2020). Micro-credentials: The potential of personalized professional development. Development and Learning in Organizations: An International Journal. Retrieved January 15, 2021, from https://www.tcd.ie/media/tcd/content-assets/pdf/Micro-credentials-The-potential-of-personalized-professional-development.pdf
iDesign. (n.d.). Retrieved May 9 2022 from https://www.idesignedu.org/
Kaplan, A. M., & Haenlein, M. (2016). Higher education and the digital revolution: About MOOCs, SPOCs, social media, and the Cookie Monster. Business horizons, 59(4), 441–450. https://doi.org/10.1016/j.bushor.2016.03.008
Keengwe, J. (Ed.). (2018). Handbook of research on virtual training and mentoring of online instructors. IGI Global.
Khalil, H., & Ebner, M. (2014). MOOCs Completion Rates and Possible Methods to Improve Retention - A Literature Review. In J. Viteli & M. Leikomaa (Eds.), Proceedings of EdMedia 2014–World Conference on Educational Media and Technology (pp. 1305–1313). Association for the Advancement of Computing in Education (AACE). Retrieved December 8, 2020 from https://www.learntechlib.org/primary/p/147656/
Kim, S. W. (2016). MOOCs in higher education. Virtual learning, 121–135. https://doi.org/10.5772/66137
Lancaster, J. W., Stein, S. M., MacLean, L. G., Van Amburgh, J., & Persky, A. M. (2014). Faculty development program models to advance teaching and learning within health science programs. American Journal of Pharmaceutical Education, 78(5). https://www.ajpe.org/content/78/5/99
Lee, J. W. (2010). Online support service quality, online learning acceptance, and student satisfaction. Internet and Higher Education, 13, 277–283. https://doi.org/10.1016/j.iheduc.2010.08.002
Li, S., Siegle, R. F., & Craig, S. D. (2021). Case report for the learning science evaluation checklist. In Proceedings of the human factors and ergonomics society annual meeting (Vol. 65, No. 1, pp. 561–565). Sage CA: Los Angeles, CA: SAGE Publications.
Lisewski, B. (2004). Implementing a learning technology strategy: Top-down strategy meets bottom-up culture. Association for Learning Technology: Research in Learning Technology, 12(2), 175–188. https://doi.org/10.1080/0968776042000216228
Liu, M., Kang, J., & McKelroy, E. (2015). Examining learners’ perspective of taking a MOOC: reasons, excitement, and perception of usefulness. Educational Media International, 52(2), 129–146. https://doi.org/10.1080/09523987.2015.1053289
Liyanagunawardena, T. R., Adams, A. A., & Williams, S. A. (2013). MOOCs: A systematic study of the published literature 2008–2012. International Review of Research in Open and Distributed Learning, 14(3), 202–227 Retrieved January 15, 2021, from https://centaur.reading.ac.uk/33109/
Liyanagunawardena, T. R., Scalzavara, S., & Williams, S. A. (2017). Open badges: a systematic review of peer-reviewed published literature (2011-2015). European Journal of Open, Distance and E-learning, 20(2), 1–16.
Luan, W. S., & Teo, T. (2009). Investigating the technology acceptance among student teachers in Malaysia: An application of the technology acceptance model (TAM). Asia-Pacific Education Researcher, 18(2), 1–1 Retrieved January 15, 2021, from. https://pdfs.semanticscholar.org/3b38/89162a95de50a60737b0f0087c5e2866253b.pdf?_ga=2.40160161.286461491.1611557988-176642363.1610760777
Ludwig-Hardman, S., & Dunlap, J. C. (2003). Learner support services for online students: Scaffolding for success. International Journal of Research in Open and Distance Learning, 4(1). https://doi.org/10.19173/irrodl.v4i1.131
Martin, S. M., & Trang, M. L. (2018). Creating personalized learning using aggregated data from students’ online conversational interactions in customized groups. Learning Engineering for Online Education: Theoretical Contexts and Design-Based Examples, 144–165.
McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. University of Prince Edward Island. Retrieved January 15, 2021, from https://www.oerknowledgecloud.org/archive/MOOC_Final.pdf
Means, B., Toyama, Y., Murphy, R., & Baki, M. (2013). The effectiveness of online and blended learning: A meta-analysis of the empirical literature. Teachers College Record, 115(3), 1–47.
Miller, S., Meier, E., Payne-Bourcy, L., Shablack, S., Newman, D. L., Wan, T. Y., & Pack, G. (2003). Technology use as a catalyst for change: A leadership framework for transforming urban teacher preparation. IEJLL: International Electronic Journal for Leadership in Learning, 7(12), 2461–2480 Retrieved January 15, 2021, from https://www.researchgate.net/publication/295922106_Technology_use_as_a_catalyst_for_change_A_leadership_framework_for_transforming_urban_teacher_preparation
Morel, R. P., Coburn, C., Catterson, A. K., & Higgs, J. (2019). The multiple meanings of scale: implications for researchers and practitioners. Educational Researcher, 48(6), 369–377. https://doi.org/10.3102/0013189X19860531
Motiwalla, L. F. (2007). Mobile learning: A framework and evaluation. Computers & Education, 49(3), 581–596. https://doi.org/10.1016/j.compedu.2005.10.011
Mulder, F., & Jansen, D. (2015). MOOCs for opening up education and the OpenupEd initiative. In C. J. Bonk, M. M. Lee, T. C. Reeves, & T. H. Reynolds (Eds.), MOOCs and open education around the world. Routledge. https://doi.org/10.4324/9781315751108-16
Newby, T., & Cheng, Z. (2019). Instructional design badges. Education Technology Research Development, 68, 1053–1067. https://doi.org/10.1007/s11423-019-09719-7
Nichols, M. (2010). Student perceptions of support services and the influence of targeted interventions on retention in distance education. Distance education, 31(1), 93–113. https://doi.org/10.1080/01587911003725048
Obexer, R. (2018). Scaling online learning: the case for a programme-level approach. In A. Altmann, B. Ebersberger, C. Mössenlechner, & D. Wieser (Eds.), The Disruptive Power of Online Education (pp. 7–25). Emerald Publishing Limited. https://doi.org/10.1108/978-1-78754-325-620181002
Park, J. H., & Choi, H. J. (2009). Factors influencing adult learners’ decision to drop out or persist in online learning. Journal of Educational Technology & Society, 12(4), 207–217.
Patru, M., & Balaji, V. (2016). Making Sense of MOOCs: A Guide for Policy-Makers in Developing Countries. http://oasis.col.org/bitstream/handle/11599/2356/2016_Guide-on-MOOCs-for-Policy-Makers-in-Developing-Countries.pdf?sequence=4&isAllowed=y
Pelletier, K., Brown, M., Brooks, D. C., McCormack, M., Reeves, J., Arbino, N. … Mondelli, V. (2021). 2021 EDUCAUSE Horizon Report Teaching and Learning Edition. EDU. Retrieved May 12, 2022 from https://www.learntechlib.org/p/219489/
Rawls, T. (2016, February 8). Achieving economies of scale in Higher Education. The EvoLLLution. Retrieved April 11, 2022, from https://evolllution.com/managing-institution/higher_ed_business/achieving-economies-of-scale-in-higher-education/
Rees-Johnstone, E. (2016, February 19). Four levers for scaling higher education. The EvoLLLution. Computers & Education, 68, 495–504. Retrieved April 11, 2022, from https://evolllution.com/managing-institution/higher_ed_business/four-levers-for-scaling-higher-education/dbehaviors
Roll, I., Russell, D. M., & Gašević, D. (2018). Learning at scale. International Journal of Artificial Intelligence in Education, 28(4), 471–477. https://doi.org/10.1007/s40593-018-0170-7
Rovai, A. P., & Downey, J. R. (2010). Why some distance education programs fail while others succeed in a global environment. Internet and Higher Education, 13(3), 141–147. Elsevier Ltd. Retrieved May 12, 2022 from https://www.learntechlib.org/p/108384/
Screnci, D. (2017, April 24). Building communities of practice to support EdTech. EDUCAUSE Review. Retrieved May 9, 2022, from https://er.educause.edu/articles/2017/4/building-communities-of-practice-to-support-edtech
Selvaratnam, R. M., & Sankey, M. D. (2021). An integrative literature review of the implementation of microcredentials in higher education: Implications for practice in Australasia. Journal of Teaching and Learning for Graduate Employability, 12(1), 1–17. https://doi.org/10.21153/JTLGE2021VOL12NO1ART942
Smith, G. G., & Kurthen, H. (2007). Front-Stage and Back-Stage in Hybrid E-Learning Face-to-Face Courses. International Journal on E-Learning, 6(3), 455–474. Association for the Advancement of Computing in Education (AACE). Retrieved May 11, 2022 from https://www.learntechlib.org/primary/p/20008/
Sohoni, S., Craig, S. D., & Vedula, K. (2017). A blueprint for an ecosystem for supporting high quality education for engineering. Journal of Engineering Education Transformations, 30(4), 58–66. https://doi.org/10.16920/jeet/2017/v30i4/114107
Technology for Education - Eberly Center - Carnegie Mellon University. (n.d.). Retrieved May 9 2022, from https://www.cmu.edu/teaching/technology/index.html
Wang, X., Talluri, S. T., & Rose, C., & Koedinger, K. (2019). UpGrade: sourcing student open-ended solutions to create scalable learning opportunities. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale. New York, NY, USA, Article 17, 1–10. https://doi.org/10.1145/3330430.3333614
Westley, F., Antadze, N., Riddell, D. J., Robinson, K., & Geobey, S. (2014). Five configurations for scaling up social innovation: case examples of nonprofit organizations from Canada. The Journal of Applied Behavioral Science, 50(3), 234–260. https://doi.org/10.1177/0021886314532945
Wong, D. (2007). A critical literature review on e-learning limitations. Journal for the Advancement of Science and Arts, 2(1), 55–62 Retrieved January 15, 2021, from, https://www.researchgate.net/profile/Dominic_Wong3/publication/228677287_A_Critical_Literature_Review_on_e-Learning_Limitations/links/57a0a26a08ae5f8b25891cee.pdf
Wylie, C., & MacDonald, J. (2020). Scaling up research-based educational practices. Oxford University Press. https://doi.org/10.1093/OBO/9780199756810-0254
Funding
The current paper was partially funded by HQ0034-19-C-0015, a contract from the United States Department of Defense Advanced Distributed Learning Initiative. The statements within this work do not reflect the opinion, view, or policy of the funding agency.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The author has no relevant financial or non-financial interests to disclose.
Ethics Approval
All procedures performed in this study were conducted in accordance with the ethical standards of the institutional review board (IRB). The methodology and research instruments for this study were approved by the Institutional Review Board at Arizona State University. Informed consent was obtained from each participant in this study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix 1
Appendix 1
Interview protocol
My name is {Interviewer Name}. I am conducting an interview to develop a case study to collect best practices for designing, delivering, and managing online learning at scale in academia and industry. The purpose of this interview and case study is to generate evidence to engage stakeholders in relevant dialogue regarding online learning at scale.
Today, I will be conducting the interview. This interview will last anywhere from 30 to 45 min. The information in this interview will be used only for research purposes. Is it ok if I record this conversation?
Overview and Understanding of the Organizations
-
1.
What does your organization do?
-
2.
Would you describe the online teaching and learning platforms that you use in your organization, such as their sizes and their objectives?
Current Ecosystems and Scale-Up
-
3.
Can you explain how your organization scaled up?
-
4.
What were the most useful organizational changes for scaling up?
-
5.
How did you support learning as the organization scales up?
-
6.
Can you relate to any other factors that had an impact on your organization as it scaled up? Specifically, can you reflect on such things as technology support, instructional strategies, organizational communication, the support structures within your organization, multidisciplinary collaboration, or other factors?
The Application of Technology
-
7.
Would you elaborate on how your organization adopts the technologies in all aspects of scaling up? For example, how did your organization deal with internal communication, staff training, and evaluation of instructors and students?
Prior Experiences, Lessons Learned, and Future Goals
-
8.
Can you share any valuable lessons learned during the adoption/creation of a large amount of technology during the scaling up of your organization?
-
9.
Where do you see the direction of your organization going over the next five years?
Rights and permissions
About this article
Cite this article
Li, S., Craig, S.D. & Schroeder, N.L. Lessons Learned from Online Learning at Scale: a Study of Exemplar Learning Organizations. TechTrends 67, 84–97 (2023). https://doi.org/10.1007/s11528-022-00761-6
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11528-022-00761-6