Customizable Tool for Online Training Evaluation

  • Cheryl A. MurphyEmail author
  • Elizabeth A. Keiffer
  • Jack A. Neal
  • Jessica Howton
Living reference work entry


A proliferation of retail online training materials exists for many industries, but often the person in charge of choosing the most appropriate online training materials is neither a training expert nor versed in best practices associated with online training. To this end, it is critical that uninformed decision makers have access to an easy-to-use evaluation tool which allows for the assessment of strengths and weaknesses among multiple online training programs. Additionally, this tool must take into account the context of the training situation to ensure the chosen program is not only instructionally sound but also meets contextually specific training needs. This article describes the creation, testing, and application of the Customizable Tool for Online Training Evaluation (CTOTE), an evaluation instrument developed to help decision makers: (1) assess multiple online training programs against known best practices and (2) consider context-specific training needs via a weighting process. The three-step development process is explained including item selection and revision, determination of content validity and reliability, and the use of a Delphi panel to inform contextualized weighting. The instrument is then tested across multiple online training programs with results compared to an established online training evaluation instrument to illustrate the impact of the contextualized weighting. Lastly, the application of the instrument in a specific industry setting (food service) is presented to demonstrate the effectiveness of the instrument in this setting and to establish the potential of the CTOTE in helping uninformed decision makers assess multiple online training programs and make effective context-specific purchasing decisions.


Evaluation instrument Online training Business and industry Customizable Contextual 



This research was supported in part by an USDA National Integrated Food Safety Initiative Grant to the authors.


  1. Agariya, A. K., & Singh, D. (2012). e-Learning quality: Scale development and validation in Indian context. Knowledge Management & E-Learning: An International Journal (KM&EL), 4(4), 500–517. Retrieved from
  2. Alkhattabi, M., Neagu, D., & Cullen, A. (2010). Information quality framework for e-learning systems. Knowledge Management & E-Learning: An International Journal (KM&EL), 2(4), 340–362. Retrieved from
  3. Almanza, B. A., & Nesmith, M. S. (2004). Food safety certification regulations in the United States. Journal of Environmental Health, 66(9), 10–14.Google Scholar
  4. ATD Research. (2015). 2014 state of the industry report. Alexandria, VA: Association for Talent Development.Google Scholar
  5. Barker, N. K. (2004). The quality dilemma in online education. In T. Anderson & F. Elloumi (Eds.), Theory and practice of online learning (pp. 385–421). Retrieved from
  6. Barker, N. K. (2007). E-learning quality standards for consumer protection and consumer confidence: A Canadian case study in e-learning quality assurance. Journal of Educational Technology and Society, 10(2), 109–119. Retrieved from
  7. Batalla-Busquets, J. M., & Pacheco-Bernal, C. (2013). On-the-job e-learning: Workers’ attitudes and perceptions. The International Review of Research in Open and Distributed Learning, 14(1), 40–64. Retrieved from
  8. Becker, K., Fleming, J., & Keijsers, W. (2012). E-learning: Ageing workforce versus technology-savvy generation. Education + Training, 54(5), 385–400. 10.1108/00400911211244687.Google Scholar
  9. Caudill, J. G., & Reeves, B. (2014). Strategic management of workplace e-learning. In S. Hai-Jew (Ed.), Remote workforce training: Effective technologies and strategies (pp. 17–27). Hershey, PA: IGI Global.CrossRefGoogle Scholar
  10. Egan, M. B., Raats, M. M., Grubb, S. M., Eves, A., Lumbers, M. L., Dean, M. S., & Adams, M. R. (2007). A review of food safety and food hygiene training studies in the commercial sector. Food Control, 18, 1180–1190. doi:10.1016/j.foodcont.2006.08.001.CrossRefGoogle Scholar
  11. Frash, R., Jr., Binkley, M., Nelson, D., & Almanza, B. (2006). Transfer of training efficacy in US food safety accreditation. Journal of Culinary Science & Technology, 4(2–3), 7–38.Google Scholar
  12. Gillis, L. (2000). Quality standards for evaluating multimedia and online training. Whitby, ON: McGraw-Hill Ryerson.Google Scholar
  13. Grollman, W. K., & Cannon, D. (2003). eLearning: A better chalkboard. Financial Executive, 19(8), 45–47.Google Scholar
  14. Howton, J., Keiffer, E., Murphy, C. A., Sirsat, A. S., O’Bryan, C. A., Ricke, S. C., Neal, J. A. (2016). A comparison of online food safety training programs using the Customizable Tool for Online Training Evaluation. Food Control, 59, 82–87. doi:10.1016/j.foodcont.2015.04.034.Google Scholar
  15. Illinois Online Network. (2010). ION’s quality online course initiative. Retrieved from
  16. Istrate, O. (2013). Criteria for e-learning programmes evaluation. Journal of Advanced Distributed Learning Technology, 1(1), 27–35. Retrieved from
  17. Jan, P. T., Lu, H. P., & Chou, T. C. (2012). The adoption of e-learning: An institutional theory perspective. Turkish Online Journal of Educational Technology-TOJET, 11(3), 326–343. Retrieved from
  18. MarylandOnline. (2011). Quality Matters rubric standards 2011–2013 edition with assigned point values. Retrieved from
  19. McCue, T. J. (2014, October). Online learning industry poised for $107 billion in 2017. Forbes. Retrieved from
  20. Mosharraf, M., & Taghiyareh, F. (2013). Qualitative development of e-learning environments through a learner relationship management methodology. Knowledge Management & E-Learning: An International Journal (KM&EL), 5(1), 56–65. Retrieved from
  21. Murphy, C. A., Keiffer, E. A., Neal, J. A., & Crandall, P. G. (2013). A customizable evaluation instrument to facilitate comparisons of existing online training programs. Knowledge Management & E-Learning: An International Journal, 5(3), 251–268. Retrieved from
  22. Neal, J. A., Dawson, M., & Madera, J. M. (2011). Identifying food safety concerns when communication barriers exist. Journal of Food Science Education, 10, 36–44. doi:10.1111/j.1541-4329.2011.00128.x.CrossRefGoogle Scholar
  23. Neal, J. A., Murphy, C. A., Crandall, P. G., O’Bryan, C. A., Keiffer, E., & Ricke, S. C. (2011). Development of an evaluation tool for online food safety training programs. Journal of Food Science Education, 10(1), 9–12. Retrieved from
  24. Nunnaly, J. (1978). Psychometric theory. New York, NY: McGraw-Hill.Google Scholar
  25. Pappas, C. (2015, January). The top e-learning statistics and facts for 2015 you need to know. eLearning Industry. Retrieved from
  26. Pisik, G. B. (1997). Is this course instructional sound? A guide to evaluating online training courses. Educational Technology, 37(4), 50–59.Google Scholar
  27. Pisik, G. B. (2004). Is this course instructionally sound? An updated guide to evaluating online training courses. Educational Technology, 44(1), 57–60.
  28. Santerre, C. R. (2005). X-train: Teaching professionals remotely. The Journal of Nutrition, 135(5), 1248–1252. Retrieved from Google Scholar
  29. Seufert, S. (2002). eLearning business models: Strategies, success factors, and best practice examples. In C. Wankel & R. DeFillippi (Eds.), Rethinking management education for the twenty-first century (pp. 109–132). Greenwich, CT: Information Age Publishing.Google Scholar
  30. Skillsoft. (2013). From boardroom to desktop: Skillsoft connects corporate strategy to the performance needs of individual learners with new Enterprise Learning Connection [Press Release]. Retrieved from
  31. Steele, J. (2015). Comparing computer-based employee training with traditional face-to-face methods. In D. Slykhuis & G. Marks (Eds.), Proceedings of Society for Information Technology & Teacher Education International Conference 2015 (pp. 504–507). Chesapeake, VA: Association for the Advancement of Computing in Education (AACE). Retrieved from
  32. Strother, J. B. (2002). An assessment of the effectiveness of e-learning in corporate training programs. The International Review of Research in Open and Distance Learning, 3(1). Retrieved from
  33. Training. (2014, November/December). 2014 training industry report. Training, 51(8), 16–29. Retrieved from
  34. Turner, R. C., & Carlson, L. (2003). Indexes of item-objective congruence for multidimensional items. International Journal of Testing, 3(2), 163–171. doi:10.1207/S15327574IJT0302_5.CrossRefGoogle Scholar
  35. Van Rooij, S. W. (2011). Instructional design and project management: Complementary or divergent? Educational Technology Research and Development, 59(1), 139–158.CrossRefGoogle Scholar
  36. Zaied, A. N. H. (2012). Multi-criteria evaluation approach for e-learning technologies: Selection criteria using AHP. International Journal on E-Learning, 11(4), 465–485.Google Scholar
  37. Zhang, D., & Nunamaker, J. F. (2003). Powering e-learning in the new millennium: An overview of e-learning and enabling technology. Information Systems Frontiers, 5(2), 207–218. doi:10.1023/A:1022609809036.CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Cheryl A. Murphy
    • 1
    Email author
  • Elizabeth A. Keiffer
    • 2
  • Jack A. Neal
    • 3
  • Jessica Howton
    • 4
  1. 1.Educational Technology, College of Education and Health ProfessionsUniveristy of ArkansasFayettevilleUSA
  2. 2.Mathematical Sciences, J. William Fulbright College of Arts and SciencesUniversity of ArkansasFayettevilleUSA
  3. 3.Conrad N. Hilton College of Hotel and Restaurant ManegementUniversity of HoustonHoustonUSA
  4. 4.H-E-B Grocery Company, LPSan AntonioUSA

Personalised recommendations