Abstract
The purpose of this qualitative study was to examine how learning design and educational technology degree programs prepare students to evaluate in practice. This study involved the curricular mapping of 16 graduate instructional design programs and 29 semi-structured interviews with program faculty and recent postgraduates. Based on information shared during the interviews, this study produced nine themes according to three metathemes: (a) the role of problem-solving in evaluation, (b) alignment of evaluation activities in instructional design, and (c) relevancy of evaluation in instructional design. The findings revealed that evaluation was not prioritized for most programs, due to lack of time, client resources, employer lack of interest, and limited faculty experience in evaluation. Other recommendations are provided for how to enhance evaluative practices in instructional design coursework.
Similar content being viewed by others
References
Armstrong, A. (2004). Instructional design in the real world: A view from the trenches. Education Technology & Society, 7, 128–129.
Bowman, S., Armstrong, A., Lane, C., & Lane, M. (2015). Evaluation in instructional Design practice: A view from the stakeholders (Doctoral dissertation). Available from ProQuest Dissertations and Theses database.
Chen, W., Moore, J. L., & Vo, N. (2012). Formative evaluation with novice designers: Two case studies within an online multimedia development course. International Journal of Instructional Media, 39(2), 95–111.
Creswell, J. W. (2015). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson Education.
Dabbagh, N., & English, M. (2015). Using student self-ratings to assess the alignment of instructional design competencies and courses in a graduate program. Tech Trends, 59(4), 22–31.
DeVaughn, P., & Stefaniak, J. (2020). An exploration of the challenges instructional designers encounter while conducting evaluations. Performance Improvement Quarterly., 21(1), 17–42.
Dooley, K. E., Lindner, J. R., Telg, R. W., Irani, T., Moore, L., & Lundy, L. (2007). Roadmap to measuring distance education instructional design competencies. Quarterly Review of Distance Education, 8(2), 151.
Endsley, M. R. (2018). Expertise and situation awareness. In K. A. Ericsson, R. R. Hofman, A. Kozbelt, & A. M. Williams (Eds.), The Cambridge handbook of expertise and expert performance (2nd ed., pp. 633–651). Cambridge: Cambridge University Press.
Everhart, D., Bushway, D., & Schejbal, D. (2016). Communicating the value of competencies. Washington, D.C.: American Council on Education. https://www.acenet.edu/news-room/Documents/Communicating-the-Value-of-Competencies.pdf
Ericsson, K. A. (2018). The differential influence of experience, practice, and deliberate practice on the development of superior individual performance of experts. In K. A. Ericsson, R. R. Hoffman, A. Kozbelt, & A. M. Williams (Eds.), The Cambridge handbook of expertise and expert performance (2nd ed., pp. 745–769). Cambridge: Cambridge University Press.
Fadde, P. J. (2009). Instructional design for advanced learners: Training recognition skills to hasten expertise. Educational Technology Research and Development, 57(3), 359–376.
Giberson, T. R., Tracey, M. W., & Harris, M. T. (2006). Confirmative evaluation of training outcomes. Performance Improvement Quarterly, 19(4), 43–61.
Guerra-Lopez, I. (2007). Evaluating impact: Evaluation and continual improvement for performance improvement practitioners. Champaign, IL: Human Resource Development Press.
Guerra-Lopez, I. J. (2008). Performance evaluation: Proven approaches for improving program and organizational performance. San Francisco, CA: Wiley.
Harwood, T., & Garry, T. (2003). An overview of content analysis. The Marketing Review, 3(4), 479–498.
Hays, D., & Singh, A. (2012). Qualitative inquiry in clinical and educational settings. New York: Guilford Press.
Herrington, A., & Herrington, J. (Eds.). (2008). What is an authentic learning environment? in authentic learning environments in higher education (pp. 68–77). London: Information Science Publishing.
Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59–71.
Klein, J. D., & Fox, E. J. (2004). Performance improvement competencies for instructional technologists. Tech Trends, 48(2), 22–25.
Klein, J. D., & Jun, S. (2014). Skills for instructional design professionals. Performance Improvement, 53(2), 41–46.
Klein, J. D., & Kelly, W. Q. (2018). Competencies for instructional designers: A view from employers. Performance Improvement Quarterly, 31(3), 225–247.
Koszalka, T. A., Russ-Eft, D. F., & Reiser, R. (2013). Instructional designer competencies: The standards (4th ed.). Information Age Publishing.
Larson, M. (2004). Instructional design career environments: Survey of the alignment of preparation and practice. Tech Trends, 49(6), 22–34.
Larson, M. B., & Lockee, B. B. (2009). Instructional design practice: Career environments, job roles, and a climate of change. Performance Improvement Quarterly, 17(1), 22–40.
Le Maistre, C. (1998). What is an expert instructional designer? Evidence of expert performance during formative evaluation. Educational Technology Research and Development, 46(3), 21–36.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.
Marshall, J., & Rossett, A. (2014). Perceptions of barriers to the evaluation of workplace learning programs. Performance Improvement Quarterly, 27(3), 7–26.
Moller, L., & Mallin, P. (1996). Evaluation practices of instructional designers and organizational supports and barriers. Performance Improvement Quarterly, 9(4), 82–92.
Parhar, M., & Mishra, S. (2000). Competencies for Web-based instructional designers. Indian Journal of Open Learning, 9(3), 415–422.
Richardson, J. C., Ashby, I., Alshammari, A. N., Cheng, Z., Johnson, B. S., Krause, T. S., ... & Wang, H. (2019). Faculty and instructional designers on building successful collaborative relationships. Educational Technology Research and Development, 67(4), 855–880.
Richey, R. C., Fields, D. C., & Foxon, M. (2001). Instructional design competencies: Thestandards (3rd ed.). Syracuse, NY: ERIC Clearinghouse on Information & Technology.
Richey, R. C., & Klein, J. D. (2005). Developmental research methods: Creating knowledge from instructional design and development practice. Journal of Computing in Higher Education, 16(2), 23–38.
Ritzhaupt, A. D., & Kumar, S. (2015). Knowledge and skills needed by instructional designers in higher education. Performance Improvement Quarterly, 28(3), 51–69.
Ritzhaupt, A. D., & Martin, F. (2014). Development and validation of the educational technologist multimedia competency survey. Educational Technology Research and Development, 62(1), 13–33.
Ritzhaupt, A. D., Martin, F., Pastore, R., & Kang, Y. (2018). Development and validation of the educational technologist competencies survey (ETCS): Knowledge, skills, and abilities. Journal of Computing in Higher Education, 30(1), 3–33.
Rozitis, C. P. (2017). Instructional design competencies for online high school teachers modifying their own courses. Tech Trends, 61(5), 428–437.
Rossett, A., & Sheldon, K. (2001). Beyond the podium: Delivering training and performance to a digital world. Education & Training, 44(2/3), 104–108.
Scriven, M. (1991). Evaluation thesaurus (4th ed.). Thousand Oak, CA: Sage.
Seels, B. B., & Richey, R. C. (1994). Instructional technology: The definition and domains of the field. Bloomington, IN: Association for Educational Communication and Technology.
Slagter van Tryon, P. J., McDonald, J., & Hirumi, A. (2018). Preparing the next generation of instructional designers: A cross-institution faculty collaboration. Journal of Computing in Higher Education, 30(1), 125–153.
Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17), 1–6.
Sugar, W. (2014). Development and formative evaluation of multimedia studies for instructional design and technology students. Tech Trends, 58(5), 36–52.
Sugar, W. (2014b). Studies of ID practices: A review and synthesis of research on ID current practices. Springer International Publishing. ProQuest Ebook Central.
Sugar, W., Hoard, B., Brown, A., & Daniels, L. (2012). Identifying multimedia production competencies and skills of instructional design and technology professionals: An analysis of recent job postings. Journal Educational Technology Systems, 40(3), 227–249.
Sugar, W., & Luterbach, K. J. (2016). Using critical incidents of instructional design and multimedia production activities to investigate instructional designers' current practices and roles. Educational Technology Research and Development, 64(2), 285–312.
Tessmer, M. (1993). Planning and conducting formative evauations. London: Kogan Page.
Tessmer, M. (1994). Formative evaluation alternatives. Performance Improvement Quarterly, 7(1), 3–18.
Thach, E. C., & Murphy, K. L. (1995). Competencies for distance education professionals. Educational Technology Research and Development, 43(1), 57–79.
Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner's mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13.
Villachica, S. W., Marker, A., & Taylor, K. (2010). But what do they really expect? Employer perceptions of the skills of entry-level instructional designers. Performance Improvement Quarterly, 22, 33–51.
Visscher-Voerman, I. (2017). Necessary ingredients for the education of designers. In A. Carr-Chellman & G. Rowland (Eds.), Issues in technology, learning, and instructional design: Classic and contemporary dialogues (pp. 73–80). New York, NY: Routledge.
Wang, G. G., & Wilcox, D. (2006). Training evaluation: knowing more than is practiced. Advances in Developing Human Resources, 8(4), 528–539.
Wedman, J., & Tessmer, M. (1993). Instructional designers' decisions and priorities: A survey of design practice. Performance Improvement Quarterly, 6(2), 43–57.
Weinberg, J. B., & Stephen, M. L. (2002). Participatory design in a human-computer interaction course: Teaching ethnography methods to computer scientists. In Proceedings of the 3rd SIGCSE technical symposium on Computer science education (pp. 237–241). New York, NY: ACM
Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, 43(3), 29–48.
Williams, P. E. (2003). Roles and competencies for distance education programs in higher education institutions. The American Journal of Distance Education, 17(1), 45–57.
Williams, D. D., South, J. B., Yanchar, S. C., Wilson, B. G., & Allen, S. (2011). How do instructional designers evaluate? A qualitative study of evaluation in practice. Education Technology Research and Development, 59, 885–907.
York, C. S., & Ertmer, P. A. (2011). Towards an understanding of instructional design heuristics: An exploratory Delphi study. Educational Technology Research and Development, 59(6), 841–863.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1
Interview questions (faculty)
Can you provide a brief description explaining your job responsibilities?
What skills should instructional design (ID) students possess to successfully demonstrate understanding of ID principles?
What activities provide the needed practice to ensure mastery of these skills?
How do you assess the students’ level of competence in these skills?
Which courses in the ID curriculum of your institution do you expect will best prepare instructional design students for work in the ID field after graduation?
What data could you share to support this expectation?
How important is evaluation in the ID process?
When should evaluation be conducted in the ID process?
Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct formative evaluation?
What course activities provide the opportunity to demonstrate mastery of this skill?
Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct summative evaluation?
What course activities provide the opportunity to demonstrate mastery of this skill?
Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct confirmative evaluation?
What course activities provide the opportunity to demonstrate mastery of this skill?
Appendix 2
Interview questions (post-graduates)
Can you provide a brief description explaining your job responsibilities?
What skills should instructional design (ID) students possess to successfully demonstrate understanding of ID principles?
What activities provide the needed practice to ensure mastery of these skills?
How should your level of competence in these skills be assessed?
Which courses in the ID curriculum of your institution do you expect will best prepare you for work in the ID field after graduation?
Why do you think this is true?
How important is evaluation in the ID process?
When should evaluation be conducted in the ID process?
Which courses in the ID curriculum of your institution have best prepared you to conduct formative evaluation?
Which course activities provided the opportunity to demonstrate mastery of this skill?
Which courses in the ID curriculum of your institution have best prepared you to conduct summative evaluation?
Which course activities provided the opportunity to demonstrate mastery of this skill?
Which courses in the ID curriculum of your institution have best prepared you to conduct confirmative evaluation?
Which course activities provided the opportunity to demonstrate mastery of this skill?
Rights and permissions
About this article
Cite this article
DeVaughn, P., Stefaniak, J. An exploration of how learning design and educational technology programs prepare instructional designers to evaluate in practice. Education Tech Research Dev 68, 3299–3326 (2020). https://doi.org/10.1007/s11423-020-09823-z
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-020-09823-z