Skip to main content

Advertisement

Log in

An exploration of how learning design and educational technology programs prepare instructional designers to evaluate in practice

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

The purpose of this qualitative study was to examine how learning design and educational technology degree programs prepare students to evaluate in practice. This study involved the curricular mapping of 16 graduate instructional design programs and 29 semi-structured interviews with program faculty and recent postgraduates. Based on information shared during the interviews, this study produced nine themes according to three metathemes: (a) the role of problem-solving in evaluation, (b) alignment of evaluation activities in instructional design, and (c) relevancy of evaluation in instructional design. The findings revealed that evaluation was not prioritized for most programs, due to lack of time, client resources, employer lack of interest, and limited faculty experience in evaluation. Other recommendations are provided for how to enhance evaluative practices in instructional design coursework.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Armstrong, A. (2004). Instructional design in the real world: A view from the trenches. Education Technology & Society, 7, 128–129.

    Google Scholar 

  • Bowman, S., Armstrong, A., Lane, C., & Lane, M. (2015). Evaluation in instructional Design practice: A view from the stakeholders (Doctoral dissertation). Available from ProQuest Dissertations and Theses database.

  • Chen, W., Moore, J. L., & Vo, N. (2012). Formative evaluation with novice designers: Two case studies within an online multimedia development course. International Journal of Instructional Media, 39(2), 95–111.

    Google Scholar 

  • Creswell, J. W. (2015). Educational research: Planning, conducting, and evaluating quantitative and qualitative research. Upper Saddle River, NJ: Pearson Education.

    Google Scholar 

  • Dabbagh, N., & English, M. (2015). Using student self-ratings to assess the alignment of instructional design competencies and courses in a graduate program. Tech Trends, 59(4), 22–31.

    Google Scholar 

  • DeVaughn, P., & Stefaniak, J. (2020). An exploration of the challenges instructional designers encounter while conducting evaluations. Performance Improvement Quarterly., 21(1), 17–42.

    Google Scholar 

  • Dooley, K. E., Lindner, J. R., Telg, R. W., Irani, T., Moore, L., & Lundy, L. (2007). Roadmap to measuring distance education instructional design competencies. Quarterly Review of Distance Education, 8(2), 151.

    Google Scholar 

  • Endsley, M. R. (2018). Expertise and situation awareness. In K. A. Ericsson, R. R. Hofman, A. Kozbelt, & A. M. Williams (Eds.), The Cambridge handbook of expertise and expert performance (2nd ed., pp. 633–651). Cambridge: Cambridge University Press.

    Google Scholar 

  • Everhart, D., Bushway, D., & Schejbal, D. (2016). Communicating the value of competencies. Washington, D.C.: American Council on Education. https://www.acenet.edu/news-room/Documents/Communicating-the-Value-of-Competencies.pdf

  • Ericsson, K. A. (2018). The differential influence of experience, practice, and deliberate practice on the development of superior individual performance of experts. In K. A. Ericsson, R. R. Hoffman, A. Kozbelt, & A. M. Williams (Eds.), The Cambridge handbook of expertise and expert performance (2nd ed., pp. 745–769). Cambridge: Cambridge University Press.

    Google Scholar 

  • Fadde, P. J. (2009). Instructional design for advanced learners: Training recognition skills to hasten expertise. Educational Technology Research and Development, 57(3), 359–376.

    Google Scholar 

  • Giberson, T. R., Tracey, M. W., & Harris, M. T. (2006). Confirmative evaluation of training outcomes. Performance Improvement Quarterly, 19(4), 43–61.

    Google Scholar 

  • Guerra-Lopez, I. (2007). Evaluating impact: Evaluation and continual improvement for performance improvement practitioners. Champaign, IL: Human Resource Development Press.

    Google Scholar 

  • Guerra-Lopez, I. J. (2008). Performance evaluation: Proven approaches for improving program and organizational performance. San Francisco, CA: Wiley.

    Google Scholar 

  • Harwood, T., & Garry, T. (2003). An overview of content analysis. The Marketing Review, 3(4), 479–498.

    Google Scholar 

  • Hays, D., & Singh, A. (2012). Qualitative inquiry in clinical and educational settings. New York: Guilford Press.

    Google Scholar 

  • Herrington, A., & Herrington, J. (Eds.). (2008). What is an authentic learning environment? in authentic learning environments in higher education (pp. 68–77). London: Information Science Publishing.

  • Herrington, J., Oliver, R., & Reeves, T. C. (2003). Patterns of engagement in authentic online learning environments. Australian Journal of Educational Technology, 19(1), 59–71.

    Google Scholar 

  • Klein, J. D., & Fox, E. J. (2004). Performance improvement competencies for instructional technologists. Tech Trends, 48(2), 22–25.

    Google Scholar 

  • Klein, J. D., & Jun, S. (2014). Skills for instructional design professionals. Performance Improvement, 53(2), 41–46.

    Google Scholar 

  • Klein, J. D., & Kelly, W. Q. (2018). Competencies for instructional designers: A view from employers. Performance Improvement Quarterly, 31(3), 225–247.

    Google Scholar 

  • Koszalka, T. A., Russ-Eft, D. F., & Reiser, R. (2013). Instructional designer competencies: The standards (4th ed.). Information Age Publishing.

  • Larson, M. (2004). Instructional design career environments: Survey of the alignment of preparation and practice. Tech Trends, 49(6), 22–34.

    Google Scholar 

  • Larson, M. B., & Lockee, B. B. (2009). Instructional design practice: Career environments, job roles, and a climate of change. Performance Improvement Quarterly, 17(1), 22–40.

    Google Scholar 

  • Le Maistre, C. (1998). What is an expert instructional designer? Evidence of expert performance during formative evaluation. Educational Technology Research and Development, 46(3), 21–36.

    Google Scholar 

  • Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Newbury Park, CA: Sage.

    Google Scholar 

  • Marshall, J., & Rossett, A. (2014). Perceptions of barriers to the evaluation of workplace learning programs. Performance Improvement Quarterly, 27(3), 7–26.

    Google Scholar 

  • Moller, L., & Mallin, P. (1996). Evaluation practices of instructional designers and organizational supports and barriers. Performance Improvement Quarterly, 9(4), 82–92.

    Google Scholar 

  • Parhar, M., & Mishra, S. (2000). Competencies for Web-based instructional designers. Indian Journal of Open Learning9(3), 415–422.

    Google Scholar 

  • Richardson, J. C., Ashby, I., Alshammari, A. N., Cheng, Z., Johnson, B. S., Krause, T. S., ... & Wang, H. (2019). Faculty and instructional designers on building successful collaborative relationships. Educational Technology Research and Development67(4), 855–880.

  • Richey, R. C., Fields, D. C., & Foxon, M. (2001). Instructional design competencies: Thestandards (3rd ed.). Syracuse, NY: ERIC Clearinghouse on Information & Technology.

    Google Scholar 

  • Richey, R. C., & Klein, J. D. (2005). Developmental research methods: Creating knowledge from instructional design and development practice. Journal of Computing in Higher Education, 16(2), 23–38.

    Google Scholar 

  • Ritzhaupt, A. D., & Kumar, S. (2015). Knowledge and skills needed by instructional designers in higher education. Performance Improvement Quarterly, 28(3), 51–69.

    Google Scholar 

  • Ritzhaupt, A. D., & Martin, F. (2014). Development and validation of the educational technologist multimedia competency survey. Educational Technology Research and Development, 62(1), 13–33.

    Google Scholar 

  • Ritzhaupt, A. D., Martin, F., Pastore, R., & Kang, Y. (2018). Development and validation of the educational technologist competencies survey (ETCS): Knowledge, skills, and abilities. Journal of Computing in Higher Education, 30(1), 3–33.

    Google Scholar 

  • Rozitis, C. P. (2017). Instructional design competencies for online high school teachers modifying their own courses. Tech Trends, 61(5), 428–437.

    Google Scholar 

  • Rossett, A., & Sheldon, K. (2001). Beyond the podium: Delivering training and performance to a digital world. Education & Training, 44(2/3), 104–108.

    Google Scholar 

  • Scriven, M. (1991). Evaluation thesaurus (4th ed.). Thousand Oak, CA: Sage.

    Google Scholar 

  • Seels, B. B., & Richey, R. C. (1994). Instructional technology: The definition and domains of the field. Bloomington, IN: Association for Educational Communication and Technology.

    Google Scholar 

  • Slagter van Tryon, P. J., McDonald, J., & Hirumi, A. (2018). Preparing the next generation of instructional designers: A cross-institution faculty collaboration. Journal of Computing in Higher Education, 30(1), 125–153.

    Google Scholar 

  • Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research & Evaluation, 7(17), 1–6.

    Google Scholar 

  • Sugar, W. (2014). Development and formative evaluation of multimedia studies for instructional design and technology students. Tech Trends, 58(5), 36–52.

    Google Scholar 

  • Sugar, W. (2014b). Studies of ID practices: A review and synthesis of research on ID current practices. Springer International Publishing. ProQuest Ebook Central.

  • Sugar, W., Hoard, B., Brown, A., & Daniels, L. (2012). Identifying multimedia production competencies and skills of instructional design and technology professionals: An analysis of recent job postings. Journal Educational Technology Systems, 40(3), 227–249.

    Google Scholar 

  • Sugar, W., & Luterbach, K. J. (2016). Using critical incidents of instructional design and multimedia production activities to investigate instructional designers' current practices and roles. Educational Technology Research and Development, 64(2), 285–312.

    Google Scholar 

  • Tessmer, M. (1993). Planning and conducting formative evauations. London: Kogan Page.

    Google Scholar 

  • Tessmer, M. (1994). Formative evaluation alternatives. Performance Improvement Quarterly, 7(1), 3–18.

    Google Scholar 

  • Thach, E. C., & Murphy, K. L. (1995). Competencies for distance education professionals. Educational Technology Research and Development, 43(1), 57–79.

    Google Scholar 

  • Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner's mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13.

    Google Scholar 

  • Villachica, S. W., Marker, A., & Taylor, K. (2010). But what do they really expect? Employer perceptions of the skills of entry-level instructional designers. Performance Improvement Quarterly, 22, 33–51.

    Google Scholar 

  • Visscher-Voerman, I. (2017). Necessary ingredients for the education of designers. In A. Carr-Chellman & G. Rowland (Eds.), Issues in technology, learning, and instructional design: Classic and contemporary dialogues (pp. 73–80). New York, NY: Routledge.

    Google Scholar 

  • Wang, G. G., & Wilcox, D. (2006). Training evaluation: knowing more than is practiced. Advances in Developing Human Resources, 8(4), 528–539.

    Google Scholar 

  • Wedman, J., & Tessmer, M. (1993). Instructional designers' decisions and priorities: A survey of design practice. Performance Improvement Quarterly, 6(2), 43–57.

    Google Scholar 

  • Weinberg, J. B., & Stephen, M. L. (2002). Participatory design in a human-computer interaction course: Teaching ethnography methods to computer scientists. In Proceedings of the 3rd SIGCSE technical symposium on Computer science education (pp. 237–241). New York, NY: ACM

  • Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, 43(3), 29–48.

    Google Scholar 

  • Williams, P. E. (2003). Roles and competencies for distance education programs in higher education institutions. The American Journal of Distance Education, 17(1), 45–57.

    Google Scholar 

  • Williams, D. D., South, J. B., Yanchar, S. C., Wilson, B. G., & Allen, S. (2011). How do instructional designers evaluate? A qualitative study of evaluation in practice. Education Technology Research and Development, 59, 885–907.

    Google Scholar 

  • York, C. S., & Ertmer, P. A. (2011). Towards an understanding of instructional design heuristics: An exploratory Delphi study. Educational Technology Research and Development, 59(6), 841–863.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jill Stefaniak.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendices

Appendix 1

Interview questions (faculty)

Can you provide a brief description explaining your job responsibilities?

What skills should instructional design (ID) students possess to successfully demonstrate understanding of ID principles?

What activities provide the needed practice to ensure mastery of these skills?

How do you assess the students’ level of competence in these skills?

Which courses in the ID curriculum of your institution do you expect will best prepare instructional design students for work in the ID field after graduation?

What data could you share to support this expectation?

How important is evaluation in the ID process?

When should evaluation be conducted in the ID process?

Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct formative evaluation?

What course activities provide the opportunity to demonstrate mastery of this skill?

Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct summative evaluation?

What course activities provide the opportunity to demonstrate mastery of this skill?

Which courses in the ID curriculum of your institution do you expect best prepare instructional designers to conduct confirmative evaluation?

What course activities provide the opportunity to demonstrate mastery of this skill?

Appendix 2

Interview questions (post-graduates)

Can you provide a brief description explaining your job responsibilities?

What skills should instructional design (ID) students possess to successfully demonstrate understanding of ID principles?

What activities provide the needed practice to ensure mastery of these skills?

How should your level of competence in these skills be assessed?

Which courses in the ID curriculum of your institution do you expect will best prepare you for work in the ID field after graduation?

Why do you think this is true?

How important is evaluation in the ID process?

When should evaluation be conducted in the ID process?

Which courses in the ID curriculum of your institution have best prepared you to conduct formative evaluation?

Which course activities provided the opportunity to demonstrate mastery of this skill?

Which courses in the ID curriculum of your institution have best prepared you to conduct summative evaluation?

Which course activities provided the opportunity to demonstrate mastery of this skill?

Which courses in the ID curriculum of your institution have best prepared you to conduct confirmative evaluation?

Which course activities provided the opportunity to demonstrate mastery of this skill?

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

DeVaughn, P., Stefaniak, J. An exploration of how learning design and educational technology programs prepare instructional designers to evaluate in practice. Education Tech Research Dev 68, 3299–3326 (2020). https://doi.org/10.1007/s11423-020-09823-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-020-09823-z

Keywords

Navigation