Skip to main content
SpringerLink
Log in

How do instructional designers evaluate? A qualitative study of evaluation in practice

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Alkin, M. C. (1991). Evaluation theory development: II. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century, ninetieth yearbook of the National Society for the Study of education, Part II. Chicago: University of Chicago Press.

    Google Scholar 

  • Bamberger, M., Rugh, J., & Mabry, L. (2006). RealWorld evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Bichelmeyer, B., Boling, E., & Gibbons, A. S. (2006). Instructional design and technology models: Their impact on research and teaching in instructional design and technology. In M. Orey, V. J. McClendon, & R. M. Branch (Eds.), Educational media and technology yearbook (Vol. 31, pp. 33–73). Littleton, CO: Libraries Unlimited, Inc.

  • Braden, R. (1996). The case for linear instructional design and development: A commentary on models, challenges, and myths. Educational Technology, 36(2), 5–23.

    Google Scholar 

  • Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. San Francisco, CA: Barrett-Koehler Publishers.

    Google Scholar 

  • Christensen, T. K., & Osguthorpe, R. T. (2004). How do instructional design practitioners make instructional-strategy decisions? Performance Improvement Quarterly, 17(3), 45–65.

    Article  Google Scholar 

  • Cox, S., & Osguthorpe, R. T. (2003). How do instructional design professionals spend their time? TechTrends, 47(3), 29, 45–47.

  • Crawford, C. (2004). Non-linear instructional design model: Eternal, synergistic design and development. British Journal of Educational Technology, 35, 413–420.

    Article  Google Scholar 

  • Ely, D. P., & Plomp, T. (Eds.). (2001). Classic writings on instructional technology (Vol. 2). Westport, CT: Libraries Unlimited.

    Google Scholar 

  • Ertmer, P. A., Stepich, D. A., York, C. S., Stickman, A., Wu, X., Zurek, S., et al. (2008). How instructional design experts use knowledge and experience to solve ill-structured problems. Performance Improvement Quarterly, 21(1), 17–42.

    Article  Google Scholar 

  • Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education, Inc.

    Google Scholar 

  • Fleming, V., Gaidys, U., & Robb, Y. (2003). Hermeneutic research in nursing: Developing a Gadamerian-based research method. Nursing Inquiry, 10, 113–120.

    Article  Google Scholar 

  • Giorgi, A. P., & Giorgi, B. M. (2003). The descriptive phenomenological psychological method. In P. M. Camic, J. E. Rhodes, & L. Yardley (Eds.), Qualitative research in psychology: Expanding perspectives in methodology and design (pp. 243–273). Washington, DC: American Psychological Association.

    Chapter  Google Scholar 

  • Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.

    Google Scholar 

  • Hardré, P. L., Ge, X., & Thomas, M. K. (2006). An investigation of development toward instructional design expertise. Performance Improvement Quarterly, 19(4), 63–90.

    Article  Google Scholar 

  • Heidegger, M. (1962). Being and time. New York: Harper & Row. (original work published 1927).

  • Kerr, S. T. (1983). Inside the black box: Making design decisions for instruction. British Journal of Educational Technology, 14, 45–58.

    Article  Google Scholar 

  • Kirkpatrick, D. (1996). Great ideas revisited: Techniques for evaluating training programs. Training and Development, 50, 54–59.

    Google Scholar 

  • Kirschner, P., Carr, C., van Merrienboer, J., & Sloep, P. (2002). How expert designers design. Performance Improvement Quarterly, 15, 86–104.

    Article  Google Scholar 

  • Klimczack, A. K., & Wedman, J. F. (1997). Instructional design project success factors: An empirical basis. Educational Technology Research and Development, 45(2), 75–83.

    Article  Google Scholar 

  • Kvale, S. (1996). InterViews: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.

    Google Scholar 

  • Liu, M., Gibby, S., Quiros, O., & Demps, E. (2002). Challenges of being an instructional designer for new media development: A view from practitioners. Journal of Educational Multimedia and Hypermedia, 11(3), 195–219.

    Google Scholar 

  • Molenda, M. (2003). In search of the elusive ADDIE model. Performance Improvement, 42(5), 34.

    Article  Google Scholar 

  • Morell, J. A. (2010). Evaluation in the face of uncertainty: Anticipating surprise and responding to the inevitable. New York: Guilford Press.

    Google Scholar 

  • Packer, M. J. (1985). Hermeneutic inquiry in the study of human conduct. American Psychologist, 40, 1081–1093.

    Article  Google Scholar 

  • Paine, M., & Masie, E. (Eds.) (2010). The Masie Center’s learning perspectives 2010. Saratoga Springs, NY: The MASIE Center & The Learning Consortium. http://www.learning2010.com/ebook.

  • Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.

    Google Scholar 

  • Perez, R. S., & Emery, C. D. (1995). Designer thinking: How novices and experts think about instructional design. Performance Improvement Quarterly, 8(3), 80–94.

    Article  Google Scholar 

  • Pieters, J. M., & Bergman, R. (1995). The empirical basis of designing instruction. Performance Improvement Quarterly, 8(3), 118.

    Article  Google Scholar 

  • Reeves, T. C. (1992). Evaluating interactive media. Educational Technology, 32(5), 47–53.

    Google Scholar 

  • Reeves, T. C., & Hedberg, J. G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.

    Google Scholar 

  • Rowland, G. (1992). What do designers actually do? An initial investigation of expert practice. Performance Improvement Quarterly, 5(2), 65–86.

    Article  Google Scholar 

  • Sanders, J. R. (1994). The program evaluation standards: How to assess evaluations of educational programs. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Schön, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

    Google Scholar 

  • Schwier, R. A., Campbell, K., & Kenny, R. (2004). Instructional designers’ observations about identity, communities of practice, and change agency. Australasian Journal of Educational Technology, 20(1), 69–100.

    Google Scholar 

  • Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, et al. (Eds.), Perspectives of curriculum evaluation (AERA monograph series on curriculum evaluation) (Vol. 1, pp. 39–83). Chicago: Rand McNally.

  • Scriven, M. (1991). Beyond formative and summative evaluation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: A quarter century. Chicago: University of Chicago Press.

    Google Scholar 

  • Seidman, I. (1998). Interviewing as qualitative research: A guide for researchers in education and the social sciences (2nd ed.). New York: Teachers College Press.

    Google Scholar 

  • Senge, P. M. (1990). The fifth discipline: The art & practice of the learning organization. New York: Currency Doubleday.

    Google Scholar 

  • Smith, J. A., & Osborn, M. (2003). Interpretative phenomenological analysis. In J. A. Smith (Ed.), Qualitative psychology: A practical guide to research methods (pp. 51–80). Thousand Oaks, CA: Sage.

  • Smith, K. M., & Boling, E. (2009). What do we make of design? Design as a concept in educational technology. Educational Technology, 49(4), 3–17.

    Google Scholar 

  • Spradley, J. P. (1979). The ethnographic interview. New York: Holt Rinehart, & Winston.

    Google Scholar 

  • Stufflebeam, D. L. (2000). The CIPP model for evaluation. In D. L. Shufflebeam, G. F. Madaus, & T. Kelleghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (2nd ed., pp. 274–317) Dublin: The Educational Research Centre.

  • Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey-Bass.

    Google Scholar 

  • Taylor, C. (1985). Philosophy and the human sciences: Philosophical papers (Vol. 2). New York: Cambridge University Press.

    Google Scholar 

  • Visscher-Voerman, I., & Gustafson, K. L. (2004). Paradigms in the theory and practice of education and training design. Educational Technology Research and Development, 52(2), 69–91.

    Article  Google Scholar 

  • Wedman, J., & Tessmer, M. (1993). Instructional designers’ decisions and priorities: A survey of design practice. Performance Improvement Quarterly, 6(2), 43–57.

    Article  Google Scholar 

  • Westerman, M. A. (2004). Theory and research on practices, theory and research as practices: Hermeneutics and psychological inquiry. Journal of Theoretical and Philosophical Psychology, 24, 123–156.

    Article  Google Scholar 

  • Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, 43(3), 29–48.

    Article  Google Scholar 

  • Williams, D. D. (2006). Measurement and assessment supporting evaluation in online settings. In D. D. Williams, M. Hricko, & S. Howell (Eds.), Online assessment, measurement and evaluation: Emerging practices. Hershey, PA: Idea Group, Inc.

    Google Scholar 

  • Williams, D. D., & Graham, C. R. (2010). Evaluating E-learning. In P. P. Peterson, E. Baker, & B. McGaw (Editors-in-Chief), International encyclopedia of education (3rd ed., pp. 530–538). Oxford: Elsevier.

  • Winer, L. R., & Vázquez-Abad, J. (1995). The present and future of ID practice. Performance Improvement Quarterly, 8(3), 55–67.

    Article  Google Scholar 

  • Yanchar, S. C., South, J. B., Williams, D. D., Allen, S., & Wilson, B. G. (2010). Struggling with theory? A Qualitative investigation of conceptual tool use in instructional design. Educational Technology Research and Development, 58(1), 39–60.

    Article  Google Scholar 

  • Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users. Thousand Oaks, CA: Sage.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Instructional Psychology and Technology, Brigham Young University, 150G MCKB, Provo, UT, 84602, USA

    David D. Williams

  2. Middlebury Interactive Languages, Provo, UT, USA

    Joseph B. South

  3. Brigham Young University, Provo, UT, USA

    Stephen C. Yanchar

  4. University of Colorado, Denver, CO, USA

    Brent G. Wilson

  5. Church of Jesus Christ of Latter-Day Saints, Salt Lake City, UT, USA

    Stephanie Allen

Authors
  1. David D. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joseph B. South
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen C. Yanchar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brent G. Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephanie Allen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David D. Williams.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Williams, D.D., South, J.B., Yanchar, S.C. et al. How do instructional designers evaluate? A qualitative study of evaluation in practice. Education Tech Research Dev 59, 885–907 (2011). https://doi.org/10.1007/s11423-011-9211-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-011-9211-8

Keywords

Search

Navigation