Educational Technology Research and Development

, Volume 59, Issue 6, pp 885–907 | Cite as

How do instructional designers evaluate? A qualitative study of evaluation in practice

  • David D. Williams
  • Joseph B. South
  • Stephen C. Yanchar
  • Brent G. Wilson
  • Stephanie Allen
Development Article

Abstract

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.

Keywords

Evaluation Instructional design Applications Qualitative inquiry 

References

  1. Alkin, M. C. (1991). Evaluation theory development: II. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: At quarter century, ninetieth yearbook of the National Society for the Study of education, Part II. Chicago: University of Chicago Press.Google Scholar
  2. Bamberger, M., Rugh, J., & Mabry, L. (2006). RealWorld evaluation: Working under budget, time, data, and political constraints. Thousand Oaks, CA: Sage.Google Scholar
  3. Bichelmeyer, B., Boling, E., & Gibbons, A. S. (2006). Instructional design and technology models: Their impact on research and teaching in instructional design and technology. In M. Orey, V. J. McClendon, & R. M. Branch (Eds.), Educational media and technology yearbook (Vol. 31, pp. 33–73). Littleton, CO: Libraries Unlimited, Inc.Google Scholar
  4. Braden, R. (1996). The case for linear instructional design and development: A commentary on models, challenges, and myths. Educational Technology, 36(2), 5–23.Google Scholar
  5. Brinkerhoff, R. O. (2006). Telling training’s story: Evaluation made simple, credible, and effective. San Francisco, CA: Barrett-Koehler Publishers.Google Scholar
  6. Christensen, T. K., & Osguthorpe, R. T. (2004). How do instructional design practitioners make instructional-strategy decisions? Performance Improvement Quarterly, 17(3), 45–65.CrossRefGoogle Scholar
  7. Cox, S., & Osguthorpe, R. T. (2003). How do instructional design professionals spend their time? TechTrends, 47(3), 29, 45–47.Google Scholar
  8. Crawford, C. (2004). Non-linear instructional design model: Eternal, synergistic design and development. British Journal of Educational Technology, 35, 413–420.CrossRefGoogle Scholar
  9. Ely, D. P., & Plomp, T. (Eds.). (2001). Classic writings on instructional technology (Vol. 2). Westport, CT: Libraries Unlimited.Google Scholar
  10. Ertmer, P. A., Stepich, D. A., York, C. S., Stickman, A., Wu, X., Zurek, S., et al. (2008). How instructional design experts use knowledge and experience to solve ill-structured problems. Performance Improvement Quarterly, 21(1), 17–42.CrossRefGoogle Scholar
  11. Fitzpatrick, J. L., Sanders, J. R., & Worthen, B. R. (2011). Program evaluation: Alternative approaches and practical guidelines (4th ed.). Upper Saddle River, NJ: Pearson Education, Inc.Google Scholar
  12. Fleming, V., Gaidys, U., & Robb, Y. (2003). Hermeneutic research in nursing: Developing a Gadamerian-based research method. Nursing Inquiry, 10, 113–120.CrossRefGoogle Scholar
  13. Giorgi, A. P., & Giorgi, B. M. (2003). The descriptive phenomenological psychological method. In P. M. Camic, J. E. Rhodes, & L. Yardley (Eds.), Qualitative research in psychology: Expanding perspectives in methodology and design (pp. 243–273). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  14. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, CA: Sage.Google Scholar
  15. Hardré, P. L., Ge, X., & Thomas, M. K. (2006). An investigation of development toward instructional design expertise. Performance Improvement Quarterly, 19(4), 63–90.CrossRefGoogle Scholar
  16. Heidegger, M. (1962). Being and time. New York: Harper & Row. (original work published 1927).Google Scholar
  17. Kerr, S. T. (1983). Inside the black box: Making design decisions for instruction. British Journal of Educational Technology, 14, 45–58.CrossRefGoogle Scholar
  18. Kirkpatrick, D. (1996). Great ideas revisited: Techniques for evaluating training programs. Training and Development, 50, 54–59.Google Scholar
  19. Kirschner, P., Carr, C., van Merrienboer, J., & Sloep, P. (2002). How expert designers design. Performance Improvement Quarterly, 15, 86–104.CrossRefGoogle Scholar
  20. Klimczack, A. K., & Wedman, J. F. (1997). Instructional design project success factors: An empirical basis. Educational Technology Research and Development, 45(2), 75–83.CrossRefGoogle Scholar
  21. Kvale, S. (1996). InterViews: An introduction to qualitative research interviewing. Thousand Oaks, CA: Sage Publications.Google Scholar
  22. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage.Google Scholar
  23. Liu, M., Gibby, S., Quiros, O., & Demps, E. (2002). Challenges of being an instructional designer for new media development: A view from practitioners. Journal of Educational Multimedia and Hypermedia, 11(3), 195–219.Google Scholar
  24. Molenda, M. (2003). In search of the elusive ADDIE model. Performance Improvement, 42(5), 34.CrossRefGoogle Scholar
  25. Morell, J. A. (2010). Evaluation in the face of uncertainty: Anticipating surprise and responding to the inevitable. New York: Guilford Press.Google Scholar
  26. Packer, M. J. (1985). Hermeneutic inquiry in the study of human conduct. American Psychologist, 40, 1081–1093.CrossRefGoogle Scholar
  27. Paine, M., & Masie, E. (Eds.) (2010). The Masie Center’s learning perspectives 2010. Saratoga Springs, NY: The MASIE Center & The Learning Consortium. http://www.learning2010.com/ebook.
  28. Patton, M. Q. (2002). Qualitative research and evaluation methods (3rd ed.). Thousand Oaks, CA: Sage.Google Scholar
  29. Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage.Google Scholar
  30. Patton, M. Q. (2011). Developmental evaluation: Applying complexity concepts to enhance innovation and use. New York: Guilford Press.Google Scholar
  31. Perez, R. S., & Emery, C. D. (1995). Designer thinking: How novices and experts think about instructional design. Performance Improvement Quarterly, 8(3), 80–94.CrossRefGoogle Scholar
  32. Pieters, J. M., & Bergman, R. (1995). The empirical basis of designing instruction. Performance Improvement Quarterly, 8(3), 118.CrossRefGoogle Scholar
  33. Reeves, T. C. (1992). Evaluating interactive media. Educational Technology, 32(5), 47–53.Google Scholar
  34. Reeves, T. C., & Hedberg, J. G. (2003). Interactive learning systems evaluation. Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
  35. Rowland, G. (1992). What do designers actually do? An initial investigation of expert practice. Performance Improvement Quarterly, 5(2), 65–86.CrossRefGoogle Scholar
  36. Sanders, J. R. (1994). The program evaluation standards: How to assess evaluations of educational programs. Thousand Oaks, CA: Sage.Google Scholar
  37. Schön, D. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.Google Scholar
  38. Schwier, R. A., Campbell, K., & Kenny, R. (2004). Instructional designers’ observations about identity, communities of practice, and change agency. Australasian Journal of Educational Technology, 20(1), 69–100.Google Scholar
  39. Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, et al. (Eds.), Perspectives of curriculum evaluation (AERA monograph series on curriculum evaluation) (Vol. 1, pp. 39–83). Chicago: Rand McNally.Google Scholar
  40. Scriven, M. (1991). Beyond formative and summative evaluation. In M. W. McLaughlin & D. C. Phillips (Eds.), Evaluation and education: A quarter century. Chicago: University of Chicago Press.Google Scholar
  41. Seidman, I. (1998). Interviewing as qualitative research: A guide for researchers in education and the social sciences (2nd ed.). New York: Teachers College Press.Google Scholar
  42. Senge, P. M. (1990). The fifth discipline: The art & practice of the learning organization. New York: Currency Doubleday.Google Scholar
  43. Smith, J. A., & Osborn, M. (2003). Interpretative phenomenological analysis. In J. A. Smith (Ed.), Qualitative psychology: A practical guide to research methods (pp. 51–80). Thousand Oaks, CA: Sage.Google Scholar
  44. Smith, K. M., & Boling, E. (2009). What do we make of design? Design as a concept in educational technology. Educational Technology, 49(4), 3–17.Google Scholar
  45. Spradley, J. P. (1979). The ethnographic interview. New York: Holt Rinehart, & Winston.Google Scholar
  46. Stufflebeam, D. L. (2000). The CIPP model for evaluation. In D. L. Shufflebeam, G. F. Madaus, & T. Kelleghan (Eds.), Evaluation models: Viewpoints on educational and human services evaluation (2nd ed., pp. 274–317) Dublin: The Educational Research Centre.Google Scholar
  47. Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey-Bass.Google Scholar
  48. Taylor, C. (1985). Philosophy and the human sciences: Philosophical papers (Vol. 2). New York: Cambridge University Press.Google Scholar
  49. Visscher-Voerman, I., & Gustafson, K. L. (2004). Paradigms in the theory and practice of education and training design. Educational Technology Research and Development, 52(2), 69–91.CrossRefGoogle Scholar
  50. Wedman, J., & Tessmer, M. (1993). Instructional designers’ decisions and priorities: A survey of design practice. Performance Improvement Quarterly, 6(2), 43–57.CrossRefGoogle Scholar
  51. Westerman, M. A. (2004). Theory and research on practices, theory and research as practices: Hermeneutics and psychological inquiry. Journal of Theoretical and Philosophical Psychology, 24, 123–156.CrossRefGoogle Scholar
  52. Weston, C., McAlpine, L., & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development, 43(3), 29–48.CrossRefGoogle Scholar
  53. Williams, D. D. (2006). Measurement and assessment supporting evaluation in online settings. In D. D. Williams, M. Hricko, & S. Howell (Eds.), Online assessment, measurement and evaluation: Emerging practices. Hershey, PA: Idea Group, Inc.Google Scholar
  54. Williams, D. D., & Graham, C. R. (2010). Evaluating E-learning. In P. P. Peterson, E. Baker, & B. McGaw (Editors-in-Chief), International encyclopedia of education (3rd ed., pp. 530–538). Oxford: Elsevier.Google Scholar
  55. Winer, L. R., & Vázquez-Abad, J. (1995). The present and future of ID practice. Performance Improvement Quarterly, 8(3), 55–67.CrossRefGoogle Scholar
  56. Yanchar, S. C., South, J. B., Williams, D. D., Allen, S., & Wilson, B. G. (2010). Struggling with theory? A Qualitative investigation of conceptual tool use in instructional design. Educational Technology Research and Development, 58(1), 39–60.CrossRefGoogle Scholar
  57. Yarbrough, D. B., Shulha, L. M., Hopson, R. K., & Caruthers, F. A. (2011). The program evaluation standards: A guide for evaluators and evaluation users. Thousand Oaks, CA: Sage.Google Scholar

Copyright information

© Association for Educational Communications and Technology 2011

Authors and Affiliations

  • David D. Williams
    • 1
  • Joseph B. South
    • 2
  • Stephen C. Yanchar
    • 3
  • Brent G. Wilson
    • 4
  • Stephanie Allen
    • 5
  1. 1.Department of Instructional Psychology and TechnologyBrigham Young UniversityProvoUSA
  2. 2.Middlebury Interactive LanguagesProvoUSA
  3. 3.Brigham Young UniversityProvoUSA
  4. 4.University of ColoradoDenverUSA
  5. 5.Church of Jesus Christ of Latter-Day SaintsSalt Lake CityUSA

Personalised recommendations