Instructional Science

, Volume 25, Issue 5, pp 369–386 | Cite as

The influence of participants in formative evaluation on the improvement of learning from written instructional materials

  • Cynthia Weston
  • Cathrine Le Maistre
  • Lynn Mcalpine
  • Tino Bordonaro

Abstract

Formative evaluation in instructional design is widely advocated as a means to improve instructional materials through tryouts with learners and experts and revision based on this feedback. Research on formative evaluation has sought to prove its effectiveness and to provide guidance as to which methods of collecting feedback and which sources of feedback are most effective in improving learning. It is difficult to determine from the research whether aspects of tryout or of revision are responsible for improved learning. This paper describes a systematic investigation of both tryout and revision to more clearly determine the influence of different formative evaluation participants (experts, learners and revisers) on the improvement of learning from written instructional materials.

It was found that revised versions incorporating learner feedback had the most impact on improving learning from the materials. It was also found that revisers have a far more powerful impact on formative evaluation outcomes than was previously supposed, in terms of how they mediate and incorporate the feedback they are given. The results refute contentions that any revision is better than none since versions revised without learner feedback did not improve learning. Implications are discussed.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Andrews, D.H. & Goodson, L.A. (1980). A comparative analysis of models of instructional design. Instructional Technology 3(4): 2–16.Google Scholar
  2. Baker, E.L. (1970). Generalizability of rules for empirical revision. Audiovisual Communication Review 18(3): 300–305.Google Scholar
  3. Baker, E.L. & Alkin, M.C. (1973). Formative evaluation of instructional development. AV Communication Review 21(4): 389–418.Google Scholar
  4. Bordonaro, T. (1993). A Comparison of the Effectiveness, Cost and Efficiency of Four Formative Evaluation Conditions. Unpublished master's thesis, McGill University, Montreal, Canada.Google Scholar
  5. Briggs, L. (1970). Handbook of Procedures for the Design of Instruction. Pittsburgh, PA: American Institutes for Research.Google Scholar
  6. Cambre, M. (1981). Historical overview of formative evaluation of instructional media products. Educational Communications and Technology Journal 29(1): 1–25.Google Scholar
  7. Collis, B.A. (1993). Evaluating instructional applications of telecommunications in distance education. Educational and Training Technology International 30(3): 266–274.Google Scholar
  8. Comber, T. (1995). Building Usable Web Pages: An HCI Perspective. [On-line]. Available: http://www.scu.edu.au/ausweb95/papers/hypertext/comber.Google Scholar
  9. Crooks, B. & Lamy, M.N. (1995). Using combinations of video, audio and print to teach French at a distance: A case study of the formative evaluation cycle. European Journal of Psychology of Education 10(2): 131–144.Google Scholar
  10. Davidove, E.A. & Reiser, R.A. (1991). Comparative acceptability and effectiveness of teacher-revised and designer-revised instruction. Educational Technology Research and Development 39(2): 29–38.Google Scholar
  11. De Ketele, J.M. & Roegiers, X. (1993). Méthodologie du receuil d'informations. Bruxelles: De Boeck-Wesmael.Google Scholar
  12. Dick, W. & Carey, L. (1990). The Systematic Design of Instruction (3rd ed.). Glenview, IL: Scott, Foresman & Company.Google Scholar
  13. Dupont, D. & Stolovitch, H.D. (1983). The effects of a systematic revision model on revisers in terms of student outcomes. NSPI Journal March: 33–37.Google Scholar
  14. Ericsson, K.A. & Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data (Revised edition). Cambridge, MA: MIT Press.Google Scholar
  15. Flagg, B.N. (1990). Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  16. Golas, K.C. (1983). Formative evaluation effectiveness and cost: Alternative models for evaluating printed instructional materials. Performance and Instruction 22: 17–19.Google Scholar
  17. Gropper, G.L. (1975). Diagnosis and Revision in the Development of Instructional Materials. Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
  18. Gustafson, K.L. (1991). Survey of Instructional Development Models (2nd ed.). Syracuse, NY: Information Resources Publications.Google Scholar
  19. Hayes, J.R. (1989). The Complete Problem Solver (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  20. Jonassen, D.H., ed. (1988). Instructional Designs for Microcomputer Courseware. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  21. Kandaswamy, S., Stolovitch, H.D.& Thiagarajan, S. (1976). Learner verification and revision: An experimental comparison of two methods. Audio-Visual Communication Review 24(3): 316–328.Google Scholar
  22. Komoski, P.K. & Woodward, A. (1985). The continuing need for learner verification and revision of textual material, in D.H. Jonassen, ed., The Technology of Text: Volume 2 (pp. 396–417). Englewood Cliffs, NJ: Educational Technology Publications.Google Scholar
  23. Le Maistre, C. (1991, April). Revision: TheMissing Link. Paper presented at the annual-meeting of the American Educational Research Association, Chicago, IL.Google Scholar
  24. Le Maistre, C. (1994). The Priorities Established Among Data Sources when Instructional Designers Revise Written Materials. Unpublished doctoral dissertation, McGill University, Montreal, Canada.Google Scholar
  25. Le Maistre, C. & Weston, C.B. (1996). The priorities established among data sources when instructional designers revise written materials. Educational Technology Research and Development 44(1): 61–70.Google Scholar
  26. McAlpine, L. (1992). Highlighting formative evaluation: An instructional design model derived from practice. Performance and Instruction Journal 31(10): 16–18.Google Scholar
  27. Melton, R.F. (1995). Developing a formative evaluation system for distance teaching. Open Learning 10(2): 53–57.Google Scholar
  28. Newell, A., (1980). Reasoning, problem-solving, and decision processes: The problem space as a fundamental category, in R. Nickerson, ed., Attention and Performance, Vol. VIII (pp. 693–719). Hillsdale, NJ: Erlbaum.Google Scholar
  29. Newell, A. & Simon, H.A. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  30. Popham, W.J. & Baker, E.L. (1970). Systematic Instruction. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  31. Price, R.V. & Repman, J. (1995). Instructional design for college level courses using instructional television. Journal of Educational Technology Systems 23(3): 251–263.Google Scholar
  32. Rahilly, T.J. (1991). An Analysis of Three Learner-Based Formative Evaluation Conditions. Unpublished master's thesis, McGill University, Montreal, Quebec.Google Scholar
  33. Reigeluth, C.M. (1983). Instructional design: What is it and why is it? in C.M. Reigeluth, ed., Instructional Design Theories and Models: An Overview of the Current Status (pp. 3–36). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  34. Rosen, M.J. (1968). An Experimental Design for Comparing the Effects of Instructional Media Programming Procedures: Subjective vs. Objective Revision Procedures. Palo Alto, CA: American Institutes for Research in the Behavioral Sciences.Google Scholar
  35. Rowland, G. (1993). Designing and instructional design. Educational Technology Research and Development 41(1): 79–91.Google Scholar
  36. Saroyan, A. (1989). The Review Process in Formative Evaluation of Instructional Text: The Role of Content Experts and Instructional Designers. Unpublished doctoral dissertation. Montreal, Canada. McGill University.Google Scholar
  37. Scriven, M. (1967). The methodology of evaluation, in R.W. Tyler et al., eds., Perspectives of Curriculum Evaluation (pp. 39–83) (AERA monograph series on curriculum evaluation, no1). Chicago: Rand McNally.Google Scholar
  38. Shanteau, J. (1992). The psychology of experts: An alternative view, in G. Wright & F. Bolger, eds., Expertise and Decision Support. New York: Plenum Press.Google Scholar
  39. Tessmer, M. (1994). Formative evaluation alternatives. Performance Improvement Quarterly 71(1): 3–18.Google Scholar
  40. Thiagarajan, S. (1991). Formative evaluation and performance technology. Performance Improvement Quarterly 4(2): 22–34.Google Scholar
  41. Tremblay, D. (1994). An Analysis of the Role of Expert Reviewers in Formative Evaluation. Unpublished master's thesis, McGill University, Montreal, Canada.Google Scholar
  42. Volman, C. & De Diana, I. (1993).Courseware evaluation and decision support based oncourseware use. Educational and Training Technology International 30(3): 255–274.Google Scholar
  43. Wager, W.W. (1993). Instructional systems fundamentals: Pressure to change. Educational Technology 33(2): 8–12.Google Scholar
  44. Waldron, J.S. (1973). Instructional Development Unit: A Guide to Organizing an Instructional Development Unit in Health Science. Bethesda, MD: National Institutes of Health (DHEW). (ERIC Document Reproduction Service No. ED 125 625).Google Scholar
  45. Weston, C.B., McAlpine, L. & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development 43(3): 29–49.Google Scholar
  46. Weston, C.B., McAlpine, L. & Tremblay, D. (April, 1995). Roles of Participants in Formative Evaluation: A Problem Space Analysis. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.Google Scholar

Copyright information

© Kluwer Academic Publishers 1997

Authors and Affiliations

  • Cynthia Weston
    • 1
  • Cathrine Le Maistre
    • 2
  • Lynn Mcalpine
    • 1
  • Tino Bordonaro
    • 2
  1. 1.Department of Educational and Counselling PsychologyCanada
  2. 2.Department of Educational Studies, Faculty of EducationMcGill UniversityMontrealCanada

Personalised recommendations