The influence of participants in formative evaluation on the improvement of learning from written instructional materials
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Formative evaluation in instructional design is widely advocated as a means to improve instructional materials through tryouts with learners and experts and revision based on this feedback. Research on formative evaluation has sought to prove its effectiveness and to provide guidance as to which methods of collecting feedback and which sources of feedback are most effective in improving learning. It is difficult to determine from the research whether aspects of tryout or of revision are responsible for improved learning. This paper describes a systematic investigation of both tryout and revision to more clearly determine the influence of different formative evaluation participants (experts, learners and revisers) on the improvement of learning from written instructional materials.
It was found that revised versions incorporating learner feedback had the most impact on improving learning from the materials. It was also found that revisers have a far more powerful impact on formative evaluation outcomes than was previously supposed, in terms of how they mediate and incorporate the feedback they are given. The results refute contentions that any revision is better than none since versions revised without learner feedback did not improve learning. Implications are discussed.
- Andrews, D.H. & Goodson, L.A. (1980). A comparative analysis of models of instructional design. Instructional Technology 3(4): 2–16.
- Baker, E.L. (1970). Generalizability of rules for empirical revision. Audiovisual Communication Review 18(3): 300–305.
- Baker, E.L. & Alkin, M.C. (1973). Formative evaluation of instructional development. AV Communication Review 21(4): 389–418.
- Bordonaro, T. (1993). A Comparison of the Effectiveness, Cost and Efficiency of Four Formative Evaluation Conditions. Unpublished master's thesis, McGill University, Montreal, Canada.
- Briggs, L. (1970). Handbook of Procedures for the Design of Instruction. Pittsburgh, PA: American Institutes for Research.
- Cambre, M. (1981). Historical overview of formative evaluation of instructional media products. Educational Communications and Technology Journal 29(1): 1–25.
- Collis, B.A. (1993). Evaluating instructional applications of telecommunications in distance education. Educational and Training Technology International 30(3): 266–274.
- Comber, T. (1995). Building Usable Web Pages: An HCI Perspective. [On-line]. Available: http://www.scu.edu.au/ausweb95/papers/hypertext/comber.
- Crooks, B. & Lamy, M.N. (1995). Using combinations of video, audio and print to teach French at a distance: A case study of the formative evaluation cycle. European Journal of Psychology of Education 10(2): 131–144.
- Davidove, E.A. & Reiser, R.A. (1991). Comparative acceptability and effectiveness of teacher-revised and designer-revised instruction. Educational Technology Research and Development 39(2): 29–38.
- De Ketele, J.M. & Roegiers, X. (1993). Méthodologie du receuil d'informations. Bruxelles: De Boeck-Wesmael.
- Dick, W. & Carey, L. (1990). The Systematic Design of Instruction (3rd ed.). Glenview, IL: Scott, Foresman & Company.
- Dupont, D. & Stolovitch, H.D. (1983). The effects of a systematic revision model on revisers in terms of student outcomes. NSPI Journal March: 33–37.
- Ericsson, K.A. & Simon, H.A. (1993). Protocol Analysis: Verbal Reports as Data (Revised edition). Cambridge, MA: MIT Press.
- Flagg, B.N. (1990). Formative Evaluation for Educational Technologies. Hillsdale, NJ: Lawrence Erlbaum Associates.
- Golas, K.C. (1983). Formative evaluation effectiveness and cost: Alternative models for evaluating printed instructional materials. Performance and Instruction 22: 17–19.
- Gropper, G.L. (1975). Diagnosis and Revision in the Development of Instructional Materials. Englewood Cliffs, NJ: Educational Technology Publications.
- Gustafson, K.L. (1991). Survey of Instructional Development Models (2nd ed.). Syracuse, NY: Information Resources Publications.
- Hayes, J.R. (1989). The Complete Problem Solver (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Jonassen, D.H., ed. (1988). Instructional Designs for Microcomputer Courseware. Hillsdale, NJ: Lawrence Erlbaum Associates.
- Kandaswamy, S., Stolovitch, H.D.& Thiagarajan, S. (1976). Learner verification and revision: An experimental comparison of two methods. Audio-Visual Communication Review 24(3): 316–328.
- Komoski, P.K. & Woodward, A. (1985). The continuing need for learner verification and revision of textual material, in D.H. Jonassen, ed., The Technology of Text: Volume 2 (pp. 396–417). Englewood Cliffs, NJ: Educational Technology Publications.
- Le Maistre, C. (1991, April). Revision: TheMissing Link. Paper presented at the annual-meeting of the American Educational Research Association, Chicago, IL.
- Le Maistre, C. (1994). The Priorities Established Among Data Sources when Instructional Designers Revise Written Materials. Unpublished doctoral dissertation, McGill University, Montreal, Canada.
- Le Maistre, C. & Weston, C.B. (1996). The priorities established among data sources when instructional designers revise written materials. Educational Technology Research and Development 44(1): 61–70.
- McAlpine, L. (1992). Highlighting formative evaluation: An instructional design model derived from practice. Performance and Instruction Journal 31(10): 16–18.
- Melton, R.F. (1995). Developing a formative evaluation system for distance teaching. Open Learning 10(2): 53–57.
- Newell, A., (1980). Reasoning, problem-solving, and decision processes: The problem space as a fundamental category, in R. Nickerson, ed., Attention and Performance, Vol. VIII (pp. 693–719). Hillsdale, NJ: Erlbaum.
- Newell, A. & Simon, H.A. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.
- Popham, W.J. & Baker, E.L. (1970). Systematic Instruction. Englewood Cliffs, NJ: Prentice-Hall.
- Price, R.V. & Repman, J. (1995). Instructional design for college level courses using instructional television. Journal of Educational Technology Systems 23(3): 251–263.
- Rahilly, T.J. (1991). An Analysis of Three Learner-Based Formative Evaluation Conditions. Unpublished master's thesis, McGill University, Montreal, Quebec.
- Reigeluth, C.M. (1983). Instructional design: What is it and why is it? in C.M. Reigeluth, ed., Instructional Design Theories and Models: An Overview of the Current Status (pp. 3–36). Hillsdale, NJ: Lawrence Erlbaum Associates.
- Rosen, M.J. (1968). An Experimental Design for Comparing the Effects of Instructional Media Programming Procedures: Subjective vs. Objective Revision Procedures. Palo Alto, CA: American Institutes for Research in the Behavioral Sciences.
- Rowland, G. (1993). Designing and instructional design. Educational Technology Research and Development 41(1): 79–91.
- Saroyan, A. (1989). The Review Process in Formative Evaluation of Instructional Text: The Role of Content Experts and Instructional Designers. Unpublished doctoral dissertation. Montreal, Canada. McGill University.
- Scriven, M. (1967). The methodology of evaluation, in R.W. Tyler et al., eds., Perspectives of Curriculum Evaluation (pp. 39–83) (AERA monograph series on curriculum evaluation, no1). Chicago: Rand McNally.
- Shanteau, J. (1992). The psychology of experts: An alternative view, in G. Wright & F. Bolger, eds., Expertise and Decision Support. New York: Plenum Press.
- Tessmer, M. (1994). Formative evaluation alternatives. Performance Improvement Quarterly 71(1): 3–18.
- Thiagarajan, S. (1991). Formative evaluation and performance technology. Performance Improvement Quarterly 4(2): 22–34.
- Tremblay, D. (1994). An Analysis of the Role of Expert Reviewers in Formative Evaluation. Unpublished master's thesis, McGill University, Montreal, Canada.
- Volman, C. & De Diana, I. (1993).Courseware evaluation and decision support based oncourseware use. Educational and Training Technology International 30(3): 255–274.
- Wager, W.W. (1993). Instructional systems fundamentals: Pressure to change. Educational Technology 33(2): 8–12.
- Waldron, J.S. (1973). Instructional Development Unit: A Guide to Organizing an Instructional Development Unit in Health Science. Bethesda, MD: National Institutes of Health (DHEW). (ERIC Document Reproduction Service No. ED 125 625).
- Weston, C.B., McAlpine, L. & Bordonaro, T. (1995). A model for understanding formative evaluation in instructional design. Educational Technology Research and Development 43(3): 29–49.
- Weston, C.B., McAlpine, L. & Tremblay, D. (April, 1995). Roles of Participants in Formative Evaluation: A Problem Space Analysis. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
- The influence of participants in formative evaluation on the improvement of learning from written instructional materials
Volume 25, Issue 5 , pp 369-386
- Cover Date
- Print ISSN
- Online ISSN
- Kluwer Academic Publishers
- Additional Links
- Author Affiliations
- 1. Department of Educational and Counselling Psychology, Canada
- 2. Department of Educational Studies, Faculty of Education, McGill University, 3700 McTavish Street, Montreal, QC, Canada, H3A 1Y2