Abstract
The objective of the study was to investigate the priorities instructional designers establish among data sources when they revise written materials and the relationship of their practice to standard models of formative evaluation.
Two modules of printed instructional material were revised by each of eight experienced instructional designers under three treatment conditions, using a counterbalanced design. Data were collected using a think-aloud procedure. The think-aloud protocol was segmented, coded and analysed.
Results show that instructional designers were significantly more likely to incorporate their own knowledge into revisions than to use feedback data, and when they did use feedback data, they preferred learner comments. They did not accurately assess their use of their own knowledge or of feedback data; they thought they used feedback data more than the results showed.
These instructional designers reflected some aspects of the standard instructional design model when they revised instructional materials, but their experience has caused many of the procedures to become internalized, so they are invisible in the protocols.
This research was funded in part by a grant from the Social Sciences and Humanities Research Council of Canada.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baker, E.L. (1974). The role of the evaluator in instructional development. In G.D. Borich (Ed.),Evaluating educational programs and products. Englewood Cliffs, NJ: Educational Technology Publications. (pp. 56–73).
Bordonaro, T. (1993).A comparison of the effectiveness, cost, and efficiency of four formative evaluation conditions. Unpublished master's thesis, McGill University, Montreal, Canada.
Ciesla, J. (1976, April).The effects of various formative evaluation procedures on instructional material revision in a large scale individualized science curriculum development project. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
Davidove, E.A., & Reiser, R.A. (1991). Comparative acceptibility and effectiveness of teacher-revised and designer-revised instruction.Educational Technology Research and Development, 39(2), 29–38.
Dick, W. (1968). A methodology for the formative evaluation of instructional materials.Journal of Educational Measurement, 5(2), 99–102.
Dick, W., & Carey, L. (1990).The systematic design of instruction. (3rd ed.). Glenview, IL: Scott, Foresman & Company.
Dick, W., & Carey, L. (1991). Formative evaluation. In L.J. Briggs, K.L. Gustafson, & M.H. Tillman (Eds.),Instructional design: Principles and applications. Englewood Cliffs, NJ: Educational Technology Publications.
Dijkstra, S. (1991, April).The instructional design for teaching to solve well- and ill-structured problems. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago.
Ericsson, K.A., & Simon, H.A. (1980). Verbal reports as data.Psychological Review, 87(3), 215–251.
Ericsson, K.A., & Simon, H.A. (1993).Protocol analysis: Verbal reports as data. (Revised edition) Cambridge, MA: MIT Press.
Flower, L., Carey, L., & Hayes, J.R., (1985).Diagnosis in revision: The experts' opinion. (Technical Report No. 27). Pittsburgh, PA: Carnegie-Mellon University, Communications Design Center.
Israeloff, A. (1992).Comparison of feedback generated by experts and learners during formative evaluation. Unpublished master's thesis, McGill University, Montreal, Canada.
Kandaswamy, S. (1980). Sequential model for appraising instructional superiority of revised materials.Educational Communication and Technology Journal, 28(3), 186–193.
Kandaswamy, S., Stolovitch, H., & Thiagarajan, S. (1976). Learner verification and revision: An experimental comparison of two methods.Audio-Visual Communication Review, 24(3), 316–328.
Krippendorff, K. (1980).Content Analysis. Beverley Hills, CA: Sage.
Le Maistre, K. (1991, April).Revision: The missing link. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
Markle, S.M. (1989). The ancient art of formative evaluation.Performance and Instruction, August 1989, 27–29
McCormick, R. (1976). Evaluation of Open University course materials.Instructional Science 5. Amsterdam: Elsevier Printing, 189–217.
Nathenson, M.B. & Henderson, E.S. (1980).Using student feedback to improve learning materials. London: Croom Helm.
Newell, A., & Simon, H.A. (1972).Human problem solving. Englewood Cliffs, NJ: Prentice-Hall Inc. (Chapter 14, pp. 787–868).
Rahilly, T.J. (1991).An analysis of three learner-based formative evaluation conditions. Unpublished master's thesis, McGill University, Montreal, Canada.
Rowland, G. (1992). What do instructional designers actually do? An initial investigation of expert practice.Performance Improvement Quarterly, 5(2), 65–86.
Rowland, G. (1993). Designing and instructional design.Educational Technology Research and Development, 41(1), 79–91.
Saroyan, A. (1989).The review process in formative evaluation of instructional text: The role of content experts and instructional designers. Unpublished doctoral thesis. McGill University, Montreal, Canada.
Smith, P.L., & Wedman, J.F., (1988). Read-thinkaloud protocols: A new data source for formative evaluation.Performance Improvement Quarterly, 1(2), 13–22.
Thiagarajan, S. (1978). Instructional product verification and revision: 20 questions and 200 speculations.Educational Communication and Technology Journal 26(2), 133–141.
Tremblay, D. (1994).An analysis of the role of expert reviewers in formative evaluation. Master's thesis, McGill University, Montreal, Canada.
Winn, W. (1990). Some implications of cognitive theory for instructional design.Instructional Science, 19, 53–69.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Le Maistre, K., Weston, C. The priorities established among data sources when instructional designers revise written materials. ETR&D 44, 61–70 (1996). https://doi.org/10.1007/BF02300326
Issue Date:
DOI: https://doi.org/10.1007/BF02300326