Abstract
Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication.
Similar content being viewed by others
References
Anderson, T., Rourke, L., Garrison, D.R., & Archer, W. (2001) Assessing teaching presence in a computer conferencing environment.Journal of Asynchronous Learning Networks,5 (2). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/jalnvol5issue2v2.htm
Bakeman, R., & Gottman, J.M. (1997).Observing interaction: An introduction to sequential analysis (2nd ed.). New York: Cambridge University Press.
Bereiter, C., & Scardemalia, M. (1987).The psychology of written composition. Hillsdale N.J.: Lawrence Erlbaum Associates.
Berelson, B. (1952).Content analysis in communication research. Glencoe, IL: Free Press.
Bullen, M. (1998). Participation and critical thinking in online university distance education.Journal of Distance Education, 13(2), 1–32.
Chou, C. (November, 2001).A model of learner-centered computer-mediated interaction for collaborative distance learning. Paper presented at the Annual Meeting of the Association for Educational and Communications Technology, Atlanta, GA.
Community of Inquiry. (2002). Critical thinking in a text-based environment: Computer conferencing in higher education. Retrieved March 6, 2002, from http://www.atl.ualberta.ca/cmc
Grocker, L., & Algina, J. (1986).Introduction to classical and modern test theory. Toronto: Harcourt Brace Jovanovich College Publishers.
Cronbach, L. (1971). Test validation. In R.L. Thorndike (Ed.),Educational Measurement (2nd ed.). Washington, D.C.: American Council on Education.
Cronbach, L. (1990).Essentials of psychological testing (5rd ed.). New York: Harper & Row.
Curtis, D., & Lawson M. (2001) Exploring collaborative learning online.Journal of Asynchronous Learning Networks,5(1). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/Vol5_issue1/Curtis/curtis.htm
Dalkey, N., & Helmer, O. (1963). An experimental application of the delphi method to the user of experts.Management Science, 9(3), 458–467.
Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span.Psychological Bulletin, 93, 179–197.
Ericsson, K., & Simon, H. (1993)Protocol analysis: Verbal reports as data. Cambridge, Mass: MIT Press.
Fahy, P. (2001). Addressing some common problems in transcript analysis.International Review of Research in Open and Distance Learning,1(2). Retrieved, March 20, 2002, from the World Wide Web at. http://www.irrodl.org/content/v1.2/research.html
Fahy, P. (2002a). Epistolary and expository interactions patterns in a computer conference transcript. Retrieved, March 1, 2002, from http://cde.athabascau.ca/softeval/reports/mag2-jde.pdf
Fahy, P. (2002b). Evaluating critical thinking in a com
Fahy, P. (in press). Use of linguistic qualifiers and intensifiers in a computer conference.American Journal of Distance Education.
Flanagan, J. (1954). The critical incident technique.Psychological Bulletin, 51(4), 327–359.
Gall, M., Borg, W., & Gall, J. (1996).Educational research: An introduction (6th ed.). White Plains, NY: Longman.
Garrison, D.R., Anderson, T., & Archer, W. (2001). Critical thinking, congitive presence, and computer conferencing in distance education.American Journal of Distance Education,15(1).
Gunawardena, C.N., Lowe, C.A., & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing.Journal of Educational Computing Research 17(4), 397–431.
Gunawardena, C.N., & Zittle, F. (1998). Social presence as a predictor of satisfaction within a computer mediated conferencing environment.The American Journal of Distance Education, 11(3), 8–25.
Hara, N., Bonk, C. & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course.Instructional Science, 28(2), 115–152.
Henri, F. (1991). Computer conferencing and content analysis. InCollaborative learning through computer conferencing (pp. 117–136). Berlin: Springer-Verlag.
Johnson, D., & Johnson, R. (1979). Conflict in the classroom: Controversy and learning.Review of Educational Research, 49, 51–70.
Johnson, D., & Johnson, R. (1986). Computer-assisted cooperative learning.Educational Technology, 26(1), 12–18.
Johnson, D., & Johnson, R. (1989).Cooperation and competition: theory and research. Edina, MN: Interaction.
Johnson, D., & Johnson, R. (1992a).Creative controversy: Intellectual challenge in the classroom. Edina, MN: Interaction.
Johnson, D., & Johnson, R. (1992b). Positive interdependence: key to effective cooperation. In R. Hertz-Lazarowitz & N. Miller (Eds.),Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 174–99). Cambridge, England: Cambridge University Press.
Johnson, D.W., & Johnson, F. (1994a).Joining together. Group theory and group skills (5th ed). Englewood Cliffs, NJ: Prentice Hall.
Johnson, D., & Johnson, R. (1994b).Leading the cooperative school (2nd ed.) Edina, MN: Interaction.
Johnson, D., & Johnson, R. (1996). Cooperation and the use of technology. In D. Jonassen (Ed.),Handbook of research for educational communications and technology (pp. 1017–1044). New York: Simon and Schuster Macmillan. (1996).
Johnson, D.W., Johnson, R., & Maruyama, G. (1983). Interdependence and interpersonal attraction among heterogeneous and homogeneous individuals: A theoretical formation and a meta-analysis of the research.Review of Educational Research, 53(5), 5–54.
Johnson, D., Johnson, R., & Stanne, M. (2000).Cooperative learning methods: A meta-analysis. Retrieved, March 6, 2003, from http://www.co-operation.org/pages/cl-methods.html
Jonassen, D., & Kwon, H. (2001). Communication patterns in computer mediated versus face-to-face group problem solving.Educational Technology Research and Development, 49(1), 35–51.
Jonassen, D., & Kwon, H. (2002). The effects of argumentation scaffolds on argumentation and problem solving.Educational Technology Research and Development, 50(3), 5–21.
Kamin, C., O'Sullivan, P., Younger, M., & Deterding, R. (2001). Measuring critical thinking in problem-based learning discourse.Teaching and Learning in Medicine, 13(1), 27–35.
Kaplan, A. (1964).The conduct of inquiry: Methodology for behavioral science. Scranton PA: Chandler.
Krippendorff, K. (1980).Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.
Lord, F., & Novick, M. (1968).Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Mason, R. (1991). Analyzing computer conference interactions.Computer in Adult Education and Training, 2(3), 161–173.
McLean, S., & Morrison, D. (2000). Learners sociodemographic characteristics and participation in computer conferencing.Journal of Distance Education,15(2). Retrieved, March 9, 2002, from http://cade.athabascau.ca/vol15.2/mclean.html
Messick, S. (1989). Validity. In R.L. Linn (Ed.),Educational measurement (3rd ed, pp. 13–103). New York: Macmillan
Norris, S., & Ennis, R. (1989).Evaluating critical thinking. CA: Critical Thinking Press and Software.
Paisley, W. (1969). Studying style as deviation from encoding norms. In G. Gerbner, O. Holsti, K. Krippendorf, W. Paisley, & P. Stone (Eds.),The analysis of communication contents: Developments in scientific theories and computer techniques (pp. 4458). New York: Wiley.
Parson, M. (1996). Look who's talking: A pilot study of the use of discussion lists by journalism educators and students. (ERIC Document Reproduction Service No. ED 400 562)
Poole, M., & Holmes, M. (1995). The longitudinal analysis of interaction. In B. Montgomery & S. Duck (Eds.),Studying interpersonal interaction (pp. 286–302). New York: Guilford. 1991.
Potter, J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis.Journal of Applied Communication Research, 27(3): 258–284.
Reeves, T. (1995). Questioning the questions of instructional technology research. [Online] Available http: //www.hbg.psu.edu/bsed/intro/docs/dean/
Richardson, J., & Swan, K. (2003). Examining social presence in online courses in relation to students' perceived learning and satisfaction.Journal of Asynchronous Learning Networks 7(1). Retrieved July 1, 2003 from the World Wide Web at http://www.aln.org/publications/jaln/v7n1/v7n 1_richardson.asp
Riffe, D., Lacy, S., & Fico, F. (1998)Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.
Rogers, W.T. (1999).Error of measurement and validity. Edmonton AB: Available from author.
Rourke, L., & Anderson, T. (2002). Social communication in computer conferencing.Journal of Interactive Learning Research, 13(3), 259–275.
Rourke, L., Anderson, T. Garrison, D.R., & Archer, W. (1999). Assessing social presence in asynchronous, text-based computer conferencing.Journal of Distance Education, 14(3), 51–70.
Rourke, L., Anderson, T., Garrison, D.R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts.International Journal of Artificial Intelligence in Education, 12(1), 8–22.
Sharon, Y., & Sharon, S. (1992).Group investigation: Expanding cooperative learning. New York: Teacher's College Press.
Sheppard, L. (1993). Evaluating test validity.Review of Research in Education, 19, 405–450.
Short, J., Williams, E., & Christie, B. (1976).The social psychology of telecommunications. London, U.K.: Wiley, 1976.
Slavin, R. (1991).Student team learning: A practical guide to cooperative learning (3rd ed.). Washington, DC: National Education Association.
Snow, R., Federico, P., & Montague, W. (Eds.). (1980).Aptitude, learning, and instruction. (Vols. 1 & 2). Hillsdale, NJ: Lawrence Erlbaum.
Sutton, L. (2001). The principle of vicarious interaction in computer mediated communication.International Journal of Educational Telecommunications, 7(3), 223–242.
Stevens, S. (1946). On the theory of scales of measurement.Science, 103, 677–680.
Weiss, R., & Morrison, G. (1998). Evaluation of a graduate seminar conducted by listserve. (ERIC Document Reproduction Service No. ED 423 868)
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Rourke, L., Anderson, T. Validity in quantitative content analysis. ETR&D 52, 5–18 (2004). https://doi.org/10.1007/BF02504769
Issue Date:
DOI: https://doi.org/10.1007/BF02504769