Validity in quantitative content analysis

  • Liam Rourke
  • Terry Anderson
Research

Abstract

Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication.

References

  1. Anderson, T., Rourke, L., Garrison, D.R., & Archer, W. (2001) Assessing teaching presence in a computer conferencing environment.Journal of Asynchronous Learning Networks,5 (2). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/jalnvol5issue2v2.htmGoogle Scholar
  2. Bakeman, R., & Gottman, J.M. (1997).Observing interaction: An introduction to sequential analysis (2nd ed.). New York: Cambridge University Press.Google Scholar
  3. Bereiter, C., & Scardemalia, M. (1987).The psychology of written composition. Hillsdale N.J.: Lawrence Erlbaum Associates.Google Scholar
  4. Berelson, B. (1952).Content analysis in communication research. Glencoe, IL: Free Press.Google Scholar
  5. Bullen, M. (1998). Participation and critical thinking in online university distance education.Journal of Distance Education, 13(2), 1–32.Google Scholar
  6. Chou, C. (November, 2001).A model of learner-centered computer-mediated interaction for collaborative distance learning. Paper presented at the Annual Meeting of the Association for Educational and Communications Technology, Atlanta, GA.Google Scholar
  7. Community of Inquiry. (2002). Critical thinking in a text-based environment: Computer conferencing in higher education. Retrieved March 6, 2002, from http://www.atl.ualberta.ca/cmcGoogle Scholar
  8. Grocker, L., & Algina, J. (1986).Introduction to classical and modern test theory. Toronto: Harcourt Brace Jovanovich College Publishers.Google Scholar
  9. Cronbach, L. (1971). Test validation. In R.L. Thorndike (Ed.),Educational Measurement (2nd ed.). Washington, D.C.: American Council on Education.Google Scholar
  10. Cronbach, L. (1990).Essentials of psychological testing (5rd ed.). New York: Harper & Row.Google Scholar
  11. Curtis, D., & Lawson M. (2001) Exploring collaborative learning online.Journal of Asynchronous Learning Networks,5(1). Retrieved, March 6, 2002, from http://www.aln.org/alnweb/journal/Vol5_issue1/Curtis/curtis.htmGoogle Scholar
  12. Dalkey, N., & Helmer, O. (1963). An experimental application of the delphi method to the user of experts.Management Science, 9(3), 458–467.CrossRefGoogle Scholar
  13. Embretson, S. (1983). Construct validity: Construct representation versus nomothetic span.Psychological Bulletin, 93, 179–197.CrossRefGoogle Scholar
  14. Ericsson, K., & Simon, H. (1993)Protocol analysis: Verbal reports as data. Cambridge, Mass: MIT Press.Google Scholar
  15. Fahy, P. (2001). Addressing some common problems in transcript analysis.International Review of Research in Open and Distance Learning,1(2). Retrieved, March 20, 2002, from the World Wide Web at. http://www.irrodl.org/content/v1.2/research.htmlGoogle Scholar
  16. Fahy, P. (2002a). Epistolary and expository interactions patterns in a computer conference transcript. Retrieved, March 1, 2002, from http://cde.athabascau.ca/softeval/reports/mag2-jde.pdfGoogle Scholar
  17. Fahy, P. (2002b). Evaluating critical thinking in a comGoogle Scholar
  18. Fahy, P. (in press). Use of linguistic qualifiers and intensifiers in a computer conference.American Journal of Distance Education.Google Scholar
  19. Flanagan, J. (1954). The critical incident technique.Psychological Bulletin, 51(4), 327–359.CrossRefGoogle Scholar
  20. Gall, M., Borg, W., & Gall, J. (1996).Educational research: An introduction (6th ed.). White Plains, NY: Longman.Google Scholar
  21. Garrison, D.R., Anderson, T., & Archer, W. (2001). Critical thinking, congitive presence, and computer conferencing in distance education.American Journal of Distance Education,15(1).Google Scholar
  22. Gunawardena, C.N., Lowe, C.A., & Anderson, T. (1997). Analysis of a global online debate and the development of an interaction analysis model for examining social construction of knowledge in computer conferencing.Journal of Educational Computing Research 17(4), 397–431.Google Scholar
  23. Gunawardena, C.N., & Zittle, F. (1998). Social presence as a predictor of satisfaction within a computer mediated conferencing environment.The American Journal of Distance Education, 11(3), 8–25.CrossRefGoogle Scholar
  24. Hara, N., Bonk, C. & Angeli, C. (2000). Content analysis of online discussion in an applied educational psychology course.Instructional Science, 28(2), 115–152.CrossRefGoogle Scholar
  25. Henri, F. (1991). Computer conferencing and content analysis. InCollaborative learning through computer conferencing (pp. 117–136). Berlin: Springer-Verlag.Google Scholar
  26. Johnson, D., & Johnson, R. (1979). Conflict in the classroom: Controversy and learning.Review of Educational Research, 49, 51–70.CrossRefGoogle Scholar
  27. Johnson, D., & Johnson, R. (1986). Computer-assisted cooperative learning.Educational Technology, 26(1), 12–18.Google Scholar
  28. Johnson, D., & Johnson, R. (1989).Cooperation and competition: theory and research. Edina, MN: Interaction.Google Scholar
  29. Johnson, D., & Johnson, R. (1992a).Creative controversy: Intellectual challenge in the classroom. Edina, MN: Interaction.Google Scholar
  30. Johnson, D., & Johnson, R. (1992b). Positive interdependence: key to effective cooperation. In R. Hertz-Lazarowitz & N. Miller (Eds.),Interaction in cooperative groups: The theoretical anatomy of group learning (pp. 174–99). Cambridge, England: Cambridge University Press.Google Scholar
  31. Johnson, D.W., & Johnson, F. (1994a).Joining together. Group theory and group skills (5th ed). Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  32. Johnson, D., & Johnson, R. (1994b).Leading the cooperative school (2nd ed.) Edina, MN: Interaction.Google Scholar
  33. Johnson, D., & Johnson, R. (1996). Cooperation and the use of technology. In D. Jonassen (Ed.),Handbook of research for educational communications and technology (pp. 1017–1044). New York: Simon and Schuster Macmillan. (1996).Google Scholar
  34. Johnson, D.W., Johnson, R., & Maruyama, G. (1983). Interdependence and interpersonal attraction among heterogeneous and homogeneous individuals: A theoretical formation and a meta-analysis of the research.Review of Educational Research, 53(5), 5–54.CrossRefGoogle Scholar
  35. Johnson, D., Johnson, R., & Stanne, M. (2000).Cooperative learning methods: A meta-analysis. Retrieved, March 6, 2003, from http://www.co-operation.org/pages/cl-methods.htmlGoogle Scholar
  36. Jonassen, D., & Kwon, H. (2001). Communication patterns in computer mediated versus face-to-face group problem solving.Educational Technology Research and Development, 49(1), 35–51.CrossRefGoogle Scholar
  37. Jonassen, D., & Kwon, H. (2002). The effects of argumentation scaffolds on argumentation and problem solving.Educational Technology Research and Development, 50(3), 5–21.CrossRefGoogle Scholar
  38. Kamin, C., O'Sullivan, P., Younger, M., & Deterding, R. (2001). Measuring critical thinking in problem-based learning discourse.Teaching and Learning in Medicine, 13(1), 27–35.CrossRefGoogle Scholar
  39. Kaplan, A. (1964).The conduct of inquiry: Methodology for behavioral science. Scranton PA: Chandler.Google Scholar
  40. Krippendorff, K. (1980).Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.Google Scholar
  41. Lord, F., & Novick, M. (1968).Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
  42. Mason, R. (1991). Analyzing computer conference interactions.Computer in Adult Education and Training, 2(3), 161–173.Google Scholar
  43. McLean, S., & Morrison, D. (2000). Learners sociodemographic characteristics and participation in computer conferencing.Journal of Distance Education,15(2). Retrieved, March 9, 2002, from http://cade.athabascau.ca/vol15.2/mclean.htmlGoogle Scholar
  44. Messick, S. (1989). Validity. In R.L. Linn (Ed.),Educational measurement (3rd ed, pp. 13–103). New York: MacmillanGoogle Scholar
  45. Norris, S., & Ennis, R. (1989).Evaluating critical thinking. CA: Critical Thinking Press and Software.Google Scholar
  46. Paisley, W. (1969). Studying style as deviation from encoding norms. In G. Gerbner, O. Holsti, K. Krippendorf, W. Paisley, & P. Stone (Eds.),The analysis of communication contents: Developments in scientific theories and computer techniques (pp. 4458). New York: Wiley.Google Scholar
  47. Parson, M. (1996). Look who's talking: A pilot study of the use of discussion lists by journalism educators and students. (ERIC Document Reproduction Service No. ED 400 562)Google Scholar
  48. Poole, M., & Holmes, M. (1995). The longitudinal analysis of interaction. In B. Montgomery & S. Duck (Eds.),Studying interpersonal interaction (pp. 286–302). New York: Guilford. 1991.Google Scholar
  49. Potter, J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis.Journal of Applied Communication Research, 27(3): 258–284.CrossRefGoogle Scholar
  50. Reeves, T. (1995). Questioning the questions of instructional technology research. [Online] Available http: //www.hbg.psu.edu/bsed/intro/docs/dean/Google Scholar
  51. Richardson, J., & Swan, K. (2003). Examining social presence in online courses in relation to students' perceived learning and satisfaction.Journal of Asynchronous Learning Networks 7(1). Retrieved July 1, 2003 from the World Wide Web at http://www.aln.org/publications/jaln/v7n1/v7n 1_richardson.aspGoogle Scholar
  52. Riffe, D., Lacy, S., & Fico, F. (1998)Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  53. Rogers, W.T. (1999).Error of measurement and validity. Edmonton AB: Available from author.Google Scholar
  54. Rourke, L., & Anderson, T. (2002). Social communication in computer conferencing.Journal of Interactive Learning Research, 13(3), 259–275.Google Scholar
  55. Rourke, L., Anderson, T. Garrison, D.R., & Archer, W. (1999). Assessing social presence in asynchronous, text-based computer conferencing.Journal of Distance Education, 14(3), 51–70.Google Scholar
  56. Rourke, L., Anderson, T., Garrison, D.R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts.International Journal of Artificial Intelligence in Education, 12(1), 8–22.Google Scholar
  57. Sharon, Y., & Sharon, S. (1992).Group investigation: Expanding cooperative learning. New York: Teacher's College Press.Google Scholar
  58. Sheppard, L. (1993). Evaluating test validity.Review of Research in Education, 19, 405–450.CrossRefGoogle Scholar
  59. Short, J., Williams, E., & Christie, B. (1976).The social psychology of telecommunications. London, U.K.: Wiley, 1976.Google Scholar
  60. Slavin, R. (1991).Student team learning: A practical guide to cooperative learning (3rd ed.). Washington, DC: National Education Association.Google Scholar
  61. Snow, R., Federico, P., & Montague, W. (Eds.). (1980).Aptitude, learning, and instruction. (Vols. 1 & 2). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  62. Sutton, L. (2001). The principle of vicarious interaction in computer mediated communication.International Journal of Educational Telecommunications, 7(3), 223–242.Google Scholar
  63. Stevens, S. (1946). On the theory of scales of measurement.Science, 103, 677–680.CrossRefGoogle Scholar
  64. Weiss, R., & Morrison, G. (1998). Evaluation of a graduate seminar conducted by listserve. (ERIC Document Reproduction Service No. ED 423 868)Google Scholar

Copyright information

© Association for Educational Communications and Technology 2004

Authors and Affiliations

  • Liam Rourke
    • 1
  • Terry Anderson
    • 2
  1. 1.Department of Educational Psychology at the University of AlbertaEdmonton
  2. 2.Athabasca University in AlbertaAlbertaCanada

Personalised recommendations