Advertisement

Evaluating online learning communities

  • Fengfeng Ke
  • Christopher Hoadley
Research Article

Abstract

This article reviews recent evaluation studies of online learning communities to provide a systematic understanding of how such communities are evaluated. Forty-two representative studies were selected and categorized into a newly developed taxonomy of online learning community evaluations. This taxonomy is divided into four components: evaluation purpose, evaluation approach, measures for evaluation, and evaluation techniques. The findings suggest that it is inappropriate to conceptualize evaluation of such communities as a one-size-fits-all, generalizable measure of “good” or “bad.” Instead, we recommend a comprehensive, on-going, diagnostic approach to measuring clusters of indicators, or syndromes, of a particular OLC and examining the causal relation assumed by the evaluators between what is measured and the success of OLC as an imputed outcome.

Keywords

Online learning communities Evaluation Literature review Sensitivity analysis 

References

  1. Alem, L., & Kravis, S. (2004). Design and evaluation of an online learning community: A case study at CSIRO. SIGGROUP Bulletin, 25(1), 20–24.Google Scholar
  2. Ardichvili, A., Page, N., & Wentling, T. (2002). Virtual knowledge-sharing communities of practice at caterpillar: Success factors and barriers. Performance Improvement Quarterly, 15(3), 94–113.Google Scholar
  3. Auyeung, L. H. (2004). Building a collaborative online learning community: A case study in Hong Kong. Journal of Educational Computing Research, 31(2), 119–136. doi: 10.2190/YCM8-XKDY-QWN2-GPEH.CrossRefGoogle Scholar
  4. Aviv, R., Erlich, Z., Ravid, G., & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1–23.Google Scholar
  5. Bielaczyc, K. (2001). Designing social infrastructure: The challenge of building computer-supported learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 106–114). Maastricht: Maastricht University.Google Scholar
  6. Bozik, M., & Tracey, K. (2002). Fostering intellectual development in a learning community: Using an electronic discussion board. In P. Comeaux (Ed.), Communication and collaboration in the online classroom: Examples and applications (pp. 207–225). Boston, MA: Anker Publishing.Google Scholar
  7. Brett, C., Woodruff, E., & Nason, R. (1999). Online community and preservice teachers’ conceptions of learning mathematics. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the computer support for collaborative learning (CSCL) (pp. 57–66). Mahwah, NJ: Erlbaum.Google Scholar
  8. Brown, A. L., Ellery, S., & Campione, J. C. (1998). Creating zones of proximal development electronically. In J. G. Greeno & S. V. Goldman (Eds.), Thinking practices in mathematics and science learning (pp. 341–367). Mahwah, NJ: Erlbaum.Google Scholar
  9. Bruckman, A. (2004). Co-evolution of technological design and pedagogy in an online learning community. In S. A. Barab, R. Kling, & J. Gray (Eds.), Designing virtual communities in the service of learning (pp. 239–255). New York: Cambridge University Press.Google Scholar
  10. Bryce-Davis, H. (2001). Virtual learning communities. Paper presented at the Multimedia in the Home Conference, TRLabs, August 22–24, 2002, Saskatoon, Saskatchewan.Google Scholar
  11. Carlen, U., & Jobring, O. (2005). The rationale of online learning communities. International Journal of Web Based Communities, 1(3), 272–295.CrossRefGoogle Scholar
  12. Chang, C. (2003). Towards a distributed web-based learning community. Innovations in Education and Teaching International, 40(1), 27–42.CrossRefGoogle Scholar
  13. Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2007). Social networks, communication styles, and learning performance in a CSCL community. Computers & Education, 49(2), 309–329.CrossRefGoogle Scholar
  14. Collins, T., Mulholland, P., & Watt, S. (2001). Using genre to support active participation in learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 156–164). Maastricht: Maastricht University.Google Scholar
  15. Conrad, D. (2002). Deep in the hearts of learners: Insights into the nature of online community. Journal of Distance Education, 17(1), 1–19.Google Scholar
  16. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.Google Scholar
  17. Cuthell, J., Limited, A., Mills, R., & Derbyshire, I. (2002). MirandaNet: A learning community––a community of learners. Journal of Interactive Learning Research, 13(1/2), 167–186.Google Scholar
  18. De Laat, M., & Lally, V. (2003). Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community. Instructional Science, 31, 7–39.CrossRefGoogle Scholar
  19. Derry, S. J., & DuRussel, L. A. (1999, July). Assessing knowledge construction in on-line learning communities. Paper presented at the Annual Meeting of the International Society for Artificial Intelligence in Education, Lemans, France (ERIC Document Reproduction No. Ed 446897).Google Scholar
  20. Engeström, Y. (1993). Developmental studies of work as a testbench of activity theory: Analyzing the work of general practitioners. In S. Chaiklin & J. Lave (Eds.), Understanding practice: Perspectives on activity and context (pp. 64–103). Cambridge: Cambridge University Press.Google Scholar
  21. Fusco, J., Gehlbach, H., & Schlager, M. (2000). Assessing the impact of a large-scale online teacher professional development community. Society for information technology and teacher education international conference 2000 (Vol. 1, pp. 2178–2183). Chesapeake, VA: AACE.Google Scholar
  22. Haythornthwaite, C., Kazmer, M., Robins, J., & Shoemaker, S. (2000). Community development among distance learners: Temporal and technological dimensions. Journal of Computer-Mediated Communication, 6(1), Retrieved January 17, 2005, from http://www.ascusc.org/jcmc/vol6/issue1/haythornthwaite.html.
  23. Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117–136). Berlin: Springer-Verlag.Google Scholar
  24. Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48(3), 23–48.CrossRefGoogle Scholar
  25. Hiltz, S. R., & Turoff, M. (2002). What makes learning networks effective? Communications of the ACM, 45(4), 56–59.CrossRefGoogle Scholar
  26. Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer support for collaborative learning 2002 (pp. 453–462). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  27. Hugo, J. (2002). Learning community history. New Directions for Adult and Continuing Education, 95, 5–26.CrossRefGoogle Scholar
  28. Hung, D., & Chen, D. (2002). Understanding how thriving Internet quasi-communities work: Distinguishing between learning about and learning to be. Educational Technology, 42(1), 23–27.Google Scholar
  29. Johnson, C. M. (2001). A survey of current research on online communities of practice. Internet and Higher Education, 4, 45–60.CrossRefGoogle Scholar
  30. Johnson, S. D., Suriya, C., Yoon, S. W., Berrett, J. V., & Fleur, J. L. (2002). Team development and group process of virtual learning teams. Computers & Education, 39, 379–393.CrossRefGoogle Scholar
  31. Ke, F., & Carr-Chellman, A. (2006). Solitary learner in online collaborative learning: A disappointing experience? Quarterly Review of Distance Education, 7(3), 249–265.Google Scholar
  32. Kilner, P. G., & Hoadley, C. M. (2005). Anonymity options and professional participation in an online community of practice. In T. Koschmann, T. W. Chan, & D. D. Suthers (Eds.), Proceedings of the computer support for collaborative learning (CSCL) 2005 (pp. 272–280). Mahwah, NJ: Erlbaum.Google Scholar
  33. Kowch, E., & Schwier, R. (1997). Considerations in the construction of technology-based virtual learning communities. Canadian Journal of Educational Communication, 26(1), 1–12.Google Scholar
  34. Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In B. A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 17–44). Cambridge, MA: MIT Press.Google Scholar
  35. Lally, V., & Barrett, E. (1999). Building a learning community on-line: Towards socio-academic interaction. Research Papers in Education, 14(2), 147–163.CrossRefGoogle Scholar
  36. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press.Google Scholar
  37. Leh, A. S. (2002). Action research on hybrid courses and their online communities. Educational Media International, 39(1), 31–38.CrossRefGoogle Scholar
  38. Linn, M. C., & Burbules, N. C. (1993). Construction of knowledge and group learning. In K. G. Tobin (Ed.), The practice of constructivism in science education (pp. 91–119). Washington, DC: American Association for the Advancement of Science (AAAS) Press.Google Scholar
  39. Littleton, K., & Whitelock, D. (2005). The negotiation and co-construction of meaning and understanding within a postgraduate online learning community. Learning, Media and Technology, 30(2), 147–164.CrossRefGoogle Scholar
  40. Lock, J. (2002). Laying the groundwork for the development of learning communities within online courses. Quarterly Review of Distance Education, 3, 395–408.Google Scholar
  41. Majdalani, M. C., Hollon, R. E., & McIntyre, S. R. (1999, February). The virtual learning technology community: Creating and sustaining professional development for K-16 learning communities. Paper presented at 10th Society for Information Technology & Teacher Education International Conference, Antonio, TX (ERIC Document Reproduction Service No. ED 432299).Google Scholar
  42. Maor, D. (2003). The teacher’s role in developing interaction and reflection in an online learning community. Educational Media International, 40(1/2), 127–137.CrossRefGoogle Scholar
  43. McMillan, D. W., & Chavis, D. M. (1986). Sense of community: A definition and theory. Journal of Community Psychology, 14(1), 6–23.CrossRefGoogle Scholar
  44. McPherson, M., & Nunes, M. B. (2004). The failure of a virtual social space (VSS) designed to create a learning community: Lessons learned. British Journal of Educational Technology, 35(3), 305–321.CrossRefGoogle Scholar
  45. Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters.Google Scholar
  46. Meyers, R., Davis, H., & Botti, J. (2002, June). Professional development: Building effective virtual communities through cooperative learning. Paper presented at National Educational Computing Conference 2002, San Antonio, Texas (ERIC Document Reproduction Service No. ED 475943).Google Scholar
  47. Miller, R. A. (1994). Medical diagonostic decision support systems––past, present, and future. Journal of the American Medical Informatics Association, 1(1), 8–27.Google Scholar
  48. Moore, K., Zaff, J., & Hair, E. (2002). Research and evaluation to serve the public: A commentary on Catalana, Berglund, Ryan, Lonczak, and Hawkins. Prevention & Treatment, 5(1).Google Scholar
  49. Oliver, M. (2000). An introduction to the evaluation of learning technology. Educational Technology & Society, 3(4), 20–30.Google Scholar
  50. Oren, A., Mioduser, D., & Nachmias, R. (2002). The development of social climate in virtual learning discussion groups. International Review of Research in Open and Distance learning, 3(1). Retrieved, April 18, 2005, from http://www.icaap.org/iuicode?149.3.1.x.
  51. Orey, M., Koenecke, L., & Crozier, J. (2003). Learning communities via the Internet a la epic learning: You can lead the horses to water, but you cannot get them to drink. Innovations in Education and Teaching International, 40(3), 260–269.CrossRefGoogle Scholar
  52. Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass Publishers.Google Scholar
  53. Patton, M. (1997). Utilization-focused evaluation. London: Sage.Google Scholar
  54. Pearson, J. (1998). Electronic networking in initial teacher education: Is a virtual faculty of education possible? Computers & Education, 32, 221–238.CrossRefGoogle Scholar
  55. Poole, D. M. (2000). Student participation in a discussion-oriented online course: A case study. Journal of Research on Computing in Education, 33(2), 162–177.Google Scholar
  56. Preece, J. (2000). Sociability and usability in online communities: Determining and measuring success. Behaviour & Information Technology, 20(5), 347–356.CrossRefGoogle Scholar
  57. Renninger, K., & Shumar, W. (2002). Building virtual communities: Learning and change in cyberspace. New York: Cambridge University Press.Google Scholar
  58. Rheingold, H. (2000). The virtual community: Homesteading on the electronic frontier. Cambridge, MA: The MIT Press.Google Scholar
  59. Rice-Lively, M. L. (1994). Wired warp and woof: An ethnographic study of a networking class. Internet Research, 4(4), 20–35.CrossRefGoogle Scholar
  60. Ricketts, J., Wolfe, F. H., Norvelle, E., & Carpenter, E. H. (2000). Multimedia: Asynchronous distributed education––a review and case study. Social Science Computer Review, 18(2), 132–146.CrossRefGoogle Scholar
  61. Romiszowsky, A., & Mason, R. (1996). Computer-mediated communication. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 438–456). New York: Simon & Schuster Macmillan.Google Scholar
  62. Rovai, A. P. (2001). Building classroom community at a distance: A case study. Educational Technology Research and Development, 49(4), 33–48.CrossRefGoogle Scholar
  63. Rovai, A. P. (2002). Sense of community, perceived cognitive learning, and persistence in aynchronous learning networks. The Internet and Higher Education, 5(4), 319–332.CrossRefGoogle Scholar
  64. Russell, M., & Ginsburg, L. (1999). Learning online: Extending the meaning of community: A review of three programs from the Southeastern United States (Report No. NCAL-TR99-01). National Center for Adult Literacy, Philadelphia, PA. SouthEast and Islands Regional Technology in Education Consortium, Greensboro, NC (ERIC Document Reproduction Service No. ED 437540).Google Scholar
  65. Scardamalia, M., Bereiter, C., Brett, C., Burtis, P. J., Calhoun, C., & Smith Lea, N. (1992). Educational applications of a networked communal database. Interactive Learning Environments, 2(1), 45–71.CrossRefGoogle Scholar
  66. Schön, D. (1983). The reflective practitioner. New York: Basic Books.Google Scholar
  67. Scriven, M. (1991). Prose and cons about goal-free evaluation. American Journal of Evaluation, 12(1), 55–62.CrossRefGoogle Scholar
  68. Shea, P., Swan, K., Li, C. S., & Pickett, A. (2005). Developing learning community in online asynchronous college course: The role of teaching presence. Journal of Asynchronous Learning Networks, 9(4), 59–82.Google Scholar
  69. Soller, A., & Lesgold, A. (2003). A computational approach to analyzing online knowledge sharing interaction. In H. U. Hoppe, M. F. Verdejo, & J. Kay (Eds.), Artificial intelligence in education (pp. 253–260). Amsterdam, Netherlands: IOS Press.Google Scholar
  70. Spitzer, W., & Wedding, K. (1995). Labnet: An international electronic community for professional development. Computer Education, 24(3), 247–255.CrossRefGoogle Scholar
  71. Squire, K. D., & Johnson, C. B. (2000). Supporting distributed communities of practice with interactive television. Educational Technology Research and Development, 48(1), 23–43.CrossRefGoogle Scholar
  72. Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In Proceedings of the 6th international conference on learning sciences, Los Angeles, California (pp. 521–528).Google Scholar
  73. Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park: Sage.Google Scholar
  74. Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication & Information, 2(1), 23–49.CrossRefGoogle Scholar
  75. Thomas, M. J. W. (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning 18(3): 351–366.Google Scholar
  76. Tu, C. H., & Corry, M. (2002). ELearning communities. The Quarterly Review of Distance Education, 3(2), 207–218.Google Scholar
  77. Upcraft, M. L., & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.Google Scholar
  78. Wang, M., Sierra, C., & Folger, T. (2003). Building a dynamic online learning community among adult learners. Education Media International, 40(1/2), 49–61.CrossRefGoogle Scholar
  79. Watkins, D. S. (1997). The development, pilot test, assessment, and evaluation of a computerized online internet community system at programs for higher education. Unpublished doctoral dissertation, Nova Southeastern University, Fort Lauderdale.Google Scholar
  80. Wegerif, R. (1998). The social dimension of asynchronous learning networks, Journal of Asynchronous Learning Networks, 2(1). Retrieved April 15, 2004, from http://www.aln.org/alnweb/journal/vo12issue1/wegerif.htm.

Copyright information

© Association for Educational Communications and Technology 2009

Authors and Affiliations

  1. 1.College of Education, Organizational Learning and Instructional Technology ProgramUniversity of New MexicoAlbuquerqueUSA
  2. 2.Steinhardt School of Culture, Education, and Human DevelopmentNew York UniversityNew YorkUSA

Personalised recommendations