Abstract
This article reviews recent evaluation studies of online learning communities to provide a systematic understanding of how such communities are evaluated. Forty-two representative studies were selected and categorized into a newly developed taxonomy of online learning community evaluations. This taxonomy is divided into four components: evaluation purpose, evaluation approach, measures for evaluation, and evaluation techniques. The findings suggest that it is inappropriate to conceptualize evaluation of such communities as a one-size-fits-all, generalizable measure of “good” or “bad.” Instead, we recommend a comprehensive, on-going, diagnostic approach to measuring clusters of indicators, or syndromes, of a particular OLC and examining the causal relation assumed by the evaluators between what is measured and the success of OLC as an imputed outcome.
Similar content being viewed by others
Notes
e.g., Educational Technology Research & Development, Journal of Educational Computing Research, British Journal of Educational Technology, Computers & Education, Instructional Science, Journal of Interactive Learning Research, Journal of Asynchronous Learning Networks, Journal of Distance Education, etc.
e.g., Proceeding of Computer Support for Collaborative Learning, Association for Educational Communications and Technology International Convention, International Conference of Learning Science, etc.
References
Alem, L., & Kravis, S. (2004). Design and evaluation of an online learning community: A case study at CSIRO. SIGGROUP Bulletin, 25(1), 20–24.
Ardichvili, A., Page, N., & Wentling, T. (2002). Virtual knowledge-sharing communities of practice at caterpillar: Success factors and barriers. Performance Improvement Quarterly, 15(3), 94–113.
Auyeung, L. H. (2004). Building a collaborative online learning community: A case study in Hong Kong. Journal of Educational Computing Research, 31(2), 119–136. doi:10.2190/YCM8-XKDY-QWN2-GPEH.
Aviv, R., Erlich, Z., Ravid, G., & Geva, A. (2003). Network analysis of knowledge construction in asynchronous learning networks. Journal of Asynchronous Learning Networks, 7(3), 1–23.
Bielaczyc, K. (2001). Designing social infrastructure: The challenge of building computer-supported learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 106–114). Maastricht: Maastricht University.
Bozik, M., & Tracey, K. (2002). Fostering intellectual development in a learning community: Using an electronic discussion board. In P. Comeaux (Ed.), Communication and collaboration in the online classroom: Examples and applications (pp. 207–225). Boston, MA: Anker Publishing.
Brett, C., Woodruff, E., & Nason, R. (1999). Online community and preservice teachers’ conceptions of learning mathematics. In C. M. Hoadley & J. Roschelle (Eds.), Proceedings of the computer support for collaborative learning (CSCL) (pp. 57–66). Mahwah, NJ: Erlbaum.
Brown, A. L., Ellery, S., & Campione, J. C. (1998). Creating zones of proximal development electronically. In J. G. Greeno & S. V. Goldman (Eds.), Thinking practices in mathematics and science learning (pp. 341–367). Mahwah, NJ: Erlbaum.
Bruckman, A. (2004). Co-evolution of technological design and pedagogy in an online learning community. In S. A. Barab, R. Kling, & J. Gray (Eds.), Designing virtual communities in the service of learning (pp. 239–255). New York: Cambridge University Press.
Bryce-Davis, H. (2001). Virtual learning communities. Paper presented at the Multimedia in the Home Conference, TRLabs, August 22–24, 2002, Saskatoon, Saskatchewan.
Carlen, U., & Jobring, O. (2005). The rationale of online learning communities. International Journal of Web Based Communities, 1(3), 272–295.
Chang, C. (2003). Towards a distributed web-based learning community. Innovations in Education and Teaching International, 40(1), 27–42.
Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2007). Social networks, communication styles, and learning performance in a CSCL community. Computers & Education, 49(2), 309–329.
Collins, T., Mulholland, P., & Watt, S. (2001). Using genre to support active participation in learning communities. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), European perspectives on computer-supported collaborative learning: Proceedings of the 1st European conference on computer-supported collaborative learning (pp. 156–164). Maastricht: Maastricht University.
Conrad, D. (2002). Deep in the hearts of learners: Insights into the nature of online community. Journal of Distance Education, 17(1), 1–19.
Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.
Cuthell, J., Limited, A., Mills, R., & Derbyshire, I. (2002). MirandaNet: A learning community––a community of learners. Journal of Interactive Learning Research, 13(1/2), 167–186.
De Laat, M., & Lally, V. (2003). Complexity, theory and praxis: Researching collaborative learning and tutoring processes in a networked learning community. Instructional Science, 31, 7–39.
Derry, S. J., & DuRussel, L. A. (1999, July). Assessing knowledge construction in on-line learning communities. Paper presented at the Annual Meeting of the International Society for Artificial Intelligence in Education, Lemans, France (ERIC Document Reproduction No. Ed 446897).
Engeström, Y. (1993). Developmental studies of work as a testbench of activity theory: Analyzing the work of general practitioners. In S. Chaiklin & J. Lave (Eds.), Understanding practice: Perspectives on activity and context (pp. 64–103). Cambridge: Cambridge University Press.
Fusco, J., Gehlbach, H., & Schlager, M. (2000). Assessing the impact of a large-scale online teacher professional development community. Society for information technology and teacher education international conference 2000 (Vol. 1, pp. 2178–2183). Chesapeake, VA: AACE.
Haythornthwaite, C., Kazmer, M., Robins, J., & Shoemaker, S. (2000). Community development among distance learners: Temporal and technological dimensions. Journal of Computer-Mediated Communication, 6(1), Retrieved January 17, 2005, from http://www.ascusc.org/jcmc/vol6/issue1/haythornthwaite.html.
Henri, F. (1992). Computer conferencing and content analysis. In A. Kaye (Ed.), Collaborative learning through computer conferencing: The Najaden papers (pp. 117–136). Berlin: Springer-Verlag.
Herrington, J., & Oliver, R. (2000). An instructional design framework for authentic learning environments. Educational Technology Research and Development, 48(3), 23–48.
Hiltz, S. R., & Turoff, M. (2002). What makes learning networks effective? Communications of the ACM, 45(4), 56–59.
Hoadley, C. (2002). Creating context: Design-based research in creating and understanding CSCL. In G. Stahl (Ed.), Computer support for collaborative learning 2002 (pp. 453–462). Mahwah, NJ: Lawrence Erlbaum Associates.
Hugo, J. (2002). Learning community history. New Directions for Adult and Continuing Education, 95, 5–26.
Hung, D., & Chen, D. (2002). Understanding how thriving Internet quasi-communities work: Distinguishing between learning about and learning to be. Educational Technology, 42(1), 23–27.
Johnson, C. M. (2001). A survey of current research on online communities of practice. Internet and Higher Education, 4, 45–60.
Johnson, S. D., Suriya, C., Yoon, S. W., Berrett, J. V., & Fleur, J. L. (2002). Team development and group process of virtual learning teams. Computers & Education, 39, 379–393.
Ke, F., & Carr-Chellman, A. (2006). Solitary learner in online collaborative learning: A disappointing experience? Quarterly Review of Distance Education, 7(3), 249–265.
Kilner, P. G., & Hoadley, C. M. (2005). Anonymity options and professional participation in an online community of practice. In T. Koschmann, T. W. Chan, & D. D. Suthers (Eds.), Proceedings of the computer support for collaborative learning (CSCL) 2005 (pp. 272–280). Mahwah, NJ: Erlbaum.
Kowch, E., & Schwier, R. (1997). Considerations in the construction of technology-based virtual learning communities. Canadian Journal of Educational Communication, 26(1), 1–12.
Kuutti, K. (1996). Activity theory as a potential framework for human-computer interaction research. In B. A. Nardi (Ed.), Context and consciousness: Activity theory and human-computer interaction (pp. 17–44). Cambridge, MA: MIT Press.
Lally, V., & Barrett, E. (1999). Building a learning community on-line: Towards socio-academic interaction. Research Papers in Education, 14(2), 147–163.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press.
Leh, A. S. (2002). Action research on hybrid courses and their online communities. Educational Media International, 39(1), 31–38.
Linn, M. C., & Burbules, N. C. (1993). Construction of knowledge and group learning. In K. G. Tobin (Ed.), The practice of constructivism in science education (pp. 91–119). Washington, DC: American Association for the Advancement of Science (AAAS) Press.
Littleton, K., & Whitelock, D. (2005). The negotiation and co-construction of meaning and understanding within a postgraduate online learning community. Learning, Media and Technology, 30(2), 147–164.
Lock, J. (2002). Laying the groundwork for the development of learning communities within online courses. Quarterly Review of Distance Education, 3, 395–408.
Majdalani, M. C., Hollon, R. E., & McIntyre, S. R. (1999, February). The virtual learning technology community: Creating and sustaining professional development for K-16 learning communities. Paper presented at 10th Society for Information Technology & Teacher Education International Conference, Antonio, TX (ERIC Document Reproduction Service No. ED 432299).
Maor, D. (2003). The teacher’s role in developing interaction and reflection in an online learning community. Educational Media International, 40(1/2), 127–137.
McMillan, D. W., & Chavis, D. M. (1986). Sense of community: A definition and theory. Journal of Community Psychology, 14(1), 6–23.
McPherson, M., & Nunes, M. B. (2004). The failure of a virtual social space (VSS) designed to create a learning community: Lessons learned. British Journal of Educational Technology, 35(3), 305–321.
Mercer, N. (1995). The guided construction of knowledge: Talk amongst teachers and learners. Clevedon: Multilingual Matters.
Meyers, R., Davis, H., & Botti, J. (2002, June). Professional development: Building effective virtual communities through cooperative learning. Paper presented at National Educational Computing Conference 2002, San Antonio, Texas (ERIC Document Reproduction Service No. ED 475943).
Miller, R. A. (1994). Medical diagonostic decision support systems––past, present, and future. Journal of the American Medical Informatics Association, 1(1), 8–27.
Moore, K., Zaff, J., & Hair, E. (2002). Research and evaluation to serve the public: A commentary on Catalana, Berglund, Ryan, Lonczak, and Hawkins. Prevention & Treatment, 5(1).
Oliver, M. (2000). An introduction to the evaluation of learning technology. Educational Technology & Society, 3(4), 20–30.
Oren, A., Mioduser, D., & Nachmias, R. (2002). The development of social climate in virtual learning discussion groups. International Review of Research in Open and Distance learning, 3(1). Retrieved, April 18, 2005, from http://www.icaap.org/iuicode?149.3.1.x.
Orey, M., Koenecke, L., & Crozier, J. (2003). Learning communities via the Internet a la epic learning: You can lead the horses to water, but you cannot get them to drink. Innovations in Education and Teaching International, 40(3), 260–269.
Palloff, R. M., & Pratt, K. (1999). Building learning communities in cyberspace: Effective strategies for the online classroom. San Francisco: Jossey-Bass Publishers.
Patton, M. (1997). Utilization-focused evaluation. London: Sage.
Pearson, J. (1998). Electronic networking in initial teacher education: Is a virtual faculty of education possible? Computers & Education, 32, 221–238.
Poole, D. M. (2000). Student participation in a discussion-oriented online course: A case study. Journal of Research on Computing in Education, 33(2), 162–177.
Preece, J. (2000). Sociability and usability in online communities: Determining and measuring success. Behaviour & Information Technology, 20(5), 347–356.
Renninger, K., & Shumar, W. (2002). Building virtual communities: Learning and change in cyberspace. New York: Cambridge University Press.
Rheingold, H. (2000). The virtual community: Homesteading on the electronic frontier. Cambridge, MA: The MIT Press.
Rice-Lively, M. L. (1994). Wired warp and woof: An ethnographic study of a networking class. Internet Research, 4(4), 20–35.
Ricketts, J., Wolfe, F. H., Norvelle, E., & Carpenter, E. H. (2000). Multimedia: Asynchronous distributed education––a review and case study. Social Science Computer Review, 18(2), 132–146.
Romiszowsky, A., & Mason, R. (1996). Computer-mediated communication. In D. Jonassen (Ed.), Handbook of research for educational communications and technology (pp. 438–456). New York: Simon & Schuster Macmillan.
Rovai, A. P. (2001). Building classroom community at a distance: A case study. Educational Technology Research and Development, 49(4), 33–48.
Rovai, A. P. (2002). Sense of community, perceived cognitive learning, and persistence in aynchronous learning networks. The Internet and Higher Education, 5(4), 319–332.
Russell, M., & Ginsburg, L. (1999). Learning online: Extending the meaning of community: A review of three programs from the Southeastern United States (Report No. NCAL-TR99-01). National Center for Adult Literacy, Philadelphia, PA. SouthEast and Islands Regional Technology in Education Consortium, Greensboro, NC (ERIC Document Reproduction Service No. ED 437540).
Scardamalia, M., Bereiter, C., Brett, C., Burtis, P. J., Calhoun, C., & Smith Lea, N. (1992). Educational applications of a networked communal database. Interactive Learning Environments, 2(1), 45–71.
Schön, D. (1983). The reflective practitioner. New York: Basic Books.
Scriven, M. (1991). Prose and cons about goal-free evaluation. American Journal of Evaluation, 12(1), 55–62.
Shea, P., Swan, K., Li, C. S., & Pickett, A. (2005). Developing learning community in online asynchronous college course: The role of teaching presence. Journal of Asynchronous Learning Networks, 9(4), 59–82.
Soller, A., & Lesgold, A. (2003). A computational approach to analyzing online knowledge sharing interaction. In H. U. Hoppe, M. F. Verdejo, & J. Kay (Eds.), Artificial intelligence in education (pp. 253–260). Amsterdam, Netherlands: IOS Press.
Spitzer, W., & Wedding, K. (1995). Labnet: An international electronic community for professional development. Computer Education, 24(3), 247–255.
Squire, K. D., & Johnson, C. B. (2000). Supporting distributed communities of practice with interactive television. Educational Technology Research and Development, 48(1), 23–43.
Steinkuehler, C. A. (2004). Learning in massively multiplayer online games. In Proceedings of the 6th international conference on learning sciences, Los Angeles, California (pp. 521–528).
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park: Sage.
Swan, K. (2002). Building learning communities in online courses: The importance of interaction. Education, Communication & Information, 2(1), 23–49.
Thomas, M. J. W. (2002). Learning within incoherent structures: The space of online discussion forums. Journal of Computer Assisted Learning 18(3): 351–366.
Tu, C. H., & Corry, M. (2002). ELearning communities. The Quarterly Review of Distance Education, 3(2), 207–218.
Upcraft, M. L., & Schuh, J. H. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.
Wang, M., Sierra, C., & Folger, T. (2003). Building a dynamic online learning community among adult learners. Education Media International, 40(1/2), 49–61.
Watkins, D. S. (1997). The development, pilot test, assessment, and evaluation of a computerized online internet community system at programs for higher education. Unpublished doctoral dissertation, Nova Southeastern University, Fort Lauderdale.
Wegerif, R. (1998). The social dimension of asynchronous learning networks, Journal of Asynchronous Learning Networks, 2(1). Retrieved April 15, 2004, from http://www.aln.org/alnweb/journal/vo12issue1/wegerif.htm.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ke, F., Hoadley, C. Evaluating online learning communities. Education Tech Research Dev 57, 487–510 (2009). https://doi.org/10.1007/s11423-009-9120-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-009-9120-2