Skip to main content
SpringerLink
Log in

When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research

  • Published:
International Journal of Computer-Supported Collaborative Learning Aims and scope Submit manuscript

Abstract

Research on computer-supported collaborative learning (CSCL) is often concerned with the question of how scaffolds or other characteristics of learning may affect learners’ social and cognitive engagement. Such engagement in socio-cognitive activities frequently materializes in discourse. In quantitative analyses of discourse, utterances are typically coded, and differences in the frequency of codes are compared between conditions. However, such traditional coding-and-counting-based strategies neglect the temporal nature of verbal data, and therefore provide limited and potentially misleading information about CSCL activities. Instead, we argue that analyses of the temporal proximity, specifically temporal co-occurrences of codes, provide a more appropriate way to characterize socio-cognitive activities of learning in CSCL settings. We investigate this claim by comparing and contrasting a traditional coding-and-counting analysis with epistemic network analysis (ENA), a discourse analysis technique that models temporal co-occurrences of codes in discourse. We apply both methods to data from a study that compared the effects of individual vs. collaborative problem solving. The results suggest that compared to a traditional coding-and-counting approach, ENA provides more insight into the socio-cognitive learning activities of students.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Andrist, S., Collier, W., Gleicher, M., Mutlu, B., & Shaffer, D. W. (2015). Look together: Analyzing gaze coordination with epistemic network analysis. Frontiers in Psychology, 6(1016).

  • Arastoopour, G., Shaffer, D. W., Swiecki, Z., Ruis, A. R., & Chesler, N. C. (2016). Teaching and assessing engineering design thinking with virtual internships and epistemic network analysis. International Journal of Engineering Education, 32(3B), 1492–1501.

    Google Scholar 

  • Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). New York, NY: Cambridge University Press.

    Book  Google Scholar 

  • Bannert, M., Reimann, P., & Sonnenberg, C. (2014). Process mining techniques for analysing patterns and strategies in students’ self-regulated learning. Metacognition and learning, 9(2), 161–185.

    Article  Google Scholar 

  • Bause, I. M., Brich, I. R., Wesslein, A. K., & Hesse, F. W. (2018). Using technological functions on a multi-touch table and their affordances to counteract biases and foster collaborative problem solving. International Journal of Computer-Supported Collaborative Learning, 13(1), 7–33.

    Article  Google Scholar 

  • Chi, M. T. H. (1997). Quantifying qualitative analyses of verbal data: A practical guide. Journal of the Learning Sciences, 6(3), 271–315.

    Article  Google Scholar 

  • Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multilevel analysis. Small Group Research, 36(5), 600–631.

    Article  Google Scholar 

  • Collier, W., Ruis, A. R., & Shaffer, D. W. (2016). Local versus global connection making in discourse. In C. K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: The International Conference of the Learning Sciences (ICLS) 2016, volume 1 (pp. 426–433). Singapore: International Society of the Learning Sciences.

    Google Scholar 

  • Cress, U., & Hesse, W. (2013). Quantitative methods for studying small groups. In C. A. Hmelo-Silver, C. Chinn, C. Chan, & A. M. O’Donnell (Eds.), The international handbook of collaborative learning (pp. 93–111). New York, NY: Routledge.

    Google Scholar 

  • Csanadi, A., Kollar, I., & Fischer, F. (2015). Internal scripts and social context as antecedents of teacher students’ scientific reasoning. Paper presented at the 16th Biennial Conference of the European Association for Research on Learning and Instruction (EARLI), Limassol, Cyprus.

  • Csanadi, A., Kollar, I., & Fischer, F. (2016). Scientific reasoning and problem solving in a practical domain: Are two heads better than one? In C. K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: The International Conference of the Learning Sciences (ICLS) 2016, volume 1 (pp. 50–57). Singapore: International Society of the Learning Sciences.

    Google Scholar 

  • Csanadi, A., Eagan, B., Shaffer, D., Kollar, I., & Fischer, F. (2017). Collaborative and individual scientific reasoning of pre-service teachers: New insights through epistemic network analysis (ENA). In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a Difference: Prioritizing Equity and Access in CSCL, 12 th International Conference on Computer-Supported Collaborative Learning (CSCL) 2017, volume 1 (pp. 215–222). Philadelphia: International Society of the Learning Sciences.

    Google Scholar 

  • Dyke, G., Kumar, R., Ai, H., & Rosé, C. P. (2012). Challenging assumptions: Using sliding window visualizations to reveal time-based irregularities in CSCL processes. In J. van Aalst, K. Thompson, M. J. Jacobson, & P. Reimann (Eds.), The future of learning: Proceedings of the 10th international conference of the learning sciences (ICLS) 2012 (Vol. 1, pp. 363–370). Sydney: International Society of the Learning Sciences.

    Google Scholar 

  • Eagan, B., & Hamilton, E. (2018). Epistemic Network Analysis of an International Digital Makerspace in Africa, Europe, and the US. Paper presented at the annual meeting of the American education research association. New York: NY.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1980). Verbal reports as data. Psychological Review, 87(3), 215–251.

    Article  Google Scholar 

  • Fischer, F., Kollar, I., Ufer, S., Sodian, B., & Hussmann, H. (2014). Pekrun, R.,…Eberle, J. Scientific reasoning and argumentation: Advancing an interdisciplinary research agenda in education. Frontline Learning Research, 5, 28–45.

    Google Scholar 

  • Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137(2), 316–344.

    Article  Google Scholar 

  • Hmelo-Silver, C. E., Jordan, R., Liu, L., & Chernobilsky, E. (2011). Representational tools for understanding complex computer-supported collaborative learning environments. In.: Puntambekar S., Erkens G., Hmelo-silver C. (Eds). Analyzing Interactions in CSCL. Computer-Supported Collaborative Learning, 12, 83–106.

    Google Scholar 

  • Hmelo-Silver, C. E., Jordan, R., & Sinha, S. (2013). Seeing to understand. Using visualizations to understand learning in technology-rich learning environments. In R. Luckin, S. Puntambekar, P. Goodyear, B. Grabowski, J. Underwood, & N. Winters (Eds.), Handbook of Design in Educational Technology (pp. 457–471). New York, NY: Routledge.

    Google Scholar 

  • Jeong, A. (2005). A guide to analyzing message–response sequences and group interaction patterns in computer-mediated communication. Distance Education, 26(3), 367–383.

    Article  Google Scholar 

  • Kapur, M. (2011). Temporality matters: Advancing a method for analyzing problem-solving processes in a computer-supported collaborative environment. International Journal of Computer-Supported Collaborative Learning, 6(1), 39–56.

    Article  Google Scholar 

  • Klahr, D., & Dunbar, K. (1988). Dual space search during scientific reasoning. Cognitive Science, 12(1), 1–48.

    Article  Google Scholar 

  • Kollar, I., Fischer, F., & Slotta, J. D. (2007). Internal and external scripts in computer-supported collaborative inquiry learning. Learning and Instruction, 17(6), 708–721.

    Article  Google Scholar 

  • Lämsä, J., Hämäläinen, R., Koskinen, P., & Viiri, J. (2018). Visualising the temporal aspects of collaborative inquiry-based learning processes in technology-enhanced physics learning. International Journal of Science Education, 40(14), 1697–1717.

    Article  Google Scholar 

  • Marquart, C. L., Hinojosa, C., Swiecki, Z., & Shaffer, D. W. (2018). Epistemic Network Analysis (Version 0.1.0) [Software]. Available from http://app.epistemicnetwork.org

  • Mercer, N. (2008). The seeds of time: Why classroom dialogue needs a temporal analysis. Journal of the Learning Sciences, 17(1), 33–59.

    Article  Google Scholar 

  • Mullins, D., Rummel, N., & Spada, H. (2011). Are two heads always better than one? Differential effects of collaboration on students’ computer-supported learning in mathematics. International Journal of Computer-Supported Collaborative Learning, 6(3), 421–443.

    Article  Google Scholar 

  • Reimann, P. (2009). Time is precious: Variable-and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257.

    Article  Google Scholar 

  • Reimann, P., & Yacef, K. (2013). Using process mining for understanding learning. In R. Luckin, S. Puntambekar, P. Goodyear, B. Grabowski, J. D. M. Underwood, & N. Winters (Eds.), Handbook of design in educational technology (pp. 472–481). New York, NY: Routledge.

    Google Scholar 

  • Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.), Computer supported collaborative learning (pp. 69–97). Berlin: Springer.

    Chapter  Google Scholar 

  • Ruis, A.R., Rosser, A.A., Quandt-Walle, C., Nathwani, J.N., Shaffer, D.W., & Pugh, C.M. (2018). The hands and head of a surgeon: Modeling operative competency with multimodal epistemic network analysis. American Journal of Surgery.

  • Schwaighofer, M., Bühner, M., & Fischer, F. (2017). Executive functions in the context of complex learning: Malleable moderators? Frontline Learning Research, 5(1), 58–75.

    Article  Google Scholar 

  • Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers & Education, 46(3), 223–234.

    Article  Google Scholar 

  • Shaffer, D. W. (2007). How computer games help children learn. New York, NY: Palgrave Macmillan.

    Google Scholar 

  • Shaffer, D. W. (2012). Models of situated action: Computer games and the problem of transfer. In C. Steinkuehler, K. Squire, & S. Barab (Eds.), Games learning, and society: Learning and meaning in the digital age (pp. 403–433). Cambridge, UK: Cambridge University Press.

    Chapter  Google Scholar 

  • Shaffer, D. W. (2017). Quantitative ethnography. Madison, WI: Cathcart.

    Google Scholar 

  • Shaffer, D. W., & Ruis, A. R. (2017). Epistemic network analysis: A worked example of theory-based learning analytics. In C. Lang, G. Siemens, A. F. Wise, & D. Gasevic (Eds.), Handbook of learning analytics (pp. 175–187) Society for Learning Analytics Research.

    Google Scholar 

  • Shaffer, D. W., Hatfield, D., Svarovsky, G., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st century assessment of learning. International Journal of Learning and Media, 1(2), 33–53.

    Article  Google Scholar 

  • Shaffer, D. W., Collier, W., & Ruis, A. R. (2016). A tutorial on epistemic network analysis: Analyzing the structure of connections in cognitive, social, and interaction data. Journal of Learning Analytics, 3(3), 9–45.

    Article  Google Scholar 

  • Shaffer, D. W., & Serlin, R. C. (2004). What Good are Statistics that Don’t Generalize? Educational Researcher, 33(9), 14–25.

    Article  Google Scholar 

  • Siebert-Evenstone, A. L., Arastoopour, G., Collier, W., Swiecki, Z., Ruis, A. R., & Shaffer, D. W. (2016). In search of conversational grain size: Modeling semantic structure using moving stanza windows. In C. K. Looi, J. L. Polman, U. Cress, & P. Reimann (Eds.), Transforming learning, empowering learners: The International Conference of the Learning Sciences (ICLS) 2016, volume 1 (pp. 631–638). Singapore: International Society of the Learning Sciences.

    Google Scholar 

  • Siebert-Evenstone, A., Arastoopour Irgens, G., Collier, W., Swiecki, Z., Ruis, A. R., & Williamson Shaffer, D. (2017). In search of conversational grain size: Modelling semantic structure using moving stanza windows. Journal of Learning Analytics, 4(3), 123–139.

    Article  Google Scholar 

  • Stegmann, K., Wecker, C., Weinberger, A., & Fischer, F. (2012). Collaborative argumentation and cognitive elaboration in a computer-supported collaborative learning environment. Instructional Science, 40(2), 297–323.

    Article  Google Scholar 

  • Strijbos, J. W., Martens, R. L., Prins, F. J., & Jochems, W. M. (2006). Content analysis: What are they talking about? Computers & Education, 46(1), 29–48.

    Article  Google Scholar 

  • Sullivan, S. A., Warner-Hillard, C., Eagan, B. R., Thompson, R., Ruis, A. R., Haines, K., & Jung, H. S. (2018). Using epistemic network analysis to identify targets for educational interventions in trauma team communication. Surgery, 163(4), 938–943.

    Article  Google Scholar 

  • Suthers, D. D. (2005). Technology affordances for intersubjective learning: A thematic agenda for CSCL. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Computer supported collaborative learning 2005: The next 10 years (pp. 662–671). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Suthers, D., & Medina, R. (2011). Tracing interaction in distributed collaborative learning. In.: Puntambekar S., Erkens G., Hmelo-silver C. (Eds). Analyzing Interactions in CSCL. Computer-Supported Collaborative Learning, 12, 341–366.

    Google Scholar 

  • Teasley, S. D. (1995). The role of talk in children’s peer collaborations. Developmental Psychology, 31(2), 207–220.

    Article  Google Scholar 

  • Vogel, F., & Weinberger, A. (2018). Quantifying qualities of collaborative learning processes. In F. Fischer, C. E. Hmelo-Silver, S. R. Goldman, & P. Reimann (Eds.), International handbook of the learning sciences. New York, NY: Routledge.

    Google Scholar 

  • Wegerif, R., & Mercer, N. (1997). Using computer-based text analysis to integrate qualitative and quantitative methods in research on collaborative learning. Language and Education, 11(4), 271–286.

    Article  Google Scholar 

  • Wise, A. F., & Chiu, M. M. (2011). Analyzing temporal patterns of knowledge construction in a role-based online discussion. International Journal of Computer-Supported Collaborative Learning, 6(3), 445–470.

    Article  Google Scholar 

Download references

Acknowledgements

This research was funded in part by the following grants: the Elitenetzwerk Bayern (K-GS-2012-209); the National Science Foundation (DRL-1661036, DRL-1713110), the Wisconsin Alumni Research Foundation, and the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin-Madison. The opinions, findings, and conclusions do not reflect the views of the funding agencies, cooperating institutions, or other individuals.

Author information

Authors and Affiliations

  1. University of Bundeswehr Munich, Werner-Heisenberg-Weg 39, 85577, Neubiberg, Germany

    Andras Csanadi

  2. University of Wisconsin-Madison, Madison, WI, USA

    Brendan Eagan & David Williamson Shaffer

  3. University of Augsburg, Augsburg, Germany

    Ingo Kollar

  4. Ludwig Maximilian University of Munich, Munich, Germany

    Andras Csanadi & Frank Fischer

  5. Aalborg University Copenhagen, Copenhagen, Denmark

    David Williamson Shaffer

Authors
  1. Andras Csanadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brendan Eagan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ingo Kollar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Williamson Shaffer
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Frank Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andras Csanadi.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Csanadi, A., Eagan, B., Kollar, I. et al. When coding-and-counting is not enough: using epistemic network analysis (ENA) to analyze verbal data in CSCL research. Intern. J. Comput.-Support. Collab. Learn 13, 419–438 (2018). https://doi.org/10.1007/s11412-018-9292-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11412-018-9292-z

Keywords

Search

Navigation