ReaderBench: Automated evaluation of collaboration based on cohesion and dialogism

  • Mihai DascaluEmail author
  • Stefan Trausan-Matu
  • Danielle S. McNamara
  • Philippe Dessus


As Computer-Supported Collaborative Learning (CSCL) gains a broader usage, the need for automated tools capable of supporting tutors in the time-consuming process of analyzing conversations becomes more pressing. Moreover, collaboration, which presumes the intertwining of ideas or points of view among participants, is a central element of dialogue performed in CSCL environments. Therefore, starting from dialogism and a cohesion-based model of discourse, we propose and validate two computational models for assessing collaboration. The first model is based on a cohesion graph and can be perceived as a longitudinal analysis of the ongoing conversation, thus accounting for collaboration from a social knowledge-building perspective. In the second approach, collaboration is regarded from a dialogical perspective as the intertwining or synergy of voices pertaining to different speakers, therefore enabling a transversal analysis of subsequent discussion slices.


Computer supported collaborative learning Dialogism Cohesion-based discourse analysis Collaboration assessment Learning analytics Automated feedback 



We would like to thank the students of University “Politehnica” of Bucharest who participated in our experiments. This research was partially supported by the FP7 2008–212578 LTfLL project, by the EC H2020 project RAGE (Realising and Applied Gaming Eco-System) Grant agreement No 644187, by the Sectorial Operational Programme Human Resources Development 2007–2013 of the Ministry of European Funds through the Financial Agreement POSDRU/159/1.5/S/134398, by the senior Fulbright scholarship program, as well as by the NSF grants 1417997 and 1418378 to Arizona State University. Moreover, we would like to thank Laura Allen for her support in conducting the statistical analyses, and we are grateful to Cecile Perret for her help in preparing this paper.

Some parts of this paper stem from Dascalu et al. (2014b), Dascalu et al. (2015a, c), nevertheless providing an integrated view and updated results for all performed experiments.


  1. Adams, P.H., & Martell, C.H. (2008). Topic detection and extraction in chat. In IEEE Int. Conf. on Semantic Computing (ICSC 2008) (pp. 581–588). Santa Clara, CA: IEEE.Google Scholar
  2. Austin, J. L. (1962). How to do things with words. Cambridge: Harvard University Press.Google Scholar
  3. Bakhtin, M.M. (1981). The dialogic imagination: Four essays (C. Emerson & M. Holquist, Trans.). Austin and London: The University of Texas Press.Google Scholar
  4. Bakhtin, M.M. (1984). Problems of Dostoevsky’s poetics (C. Emerson, Trans. C. Emerson Ed.). Minneapolis: University of Minnesota Press.Google Scholar
  5. Bakhtin, M.M. (1986). Speech genres and other late essays (V. W. McGee, Trans.). Austin: University of Texas.Google Scholar
  6. Bereiter, C. (2002). Education and mind in the knowledge age. Mahwah: Lawrence Erlbaum Associates.Google Scholar
  7. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent Dirichlet allocation. Journal of Machine Learning Research, 3(4–5), 993–1022.Google Scholar
  8. Budanitsky, A., & Hirst, G. (2006). Evaluating WordNet-based measures of lexical semantic relatedness. Computational Linguistics, 32(1), 13–47.CrossRefGoogle Scholar
  9. Cress, U. (2013). Mass collaboration and learning. In R. Luckin, S. Puntambekar, P. Goodyear, B. Grabowski, J. Underwood, & N. Winters (Eds.), Handbook of design in educational technology (pp. 416–424). New York: Routledge.Google Scholar
  10. Dascalu, M. (2014). Analyzing discourse and text complexity for learning and collaborating, Studies in Computational Intelligence (Vol. 534). Switzerland: Springer.CrossRefGoogle Scholar
  11. Dascalu, M., Rebedea, T., & Trausan-Matu, S. (2010). A deep insight in chat analysis: Collaboration, evolution and evaluation, summarization and search. In D. Dochev & D. Dicheva (Eds.), 14th Int. Conf. on Artificial Intelligence: Methodology, Systems, Applications (AIMSA 2010) (pp. 191–200). Varna: Springer.Google Scholar
  12. Dascalu, M., Rebedea, T., Trausan-Matu, S., & Armitt, G. (2011). PolyCAFe: Collaboration and utterance assessment for online CSCL conversations. In H. Spada, G. Stahl, N. Miyake & N. Law (Eds.), 9th Int. Conf. on Computer-Supported Collaborative Learning (CSCL 2011) (pp. 781–785). Hong Kong: ISLS.Google Scholar
  13. Dascalu, M., Trausan-Matu, S., & Dessus, P. (2013a). Cohesion-based analysis of CSCL conversations: Holistic and individual perspectives. In N. Rummel, M. Kapur, M. Nathan & S. Puntambekar (Eds.), 10th Int. Conf. on Computer-Supported Collaborative Learning (CSCL 2013) (pp. 145–152). Madison: ISLS.Google Scholar
  14. Dascalu, M., Dessus, P., Trausan-Matu, S., Bianco, M., & Nardy, A. (2013b). ReaderBench, an environment for analyzing text complexity and reading strategies. In H. C. Lane, K. Yacef, J. Mostow & P. Pavlik (Eds.), 16th Int. Conf. on Artificial Intelligence in Education (AIED 2013) (pp. 379–388). Memphis: Springer.Google Scholar
  15. Dascalu, M., Trausan-Matu, S., & Dessus, P. (2013c). Voices’ inter-animation detection with ReaderBench – Modelling and assessing polyphony in CSCL chats as voice synergy. In 2nd Int. Workshop on Semantic and Collaborative Technologies for the Web, in conjunction with the 2nd Int. Conf. on Systems and Computer Science (ICSCS) (pp. 280–285). Villeneuve d'Ascq, France: IEEE.Google Scholar
  16. Dascalu, M., Dessus, P., Bianco, M., Trausan-Matu, S., & Nardy, A. (2014a). Mining texts, learners productions and strategies with ReaderBench. In A. Peña-Ayala (Ed.), Educational data mining: Applications and trends (pp. 335–377). Switzerland: Springer.Google Scholar
  17. Dascalu, M., Trausan-Matu, S., & Dessus, P. (2014b). Validating the Automated Assessment of Participation and of Collaboration in Chat Conversations. In S. Trausan-Matu, K. E. Boyer, M. Crosby & K. Panourgia (Eds.), 12th Int. Conf. on Intelligent Tutoring Systems (ITS 2014) (pp. 230–235). Honolulu: Springer.Google Scholar
  18. Dascalu, M., Trausan-Matu, S., Dessus, P., & McNamara, D.S. (2015a). Dialogism: A Framework for CSCL and a Signature of Collaboration. In O. Lindwall, P. Häkkinen, T. Koschmann, P. Tchounikine & S. Ludvigsen (Eds.), 11th Int. Conf. on Computer-Supported Collaborative Learning (CSCL 2015) (pp. 86–93). Gothenburg: ISLS.Google Scholar
  19. Dascalu, M., Stavarache, L.L., Dessus, P., Trausan-Matu, S., McNamara, D.S., & Bianco, M. (2015b). Predicting Comprehension from Students’ Summaries. In 17th Int. Conf. on Artificial Intelligence in Education (AIED 2015) (pp. 95–104). Madrid, Spain: Springer.Google Scholar
  20. Dascalu, M., Trausan-Matu, S., Dessus, P., & McNamara, D.S. (2015c). Discourse cohesion: A signature of collaboration. In 5th Int. Learning Analytics & Knowledge Conf. (LAK’15) (pp. 350–354). Poughkeepsie: ACM.Google Scholar
  21. Dong, A. (2005). The latent semantic approach to studying design team communication. Design Studies, 26(5), 445–461.CrossRefGoogle Scholar
  22. Dong, A. (2006). Concept formation as knowledge accumulation: A computational linguistics study. AIE EDAM: Artificial Intelligence for Engineering Design, Analysis, and Manufacturing, 20(1), 35–53.Google Scholar
  23. Dong, A. (2009). The language of design: Theory and computation. New York: Springer.Google Scholar
  24. Fano, R. M. (1961). Transmission of information: A statistical theory of communication. Cambridge: MIT Press.Google Scholar
  25. François, F. (1993). Pratiques de l’oral. Dialogique, jeu et variations de figures du sens. Paris: Nathan Pédagogie.Google Scholar
  26. Galley, M., & McKeown, K. (2003). Improving word sense disambiguation in lexical chaining. In G. Gottlob & T. Walsh (Eds.), 18th International Joint Conference on Artificial Intelligence (IJCAI’03) (pp. 1486–1488). Acapulco: Morgan Kaufmann Publishers, Inc.Google Scholar
  27. Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavioral Research Methods, Instruments, and Computers, 36(2), 193–202.CrossRefGoogle Scholar
  28. Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. London: Longman.Google Scholar
  29. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning (2nd ed.). New York: Springer.CrossRefGoogle Scholar
  30. Holmer, T., Kienle, A., & Wessner, M. (2006). Explicit referencing in learning chats: Needs and acceptance. In W. Nejdl & K. Tochtermann (Eds.), Innovative approaches for learning and knowledge sharing, first European conference on technology enhanced learning, EC-TEL 2006 (pp. 170–184). Crete: Springer.Google Scholar
  31. Hudelot, C. (1994). La circulation interactive du sens dans le dialogue. In A. Trognon, U. Dausendschön-Gay, U. Krafft, & C. Riboni (Eds.), La construction interactive du quotidien (p. 15). Nancy: Presses Universitaires de Nancy.Google Scholar
  32. Hummel, G. K., Houcke, J., van Nadolski, R. J., Hiele, T., van der Kurvers, H., & Löhr, A. (2011). Scripted collaboration in serious gaming for complex learning: Effects of multiple perspectives when acquiring water management skills. British Journal of Educational Technology, 42(6), 1029–1041.CrossRefGoogle Scholar
  33. Joshi, M., & Rosé, C.P. (2007). Using Transactivity in Conversation Summarization in Educational Dialog. In SLaTE Workshop on Speech and Language Technology in Education. Farmington, Pennsylvania, USA.Google Scholar
  34. Jurafsky, D., & Martin, J. H. (2009). An introduction to natural language processing. Computational linguistics, and speech recognition (2nd ed.). London: Pearson Prentice Hall.Google Scholar
  35. Kontostathis, A., Edwards, L., Bayzick, J., McGhee, I., Leatherman, A., & Moore, K. (2009). Comparison of Rule-based to Human Analysis of Chat Logs. In P. Meseguer, L. Mandow & R. M. Gasca (Eds.), 1st International Workshop on Mining Social Media Programme, Conferencia de la Asociación Española Para La Inteligencia Artificial (pp. 12). Seville: Springer.Google Scholar
  36. Koschmann, T. (1999). Toward a dialogic theory of learning: Bakhtin’s contribution to understanding learning in settings of collaboration. In C. M. Hoadley & J. Roschelle (Eds.), Int. Conf. on Computer Support for Collaborative Learning (CSCL’99) (pp. 308–313). Palo Alto: ISLS.Google Scholar
  37. Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. Psychological Review, 104(2), 211–240.CrossRefGoogle Scholar
  38. Lefebvre, H. (2004). Rhythmanalysis: Space, Time and Everyday Life (S. Elden & G. Moore, Trans.). London: Continuum.Google Scholar
  39. Linell, P. (2009). Rethinking language, mind, and world dialogically: Interactional and contextual theories of human sense-making. Charlotte: Information Age Publishing.Google Scholar
  40. Manning, C. D., & Schütze, H. (1999). Foundations of Statistical Natural Language Processing. Cambridge: MIT Press.Google Scholar
  41. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval (Vol. 1). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  42. Marková, I., Linell, P., Grossen, M., & Salazar Orvig, A. (2007). Dialogue in focus groups: Exploring socially shared knowledge. London: Equinox.Google Scholar
  43. McNamara, D. S., Louwerse, M. M., McCarthy, P. M., & Graesser, A. C. (2010). Coh-Metrix: Capturing linguistic features of cohesion. Discourse Processes, 47(4), 292–330.CrossRefGoogle Scholar
  44. McNamara, D. S., Graesser, A. C., McCarthy, P., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  45. Medina, R., & Suthers, D. (2009). Using a contingency graph to discover representational practices in an online collaborative environment. Research and Practice in Technology Enhanced Learning, 4(3), 281–305.CrossRefGoogle Scholar
  46. Miller, G. A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2), 81–97.CrossRefGoogle Scholar
  47. Miller, G. A. (1995). WordNet: A lexical database for English. Communications of the ACM, 38(11), 39–41.CrossRefGoogle Scholar
  48. Mislove, A., Marcon, M., Gummadi, K.P., Druschel, P., & Bhattacharjee, B. (2007). Measurement and analysis of online social networks. In 7th ACM SIGCOMM Conference on Internet measurement (pp. 29–42). San Diego: ACM.Google Scholar
  49. Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., & Manning, C.D. (2010). A Multi-Pass Sieve for Coreference Resolution. In Conference on Empirical Methods in Natural Language Processing (EMNLP ’10) (pp. 492–501). Cambridge: ACL.Google Scholar
  50. Randel, D. M. (Ed.) (2003). The New Harvard Dictionary of Music (4th ed.). Cambridge: Harvard University Press.Google Scholar
  51. Rebedea, T. (2012). Computer-Based Support and Feedback for Collaborative Chat Conversations and Discussion Forums. (Doctoral dissertation), University Politehnica of Bucharest: Bucharest.Google Scholar
  52. Rosé, C. P., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., & Fischer, F. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer Supported Collaborative Learning, 3(3), 237–271.CrossRefGoogle Scholar
  53. Salazar Orvig, A. (1999). Les mouvements du discours: Style, réfèrences et dialogue dans des entretiens cliniques. Paris, France: L’Harmattan.Google Scholar
  54. Scardamalia, M. (2002). Collective cognitive responsibility for the advancement of knowledge. In B. Smith & C. Bereiter (Eds.), Liberal education in a knowledge society (pp. 67–98). Chicago: Open Court Publishing.Google Scholar
  55. Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97–118). New York: Cambridge University Press.Google Scholar
  56. Schiffrin, D. (1987). Discourse markers. London: Cambridge University Press.CrossRefGoogle Scholar
  57. Searle, J. (1969). Speech acts: An essay in the philosophy of language. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  58. Shannon, C.E. (1948). A mathematical theory of communication. The Bell System Technical Journal, 27, 379–423 & 623–656.Google Scholar
  59. Stahl, G. (2006). Group cognition. Computer support for building collaborative knowledge. Cambridge: MIT Press.Google Scholar
  60. Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409–426). Cambridge: Cambridge University Press.Google Scholar
  61. Stent, A. J., & Allen, J. F. (2000). Annotating argumentation acts in spoken dialogue. Rochester: University of Rochester. Computer Science Department.Google Scholar
  62. Stolcke, A., Ries, K., Coccaro, N., Shriberg, J., Bates, R., Jurafsky, D., & Meteer, M. (2000). Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational Linguistics, 26(3), 339–373.CrossRefGoogle Scholar
  63. Suthers, D., & Desiato, C. (2012). Exposing chat features through analysis of uptake between Contributions. In 45th Hawaii International Conference on System Sciences (pp. 3368–3377). Maui: IEEE.Google Scholar
  64. Tapiero, I. (2007). Situation models and levels of coherence. Mahwah: Erlbaum.Google Scholar
  65. Teplovs, C. (2008). The Knowledge Space Visualizer: A Tool for Visualizing Online Discourse. In Workshop on A Common Framework for CSCL Interaction Analysis, ICLS 2008 (pp. 12). Utrecht, Netherland.Google Scholar
  66. Trausan-Matu, S. (2010a). Automatic support for the analysis of online collaborative learning chat conversations. In P. M. Tsang, S. K. S. Cheung, V. S. K. Lee & R. Huang (Eds.), 3rd Int. Conf. on Hybrid Learning (pp. 383–394). Beijing: Springer.Google Scholar
  67. Trausan-Matu, S. (2010b). Computer support for creativity in small groups using chats. Annals of the Academy of Romanian Scientists, Series on Science and Technology of Information, 3(2), 81–90.Google Scholar
  68. Trausan-Matu, S. (2013). From two-part inventions for three voices, to fugues and creative discourse building in CSCL Chats. Unpublished manuscript.Google Scholar
  69. Trausan-Matu, S., & Rebedea, T. (2010). A polyphonic model and system for inter-animation analysis in chat conversations with multiple participants. In A. F. Gelbukh (Ed.), 11th Int. Conf. Computational Linguistics and Intelligent Text Processing (CICLing 2010) (pp. 354–363). New York: Springer.Google Scholar
  70. Trausan-Matu, S., & Stahl, G. (2007). Polyphonic inter-animation of voices in chats. In CSCL’07 Workshop on Chat Analysis in Virtual Math Teams (pp. 12). New Brunwick: ISLS.Google Scholar
  71. Trausan-Matu, S., Stahl, G., & Zemel, A. (2005). Polyphonic inter-animation in collaborative problem solving chats. Philadelphia: Drexel University.Google Scholar
  72. Trausan-Matu, S., Stahl, G., & Sarmiento, J. (2006). Polyphonic Support for Collaborative Learning. In Y. A. Dimitriadis, I. Zigurs & E. Gómez-Sánchez (Eds.), Groupware: Design, Implementation, and Use, 12th International Workshop (CRIWG 2006) (pp. 132–139). Medina del Campo: Springer.Google Scholar
  73. Trausan-Matu, S., Rebedea, T., Dragan, A., & Alexandru, C. (2007a). Visualisation of learners’ contributions in chat conversations. In J. Fong & F. L. Wang (Eds.), Blended learning (pp. 217–226). Singapour: Pearson/Prentice Hall.Google Scholar
  74. Trausan-Matu, S., Stahl, G., & Sarmiento, J. (2007b). Supporting polyphonic collaborative learning. Indiana University Press, E-service Journal, 6(1), 58–74.Google Scholar
  75. Trausan-Matu, S., Dascalu, M., & Dessus, P. (2012). Textual complexity and discourse structure in Computer-Supported Collaborative Learning. In S. A. Cerri, W. J. Clancey, G. Papadourakis & K. Panourgia (Eds.), 11th Int. Conf. on Intelligent Tutoring Systems (ITS 2012) (pp. 352–357). Chania: Springer.Google Scholar
  76. Trausan-Matu, S., Dascalu, M., & Rebedea, T. (2012). A system for the automatic analysis of Computer-Supported Collaborative Learning chats. In C. Giovannella, D.G. Sampson & I. Aedo (Eds.), 12th IEEE Int. Conf. on Advanced Learning Technologies (ICALT 2012) (pp. 95–99). Rome: IEEE.Google Scholar
  77. Trausan-Matu, S., Dascalu, M., & Rebedea, T. (2014). PolyCAFe – Automatic support for the analysis of CSCL chats. International Journal of Computer-Supported Collaborative Learning, 9(2), 127–156. doi: 10.1007/s11412-014-9190-y.CrossRefGoogle Scholar
  78. Upton, G., & Cook, I. (2008). A dictionary of statistics. Oxford: Oxford University Press.CrossRefGoogle Scholar
  79. van Dijk, T. A. (1977). Coherence text and context: Exploration in the semantics and pragmatics of discourse (pp. 93–129). London: Longman.Google Scholar
  80. van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.Google Scholar
  81. Vygotsky, L. S. (1978). Mind in society. Cambridge: Harvard University Press.Google Scholar
  82. Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  83. Wegerif, R. (2006). A dialogical understanding of the relationship between CSCL and teaching thinking skills. International Journal of Computer-Supported Collaborative Learning, 1(1), 143–157.CrossRefGoogle Scholar
  84. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46, 71–95.CrossRefGoogle Scholar
  85. Wu, Z., & Palmer, M. (1994). Verb semantics and lexical selection. In 32nd Annual Meeting of the Association for Computational Linguistics, ACL ’94 (pp. 133–138). New Mexico: ACL.Google Scholar

Copyright information

© International Society of the Learning Sciences, Inc. 2015

Authors and Affiliations

  • Mihai Dascalu
    • 1
    Email author
  • Stefan Trausan-Matu
    • 1
  • Danielle S. McNamara
    • 2
  • Philippe Dessus
    • 3
  1. 1.Department of Computer ScienceUniversity “Politehnica” of BucharestBucharestRomania
  2. 2.Department of PsychologyArizona State UniversityTempeUSA
  3. 3.Laboratory Sciences de l’EducationUniversity Grenoble AlpesGrenoble Cedex 9France

Personalised recommendations