Skip to main content
SpringerLink
Log in

Assessment of Discussion in Learning Contexts

  • Chapter
  • First Online:
Innovative Assessment of Collaboration

Part of the book series: Methodology of Educational Measurement and Assessment ((MEMA))

Abstract

This chapter reports on our efforts to develop automated assessment of collaborative processes, in order to support effective participation in learning-relevant discussion. This chapter presents resources that can be offered to this assessment community by machine learning and computational linguistics. The goal is to raise awareness of opportunities for productive synergy between research communities. In particular, we present a three-part pipeline for expediting automated assessment of collaborative processes in discussion in order to trigger interventions, with pointers to sharable software and other opportunities for support. The pipeline begins with computational modeling of analytic categories, motivated by the learning sciences and linguistics. It also includes a data infrastructure for uniform representation of heterogeneous data sources that enables association between process and outcome variables. Finally, it includes supportive technologies that can be triggered through real-time, automated application of that analysis in order to achieve positive impact on outcomes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://dance.cs.cmu.edu.

  2. 2.

    https://discoursedb.github.io/.

  3. 3.

    https://www.edx.org/course/data-analytics-learning-utarlingtonx-link5-10x.

References

  • Ahuja, R. K., Magnanti, T. L., & Orlin, J. B. (1993). Network flows: Theory, algorithms, and applications. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Ai, H., Kumar, R., Nguyen, D., Nagasunder, A., & Rosé, C. P. (2010). Exploring the effectiveness of social capabilities and goal alignment in computer supported collaborative learning. Lecture Notes in Computer Science, 6095, 134–143.

    Article  Google Scholar 

  • Azmitia, M., & Montgomery, R. (1993). Friendship, transactive dialogues, and the development of scientific reasoning. Social Development, 2(3), 202–221.

    Article  Google Scholar 

  • Berkowitz, M., & Gibbs, J. (1979). A preliminary manual for coding transactive features of dyadic discussion. Unpublished manuscript, Marquette University, Milwaukee, WI, 2(1), 6–1.

    Google Scholar 

  • Chan, C. K. K. (2013). Collaborative knowledge building: Towards a knowledge creation perspective. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan & A. M. O’Donnell (Eds.), International handbook of collaborative learning (pp. 437–461). New York, NY: Taylor and Francis.

    Google Scholar 

  • Chinn, C. A., & Clark, D. B. (2013). Learning through collaborative argumentation. In C. E. Hmelo-Silver, C. A. Chinn, C. K. K. Chan & A. M. O’Donnell (Eds.), International handbook of collaborative learning (pp. 437–461). New York, NY: Taylor and Francis.

    Google Scholar 

  • Clark, H., & Bresnan, J. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine & S. D. Teasley (Eds.), Perspectives on socially shared cognition, (pp. 127–149). Washington, DC: American Psychological Association.

    Google Scholar 

  • Clark, H., & Schaefer, E. (1989). Contributing to discourse. Cognitive Science, 13(2), 259–294.

    Article  Google Scholar 

  • de Lisi, R., & Golbeck, S. L. (1999). Implications of the Piagetian theory for peer learning. In A. M. O’Donnell & A. King (Eds.), Cognitive perspectives on peer learning (pp. 3–37). Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Dyke, G., Howley, I., Adamson, D., Kumar, R., & Rosé, C. P. (2013). Towards academically productive talk supported by conversational agents. In D. D. Suthers, K. Lund, C. P. Rosé, C. Teplovs & N. Law (Eds.), Productive multivocality in the analysis of group interactions (pp. 459-476). New York, NY: Springer.

    Google Scholar 

  • Ferschke, O., Yang, D., & Rosé, C. P. (2015). A lightly supervised approach to role identification in Wikipedia talk page discussions. In International AAAI conference on web and social media. Retrieved March 29, 2016 from http://www.aaai.org/ocs/index.php/ICWSM/ICWSM15/paper/view/10634

  • Ferrucci, D., & Lally, A. (2004). UIMA: An architectural approach to unstructured information processing in the corporate research environment. Natural Language Engineering, 10(3–4), 327–348.

    Article  Google Scholar 

  • Gweon, G., Jain, M., Mc Donough, J., Raj, B., & Rosé, C. P. (2013). Measuring prevalence of other-oriented transactive contributions using an automated measure of speech style accommodation. International Journal of Computer Supported Collaborative Learning, 8(2), 245–265.

    Article  Google Scholar 

  • Gweon, G., Kane, A., & Rosé, C. P. (2011, July). Facilitating knowledge transfer between groups through idea co-construction processes. Paper presented at the annual meeting of the Interdisciplinary Network for Group Research (INGRoup), Minneapolis, MN.

    Google Scholar 

  • Howley, I., Mayfield, E., & Rosé, C. P. (2011). Missing something? Authority in collaborative learning. Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL 2011 Conference Proceedings—Long Papers, 9th International Computer-Supported Collaborative Learning Conference (Vol. 1, pp. 366–373). Retrieved from http://www.scopus.com/inward/record.url?eid=2-s2.0-84858400613&partnerID=tZOtx3y1

  • Howley, I., Adamson, D., Dyke, G., Mayfield, E., Beuth, J., & Rosé, C. P. (2012). Group composition and intelligent dialogue tutors for impacting students’ self-efficacy. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 7315, 551–556.

    Google Scholar 

  • Howley, I., Tomar, G., Yang, D., Ferschke, O., & Rosé, C. P. (2015). Alleviating the negative effect of up and downvoting on help seeking in MOOC discussion forums. In Proceedings of the 17th international conference on artificial intelligence in education (AIED 2015), IOS Press.

    Google Scholar 

  • Howley, I., Mayfield, E., & Rosé, C. P. (2013). Linguistic analysis methods for studying small groups. In C. Hmelo-Silver, A. O’Donnell, C. Chan & C Chin (Eds.) International handbook of collaborative learning. London: Taylor and Francis, Inc.

    Google Scholar 

  • Jo, Y., Tomar, G., Ferschke, O., Rosé, C. P., & Gaesevic, D. (in press). Pipeline for expediting learning analytics and student support from data in social learning. Proceedings of the 6th international learning, analytics, and knowledge conference (LAK16) (poster).

    Google Scholar 

  • Joshi, M., & Rosé, C. P. (2007, October). Using transactivity in conversation summarization in educational dialog. Proceedings of the ISCA special interest group on speech and language technology in education workshop (SLaTE), Farmington, PA. Retrieved from http://www.isca-speech.org/archive_open/archive_papers/slate_2007/sle7_053.pdf

  • Kumar, R., Beuth, J., & Rosé, C. P. (2011). Conversational strategies that support idea generation productivity in groups. Connecting computer-supported collaborative learning to policy and practice: CSCL 2011 Conference Proceedings—Long Papers, 9th International Computer-Supported Collaborative Learning Conference (Vol. 1. pp 398–405).

    Google Scholar 

  • Martin, J. R., & Rose, D. (2003). Working with discourse: Meaning beyond the clause. New York, NY: Continuum.

    Google Scholar 

  • Martin, J. R., & White, P. R. R. (2005). The language of evaluation: Appraisal in English. New York: Palgrave/Macmillan.

    Book  Google Scholar 

  • Mayfield, E., Laws, B., Wilson, I., & Rosé, C. P. (2014). Automating annotation of information flow for analysis of clinical conversation. Journal of the American Medical Informatics Association, 21(1), 122–128.

    Article  Google Scholar 

  • Mayfield, E., & Rosé, C. P. (2011). Recognizing authority in dialogue with an integer linear programming constrained model. In Proceedings of the 49th annual meeting of the association for computational linguistics, 1018–1026. Retrieved from http://www.aclweb.org/anthology-new/P/P11/P11-1102.pdf

  • Resnick, L., Asterhan, C., & Clarke, S. (2015). Socializing intelligence through academic talk and dialogue. Washington, DC: American Educational Research Association.

    Book  Google Scholar 

  • Rosé, C. P., Carlson, R., Yang, D., Wen, M., Resnick, L., Goldman, P., & Sherer, J. (2014). Social factors that contribute to attrition in MOOCs. In Proceedings of the first ACM conference on learning @ Scale. - L@S ’14, 197–198. Retrieved from http://dl.acm.org/citation.cfm?id=2556325.2567879

  • Scardamalia, M., & Bereiter, C. (1993). Technologies for knowledge-building discourse. Communications of the ACM, 36(5), 37–41.

    Article  Google Scholar 

  • Scardamalia, M., & Bereiter, C. (2006). Knowledge building: Theory, pedagogy, and technology. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 97–118). New York, NY: Cambridge University Press.

    Google Scholar 

  • Strijbos, J. W. (2011). Assessment of (computer-supported) collaborative learning. IEEE Transactions on Learning Technologies, 4(1), 59–73.

    Article  Google Scholar 

  • Suthers, D. (2006). Technology affordances for inter-subjective meaning making: A research agenda for CSCL. International Journal of Computer Supported Collaborative Learning, 1(3), 315–337.

    Article  Google Scholar 

  • Teasley, S. D. (1997). Talking about reasoning: How important is the peer in peer collaborations? In L. B. Resnick, R. Saljo, C. Pontecorvo & B. Burge (Eds.), Discourse, tools, and reasoning: Situated cognition and technologically supported environments (pp. 361–384). Heidelberg, Germany: Springer-Verlag.

    Google Scholar 

  • van Aalst, J. (2009). Distinguishing between knowledge sharing, knowledge creating, and knowledge construction discourses. International Journal of Computer Supported Collaborative Learning, 4(3), 259–288.

    Article  Google Scholar 

  • Wang, X., Wen, M., & Rosé, C. P. (2016). Towards triggering higher-order thinking behaviors in MOOCs. In Proceedings of the 6th international learning, analytics, and knowledge conference (LAK16).

    Google Scholar 

  • Wang, X., Yang, D., Wen, M., Koedinger, K. R., & Rosé, C. P. (2015). Investigating how student’s cognitive behavior in MOOC discussion forums affect learning gains. In Proceedings of the 8th international educational data mining conference (EDM15).

    Google Scholar 

  • Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46, 71–95.

    Article  Google Scholar 

  • Wen, M., Yang, D., & Rosé, D. (2014a). Linguistic reflections of student engagement in massive open online courses. In Proceedings of the international conference on weblogs and social media. Retrieved from http://www.aaai.org/ocs/index.php/ICWSM/ICWSM14/paper/view/8057/8153

  • Wen, M., Yang, D., & Rosé, C. P. (2014b). Sentiment analysis in MOOC discussion forums: What does it tell us? In Proceedings of the 7th international educational data mining conference (EDM14).

    Google Scholar 

  • Yang, D., Wen, M., & Rose, C. P. (2014a). Towards identifying the resolvability of threads in MOOCs. In Association for computational linguistics (Ed.), Proceedings of the EMNLP workshop on modeling large scale social interaction in massively open online courses (pp. 21–31). Doha, Qatar: Association for Computational Linguistics.

    Google Scholar 

  • Yang, D., Wen, M., & Rosé, C. P. (2014b). Peer influence on attrition in massively open online courses. In Proceedings of the 7th international educational data mining conference (EDM14).

    Google Scholar 

  • Yang, D. & Rosé, C. P. (2014c). Constrained question recommendation in MOOCs via submodality, Proceedings of the 2014 ACM international conference on information and knowledge management, pp. 1987–1990.

    Google Scholar 

  • Yang, D., Wen, M., & Rosé, C. P. (2015). Weakly supervised role identification in teamwork interactions. In Proceedings of the 53rd annual meeting of the association for computational linguistics.

    Google Scholar 

Download references

Acknowledgements

This work was funded in part by NSF grants ACI-1443068 and OMA-0836012 and funding from Google.

Author information

Authors and Affiliations

  1. Language Technologies Institute and Human-Computer Interaction Institution, Carnegie Mellon University, Pittsburgh, PA, USA

    Carolyn Penstein Rosé, Iris Howley, Miaomiao Wen, Diyi Yang & Oliver Ferschke

Authors
  1. Carolyn Penstein Rosé
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Iris Howley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miaomiao Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Diyi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Oliver Ferschke
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carolyn Penstein Rosé .

Editor information

Editors and Affiliations

  1. ACT , Iowa City, Iowa, USA

    Alina A. von Davier

  2. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Mengxiao Zhu

  3. Research and Development Division, Educational Testing Service, Princeton, New Jersey, USA

    Patrick C. Kyllonen

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Rosé, C.P., Howley, I., Wen, M., Yang, D., Ferschke, O. (2017). Assessment of Discussion in Learning Contexts. In: von Davier, A., Zhu, M., Kyllonen, P. (eds) Innovative Assessment of Collaboration. Methodology of Educational Measurement and Assessment. Springer, Cham. https://doi.org/10.1007/978-3-319-33261-1_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-33261-1_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-33259-8

  • Online ISBN: 978-3-319-33261-1

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation