Asia Information Retrieval Symposium

Information Retrieval Technology pp 227-238 | Cite as

Structure Matters: Adoption of Structured Classification Approach in the Context of Cognitive Presence Classification

  • Zak Waters
  • Vitomir Kovanović
  • Kirsty Kitto
  • Dragan Gašević
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9460)


Within online learning communities, receiving timely and meaningful insights into the quality of learning activities is an important part of an effective educational experience. Commonly adopted methods–such as the Community of Inquiry framework–rely on manual coding of online discussion transcripts, which is a costly and time consuming process. There are several efforts underway to enable the automated classification of online discussion messages using supervised machine learning, which would enable the real-time analysis of interactions occurring within online learning communities. This paper investigates the importance of incorporating features that utilise the structure of online discussions for the classification of “cognitive presence”–the central dimension of the Community of Inquiry framework focusing on the quality of students’ critical thinking within online learning communities. We implemented a Conditional Random Field classification solution, which incorporates structural features that may be useful in increasing classification performance over other implementations. Our approach leads to an improvement in classification accuracy of 5.8 % over current existing techniques when tested on the same dataset, with a precision and recall of 0.630 and 0.504 respectively.


Text classification Conditional random fields Online learning Online discussions 


  1. 1.
    Arbaugh, J., Cleveland-Innes, M., Diaz, S.R., Garrison, D.R., Ice, P., Richardson, J.C., Swan, K.P.: Developing a community of inquiry instrument: testing a measure of the community of inquiry framework using a multi-institutional sample. Internet High. Educ. 11(3–4), 133–136 (2008)CrossRefGoogle Scholar
  2. 2.
    Corich, S., Hunt, K., Hunt, L.: Computerised content analysis for measuring critical thinking within discussion forums. J. e Learn. Knowl. Soc. 2(1), 47–60 (2012)Google Scholar
  3. 3.
    Culotta, A., Kulp, D., McCallum, A.: Gene prediction with conditional random fields (2005)Google Scholar
  4. 4.
    Donnelly, R., Gardner, J.: Content analysis of computer conferencing transcripts. Interact. Learn. Environ. 19(4), 303–315 (2011)CrossRefGoogle Scholar
  5. 5.
    Feng, D., Shaw, E., Kim, J., Hovy, E.: An intelligent discussion-bot for answering student queries in threaded discussions. In: Proceedings of the 11th International Conference on Intelligent User Interfaces, pp. 171–177. ACM (2006)Google Scholar
  6. 6.
    FitzGerald, N., Carenini, G., Murray, G., Joty, S.: Exploiting conversational features to detect high-quality blog comments. In: Butz, C., Lingras, P. (eds.) Canadian AI 2011. LNCS, vol. 6657, pp. 122–127. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  7. 7.
    Garrison, D.R.: Thinking Collaboratively: Learning in a Community of Inquiry. Routledge, New York (2015)Google Scholar
  8. 8.
    Garrison, D.R., Anderson, T., Archer, W.: Critical thinking, cognitive presence, and computer conferencing in distance education. Am. J. Distance Educ. 15(1), 7–23 (2001)CrossRefGoogle Scholar
  9. 9.
    Garrison, D.R., Anderson, T., Archer, W.: The first decade of the community of inquiry framework: a retrospective. Internet High. Educ. 13(1–2), 5–9 (2010a)CrossRefGoogle Scholar
  10. 10.
    Garrison, D.R., Arbaugh, J.: Researching the community of inquiry framework: review, issues, and future directions. Internet High. Educ. 10(3), 157–172 (2007)CrossRefGoogle Scholar
  11. 11.
    Garrison, R., Cleveland-Innes, M., Fung, T.S.: Exploring causal relationships among teaching, cognitive and social presence: student perceptions of the community of inquiry framework. Internet High. Educ. 13(1–2), 31–36 (2010b)CrossRefGoogle Scholar
  12. 12.
    Gašević, D., Adesope, O., Joksimović, S., Kovanović, V.: Externally-facilitated regulation scaffolding and role assignment to develop cognitive presence in asynchronous online discussions. Internet High. Educ. 24, 53–65 (2015)CrossRefGoogle Scholar
  13. 13.
    Graesser, A.C., McNamara, D.S., Kulikowich, J.M.: Coh-metrix providing multilevel analyses of text characteristics. Educ. Res. 40(5), 223–234 (2011)CrossRefGoogle Scholar
  14. 14.
    Jin, W.: Blog comments classification using tree structured conditional random fields. Ph.D. Thesis, University of British Columbia (Vancouver (2012)Google Scholar
  15. 15.
    Kovanovic, V., Gasevic, D., Hatala, M.: Learning analytics for communities of inquiry. J. Learn. Analytics 1(3), 195–198 (2014)Google Scholar
  16. 16.
    Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., Adesope, O.: Analytics of communities of inquiry: effects of learning technology use on cognitive presence in asynchronous online discussions. Internet High. Educ. 27, 74–89 (2015)CrossRefGoogle Scholar
  17. 17.
    Kovanović, V., Joksimović, S., Gašević, D., Hatala, M.: Automated contentanalysis of online discussion transcripts. In: Proceedings of the Workshops at the LAK 2014 Conferenceco-Located with 4th International Conference on Learning Analytics and Knowledge (LAK 2014). Indianapolis, March 2014.
  18. 18.
    Lafferty, J., McCallum, A., Pereira, F.C.N.: Conditional random fields: probabilistic models for segmenting and labeling sequence data (2001)Google Scholar
  19. 19.
    Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174 (1977)CrossRefMathSciNetMATHGoogle Scholar
  20. 20.
    Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The stanford CoreNLP natural language processing toolkit. In: Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55–60 (2014).
  21. 21.
    McCallum, A.K.: Mallet: A machine learning for language toolkit. Technical report (2002).
  22. 22.
    McKlin, T., Harmon, S., Evans, W., Jones, M.: Cognitive presence in web-based learning: a content analysis of students’ online discussions. In: IT Forum, vol. 60 (2002)Google Scholar
  23. 23.
    Ravi, S., Kim, J.: Profiling student interactions in threaded discussions with speech act classifiers. Front. Artif. Intell. Appl. 158, 357 (2007)Google Scholar
  24. 24.
    Shen, D., Sun, J.T., Li, H., Yang, Q., Chen, Z.: Document summarization using conditional random fields. In: IJCAI International Joint Conference on Artificial Intelligence, pp. 2862–2867 (2007)Google Scholar
  25. 25.
    Soller, A., Lesgold, A.: A computational approach to analyzing online knowledge sharing interaction. In: Proceedings of Artificial Intelligence in Education, pp. 253–260 (2003)Google Scholar
  26. 26.
    Sutton, C., McCallum, A.: An introduction to conditional random fields. Found. Trends Mach. Learn. 4(4), 267–373 (2011)CrossRefMATHGoogle Scholar
  27. 27.
    Tausczik, Y.R., Pennebaker, J.W.: The psychological meaning of words: LIWC and computerized text analysis methods. J. Lang. Soc. Psychol. 29(1), 24–54 (2010)CrossRefGoogle Scholar
  28. 28.
    Wang, H., Wang, C., Zhai, C.X., Han, J.: Learning online discussion structures by conditional random fields. In: SIGIR 2011 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, Beijing, pp. 435–444 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Zak Waters
    • 1
  • Vitomir Kovanović
    • 2
  • Kirsty Kitto
    • 1
  • Dragan Gašević
    • 2
  1. 1.Queensland University of TechnologyBrisbaneAustralia
  2. 2.The University of EdinburghEdinburghUK

Personalised recommendations