Skip to main content
SpringerLink
Log in

Identifying patterns in students’ scientific argumentation: content analysis through text mining using Latent Dirichlet Allocation

  • Research Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Constructing scientific arguments is an important practice for students because it helps them to make sense of data using scientific knowledge and within the conceptual and experimental boundaries of an investigation. In this study, we used a text mining method called Latent Dirichlet Allocation (LDA) to identify underlying patterns in students written scientific arguments about a complex scientific phenomenon called Albedo Effect. We further examined how identified patterns compare to existing frameworks related to explaining evidence to support claims and attributing sources of uncertainty. LDA was applied to electronically stored arguments written by 2472 students and concerning how decreases in sea ice affect global temperatures. The results indicated that each content topic identified in the explanations by the LDA— “data only,” “reasoning only,” “data and reasoning combined,” “wrong reasoning types,” and “restatement of the claim”—could be interpreted using the claim–evidence–reasoning framework. Similarly, each topic identified in the students’ uncertainty attributions— “self-evaluations,” “personal sources related to knowledge and experience,” and “scientific sources related to reasoning and data”—could be interpreted using the taxonomy of uncertainty attribution. These results indicate that LDA can serve as a tool for content analysis that can discover semantic patterns in students’ scientific argumentation in particular science domains and facilitate teachers’ providing help to students.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Abdous, M., & He, W. (2011). Using text mining to uncover students’ technology-related problems in live video streaming. British Journal of Educational Technology, 40(5), 40–49.

    Google Scholar 

  • Abdous, M. H., Wu, H., & Yen, C. J. (2012). Using data mining for predicting relationships between online question theme and final grade. Journal of Educational Technology & Society, 15(3), 77–88.

    Google Scholar 

  • Allchin, D. (2012). Teaching the nature of science through scientific errors. Science Education, 96(5), 904–926.

    Google Scholar 

  • Akçapınar, G. (2015). How automated feedback through text mining changes plagiaristic behavior in online assignments. Computers & Education, 87, 123–130.

    Google Scholar 

  • Aufschnaiter, C. V., Erduran, S., Osborne, J., & Simon, S. (2008). Arguing to learn and learning to argue: Case studies of how students’ argumentation relates to their scientific knowledge. Journal of Research in Science Teaching, 45, 101–131.

    Google Scholar 

  • Beggrow, E. P., Ha, M., Nehm, R. H., Pearl, D., & Boone, W. J. (2014). Assessing scientific practices using machine-learning methods: How closely do they match clinical interview performance? Journal of Science Education and Technology, 23(1), 160–182.

    Google Scholar 

  • Berland, L. K., & Reiser, B. J. (2009). Making sense of argumentation and explanation. Science Education, 93, 26–55.

    Google Scholar 

  • Bricker, L. A., & Bell, P. (2008). Conceptualizations of argumentation from science studies and the learning sciences and their implications for the practices of science education. Science Education, 92, 473–493.

    Google Scholar 

  • Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., & Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. In Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in neural information processing systems (pp. 288–296). New York: Curran Associates.

    Google Scholar 

  • Chen, B. (2014). Visualizing semantic space of online discourse: The Knowledge Forum case. In Proceedings of the 4th international conference on Learning Analytics and Knowledge (LAK ’14), 24–28 March 2014, Indianapolis, IN (pp. 271–272). New York: ACM. https://doi.org/10.1145/2567574.2567595.

  • Chen, B., Chen, X., & Xing, W. (2015). “Twitter archeology” of learning analytics and knowledge conferences. In Proceedings of the 5th international conference on Learning Analytics and Knowledge (LAK ’15), 16–20 March 2015, Poughkeepsie, NY (pp. 340–349). New York: ACM. https://doi.org/10.1145/2723576.2723584.

  • Chen, N. S., Wei, C. W., & Chen, H. J. (2008). Mining e-Learning domain concept map from academic articles. Computers & Education, 50(3), 1009–1021.

    Google Scholar 

  • Clark, D. B., & Sampson, V. (2008). Assessing dialogic argumentation in online environments to relate structure, grounds, and conceptual quality. Journal of Research in Science Teaching, 45, 293–321.

    Google Scholar 

  • de Vries, E., Lund, K., & Baker, M. (2002). Computer-mediated epistemic dialogue: Explanation and argumentation as vehicles for understanding scientific notions. The Journal of the Learning Sciences, 11(1), 63–103.

    Google Scholar 

  • Dikli, S. (2006). An overview of automated scoring of essays. The Journal of Technology, Learning and Assessment, 5(1), 1–36. Retrieved from https://ejournals.bc.edu/index.php/jtla/article/view/1640.

  • Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271–280.

    Google Scholar 

  • Duschl, R. A., Schweingruber, H. A., & Shouse, A. W. (2007). Taking science to school: Learning and teaching science in grades K-8. Washington, DC: National Academy Press.

    Google Scholar 

  • Erduran, S., Simon, S., & Osborne, J. (2004). TAPping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education, 88, 915–933.

    Google Scholar 

  • Ezen-Can, A., Boyer, K. E., Kellogg, S., & Booth, S. (2015). Unsupervised modeling for understanding MOOC discussion forums: A learning analytics approach. Proceedings of the fifth international conference on learning analytics and knowledge (pp. 146–150). New York: ACM. https://doi.org/10.1145/2723576.2723589.

  • Feldman, R. D. (1995). Knowledge discovery in textual databases (KDT). Paper presented at the first international conference on knowledge discovery and data mining (KDD-95), August 20–21, Montreal, Canada.

  • Harish, B. S., Guru, D. S., & Manjunath, S. (2010). Representation and classification of text documents: A brief review. International Journal of Computer Applications, Special Issue on RTIPPR, 2(1), 110–119.

    Google Scholar 

  • Hotho, A., Nürnberger, A., & Paaß, G. (2005). A brief survey of text mining. Ldv Forum, 20(1), 19–62.

    Google Scholar 

  • Huff, J. D., & Nietfeld, J. L. (2009). Using strategy instruction and confidence judgments to improve metacognitive monitoring. Metacognition and Learning, 4(2), 161–176.

    Google Scholar 

  • Hung, J. (2012). Trends of e-learning research from 2000 to 2008: Use of text mining and bibliometrics. British Journal of Educational Technology, 43(1), 5–16.

    Google Scholar 

  • Janasik, N., Honkela, T., & Bruun, H. (2009). Text mining in qualitative research. Organizational Research Methods, 12(3), 436–460.

    Google Scholar 

  • Kahneman, D., Slovic, S. P., Slovic, P., & Tversky, A. (Eds.). (1982). Judgment under uncertainty: Heuristics and biases. Cambridge: Cambridge University Press.

    Google Scholar 

  • Kuhn, D. (1993). Science as argument: Implications for teaching and learning scientific thinking. Science Education, 77(3), 319–337.

    Google Scholar 

  • Lee, H. S., Liu, O. L., Pallant, A., Roohr, K. C., Pryputniewicz, S., & Buck, Z. E. (2014). Assessment of uncertainty-infused scientific argumentation. Journal of Research in Science Teaching, 51(5), 581–605.

    Google Scholar 

  • Lee, H. S., Pallant, A., Pryputniewicz, S., & Lord, T. (2017). Articulating uncertainty attribution as part of critical epistemic practice of scientific argumentation. In Proceedings from CSCL 2017: Making a difference: Prioritizing equity and access in CSCL, 12th international conference on Computer Supported Collaborative Learning (CSCL) 2017 (Vol. 1, pp. 135–142). Philadelphia, PA: International Society of the Learning Sciences.

  • Lin, F. R., Hsieh, L. S., & Chuang, F. T. (2009). Discovering genres of online discussion threads via text mining. Computers & Education, 52(2), 481–495.

    Google Scholar 

  • Liu, O. L., Rios, J. A., Heilman, M., Gerard, L., & Linn, M. C. (2016). Validation of automated scoring of science assessments. Journal of Research in Science Teaching, 53(2), 215–233.

    Google Scholar 

  • McNeill, K. L., Lizotte, D. J., Krajcik, J., & Marx, R. W. (2006). Supporting students’ construction of scientific explanations by fading scaffolds in instructional materials. The Journal of the Learning Sciences, 15(2), 153–191.

    Google Scholar 

  • National Research Council. (2012). A framework for K-12 science education: Practices, crosscutting concepts, and core ideas. Washington, DC: National Academies Press.

    Google Scholar 

  • NGSS Lead States. (2013). Next generation science standards: For states, by states. Washington, DC: National Academies Press.

    Google Scholar 

  • Osborne, J., Erduran, S., & Simon, S. (2004). Enhancing the quality of argumentation in school science. Journal of Research in Science Teaching, 41(10), 994–1020.

    Google Scholar 

  • Rosé, C., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., et al. (2008). Analyzing collaborative learning processes automatically: Exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer-Supported Collaborative Learning, 3(3), 237–271.

    Google Scholar 

  • Sadler, T. D., & Fowler, S. R. (2006). A threshold model of content knowledge transfer for socioscientific argumentation. Science Education, 90, 986–1004.

    Google Scholar 

  • Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science Education, 92, 447–472.

    Google Scholar 

  • Sandoval, W. A. (2003). Conceptual and epistemic aspects of students’ scientific explanations. The Journal of the Learning Sciences, 12(1), 5–51.

    Google Scholar 

  • Sandoval, W. A., & Millwood, K. A. (2005). The quality of students' use of evidence in written scientific explanations. Cognition and Instruction, 23(1), 23–55.

    Google Scholar 

  • Shermis, M. D., & Burstein, J. C. (Eds.). (2003). Automated essay scoring: A cross-disciplinary perspective. New Jersey: Routledge.

    Google Scholar 

  • Sievert, C., & Shirley, K. (2014). LDAvis: A method for visualizing and interpreting topics. Proceedings of the workshop on interactive language learning, visualization, and interfaces (pp. 63–70).

  • Simosi, M. (2003). Using Toulmin’s framework for the analysis of everyday argumentation: Some methodological considerations. Argumentation, 17, 185–202.

    Google Scholar 

  • Southavilay, V., Yacef, K., Reimann, P., & Calvo, R. A. (2013). Analysis of collaborative writing processes using revision maps and probabilistic topic models. Proceedings of the 3rd international conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 38–47). New York: ACM. https://doi.org/10.1145/2460296.2460307.

  • Staley, K. W. (2014). Experimental knowledge in the face of theoretical error. In M. Boumans, G. Hon, & A. C. Petersen (Eds.), Error and uncertainty in scientific practice: History and philosophy of technoscience (pp. 39–56). London: Routledge.

    Google Scholar 

  • Stone, N. J. (2000). Exploring the relationship between calibration and self-regulated learning. Educational Psychology Review, 12(4), 437–475.

    Google Scholar 

  • Tane, J., Schmitz, C., & Stumme, G. (2004). Semantic resource management for the web: An e-learning application. Proceedings of the WWW conference New York, USA, 2004 (pp. 1–10)

  • Tawfik, A. A., Law, V., Ge, X., Xing, W., & Kim, K. (2018). The effect of sustained vs. faded scaffolding on students’ argumentation in ill-structured problem solving. Computers in Human Behavior, 87, 436–449.

    Google Scholar 

  • Toulmin, S. (1958). The uses of argument. New York: Cambridge University Press.

    Google Scholar 

  • Tseng, Y. H., Chang, C. Y., Rundgren, S. N. C., & Rundgren, C. J. (2010). Mining concept maps from news stories for measuring civic scientific literacy in media. Computers & Education, 55(1), 165–177.

    Google Scholar 

  • Yu, C. H., Jannasch-Pennell, A., & Digangi, S. (2011). The qualitative report compatibility between text mining and qualitative research in the perspectives of grounded theory, content analysis, and reliability recommended APA citation compatibility between text mining and qualitative research in the perspect. The Qualitative Report, 16(3), 730–744.

    Google Scholar 

  • Walton, D., Reed, C., & Macagno, F. (2008). Argumentation schemes. New York: Cambridge University Press.

    Google Scholar 

  • Xing, W., & Gao, F. (2018). Exploring the relationship between online discourse and commitment in Twitter professional learning communities. Computers & Education, 126, 388–398.

    Google Scholar 

  • Xing, W., Popov, V., Zhu, G., Horwitz, P., & McIntyre, C. (2019a). The effects of transformative and non-transformative discourse on individual performance in collaborative-inquiry learning. Computers in Human Behavior, 98, 267–276.

    Google Scholar 

  • Xing, W., Tang, H., & Pei, B. (2019b). Beyond positive and negative emotions: Looking into the role of achievement emotions in discussion forums of MOOCs. The Internet and Higher Education, 43, 100690.

    Google Scholar 

  • Zhang, F., & Litman, D. (2015). Annotation and classification of argumentative writing revisions. Proceedings of the tenth workshop on innovative use of NLP for building educational applications (pp. 133–143).

  • Zhu, G., Xing, W., Costa, S., Scardamalia, M., & Pei, B. (2019). Exploring emotional and cognitive dynamics of knowledge building in grades 1 and 2. User Modeling and User-Adapted Interaction, 29(4), 789–820.

    Google Scholar 

Download references

Acknowledgements

This work is supported by the National Science Foundation (NSF) of the United States under grant numbers DRL-1220756, and DRL-1418019. Any opinions, findings, and conclusions or recommendations expressed in this paper, however, are those of the authors and do not necessarily reflect the views of the NSF.

Author information

Authors and Affiliations

  1. School of Teaching & Learning, University of Florida, Gainesville, FL, 32611, USA

    Wanli Xing

  2. The Concord Consortium, West Coast Office, Emeryville, CA, 94608, USA

    Hee-Sun Lee

  3. Faculty of Transdisciplinary Innovation, Univeristy of Technology, Sydney, 15 Broadway, Ultimo, NSW, Australia

    Antonette Shibani

Authors
  1. Wanli Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hee-Sun Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonette Shibani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wanli Xing.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

In the scientific argumentation data about the albedo effect described above, each open-ended response a student generates is a text document. That is, each explanation response can include several topics. So does each uncertainty attribution response. Students’ explanation or uncertainty attribution responses are made up of topics that are made up of words. Therefore, LDA describes a document as a probability distribution of a mixture of topics, each of which is expressed with another probability distribution of words. Topics generated by LDA are a combination of words that contribute to the particular topic based on probabilities. LDA analysis results should be further interpreted by human insights about the context in which documents are generated. In this study, LDA is implemented in the following steps:

Step 1: Pre-processing and data preparation

To remove noise in the text data and format the data for input, pre-processing techniques are applied as follows:

  1. 1)

    All non-letter symbols, numbers, and punctuation are removed.

  2. 2)

    Common stop words such as “a,” “and,” “it,” and “the” are removed.

  3. 3)

    Stemming is performed on the text to convert variations of the same word to a non-changing root word form. For instance, the root word “produc” captures several variations of the word like produced, producing, production, etc.

  4. 4)

    Infrequent words are filtered out.

These steps result in a data corpus in the form of a bag-of-words that takes into account the occurrences of words, but does not consider their ordering. Each text document is represented by a document matrix defined as a vector of the words found in the entire corpus along with the frequency of each word found in the document. This is the input for the LDA algorithm.

Step 2: Determining the number of topics K

For the LDA algorithm to work, the number of topics to be extracted from the data corpus, K, needs to be specified. K can be determined by statistical derivations or informed by researchers’ insights about the documents. An optimum value of K can be determined through the Bayesian model selection and approximated using a harmonic mean estimator. The log-likelihood plots can show the best K value for the text corpus. Note that this statistically derived K value is not the absolute measure for K. Expert judgment based on data, knowledge, and experience can be important in selecting the most meaningful K value. In this study, we combine the log-likelihood method with human judgment to determine the K value for the scientific argument data corpus. Different K values were explored before determining the optimal number. Given the K value, LDA generates a list of relevant words for each topic (topical words) and which topics are contained in each document.

Step 3: Application of the LDA algorithm

Collapsed Gibbs sampling is applied as follows:

  1. 1)

    Each word in the corpus is randomly assigned to the K number of topics. Each topic now constitutes an initial random word distribution based on Dirichlet, which will be iteratively improved in the following steps.

  2. 2)

    For each word in a document,

  3. •

    Compute the proportion of words assigned to a topic in the document, P(topic|document), and the proportion of words assigned to that topic from all documents, P(word|topic).

  4. •

    Reassign the word to a new topic with the probability of P(topic|document) * P(word|topic).

  5. 3)

    Repeat Step 2 numerous times until the topic-word assignments are stabilized.

  6. 4)

    Use the topic assignments to calculate the proportion of topics in each document.

Distinctive words that appear in a topic and do not appear in other topics can be very useful to characterize the topic. If all the documents contain similar words, it is harder to cluster the words into topics, requiring expert evaluation.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xing, W., Lee, HS. & Shibani, A. Identifying patterns in students’ scientific argumentation: content analysis through text mining using Latent Dirichlet Allocation. Education Tech Research Dev 68, 2185–2214 (2020). https://doi.org/10.1007/s11423-020-09761-w

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-020-09761-w

Keywords

Search

Navigation