Advance in Detecting Key Concepts as an Expert Model: Using Student Mental Model Analyzer for Research and Teaching (SMART)

  • Min Kyu KimEmail author
  • Cassandra J. Gaul
  • So Mi Kim
  • Reeny J. Madathany
Original research


While key concepts embedded within an expert’s textual explanation have been considered an aspect of expert model, the complexity of textual data makes determining key concepts demanding and time consuming. To address this issue, we developed Student Mental Model Analyzer for Teaching and Learning (SMART) technology that can analyze an expert’ textual explanation to elicit an expert concept map from which key concepts are automatically derived. SMART draws on four graph-based metrics (i.e., clustering coefficient, betweenness, PageRank, and closeness) to automatically filter key concepts from experts’ concept maps. This study investigated which filtering method extract key concepts most accurately. Using 18 expert textual data, we compared the accuracy levels of those four competing filtering methods by referring to four accuracy measures (i.e., precision, recall, F-measure, and N-similarity). The results showed the PageRank filtering method outperformed the other methods in all accuracy measures. For example, on average, PageRank derived 79% of key concepts as accurately as human experts. SMART’s automatic filtering methods can help human experts save time when building an expert model, and it can validate their decision making on a list of key concepts.


SMART Expert model Concept map Key concepts Natural language processing Formative assessment 


Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interests.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or the national research committee.


  1. Allen, L. K., Snow, E. L., & McNamara, D. S. (2015). Are you reading my mind? In Proceedings of the fifth international conference on learning analytics and knowledgeLAK ’15 (pp. 246–254). New York, NY: ACM Press.Google Scholar
  2. Anthonisse, J. M. (1971). The rush in a graph. Amsterdam: Mathematische Centrum.Google Scholar
  3. Anzai, Y., & Yokoyama, T. (1984). Internal models in physics problem-solving. Cognition and Instruction, 1(4), 397–450.Google Scholar
  4. Axelrod, R. (1976). Structure of decision: The cognitive maps of political elites. Princeton: Princeton University Press.Google Scholar
  5. Baroni, M., Dinu, G., & Kruszewski, G. (2014). Don’t count, predict! A systematic comparison of context-counting versus context-predicting semantic vectors. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers) (Vol. 1, pp. 238–247).Google Scholar
  6. Beamer, B., Rozovskaya, A., & Girju, R. (2008). Automatic semantic relation extraction with multiple boundary generations. In Proceedings of AAAI (pp. 824–829). Chicago: AAAI Press.Google Scholar
  7. Boleda, G., & Herbelot, A. (2016). Formal distributional semantics: Introduction to the special issue. Computational Linguistics, 42(4), 619–635.Google Scholar
  8. Brandes, U. (2001). A faster algorithm for betweenness centrality. The Journal of Mathematical Sociology, 25(2), 163–177.Google Scholar
  9. Burstein, J., Tetreault, J., & Madnani, N. (2013). The e-rater automated essay scoring system. In M. D. Shermis & J. Burstein (Eds.), Handbook of automated essay evaluation: Current applications and new directions (pp. 55–67). New York: Routledge.Google Scholar
  10. Carley, K., & Palmquist, M. (1992). Extracting, representing, and analyzing mental models. Social Forces, 70(3), 601–636.Google Scholar
  11. Cho, H., Gay, G., Davidson, B., & Ingraffea, A. (2007). Social networks, communication styles, and learning performance in a CSCL community. Computers & Education, 49(2), 309–329.Google Scholar
  12. Clariana, R., Wallace, P., & Godshalk, V. (2009). Deriving and measuring group knowledge structure from essays: The effects of anaphoric reference. Educational Technology Research and Development, 57(6), 725–737.Google Scholar
  13. Clariana, R. B. (2010). Multi-decision approaches for eliciting knowledge structure. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 41–59). New York: Springer.Google Scholar
  14. Clark, S. (2015). Vector space models of lexical meaning. In S. Lappin & C. Fox (Eds.), Handbook of contemporary semantics (2nd ed., pp. 493–534). Malden, MA: Blackwell.Google Scholar
  15. Cohen, E., Delling, D., Pajor, T., & Werneck, R. F. (2014). Computing classic closeness centrality, at scale. In Proceedings of the second ACM conference on Online social networks (pp. 37–50). ACM.Google Scholar
  16. Collins, A., & Gentner, D. (1987). How people construct mental models. In D. Holland & N. Quinn (Eds.), Cultural models in language and thought (pp. 243–265). Cambridge: Cambridge University Press.Google Scholar
  17. Collins, A. M., & Loftus, E. F. (1975). A spreading–activation theory of semantic processing. Psychological Review, 82, 407–428.Google Scholar
  18. Coronges, K. A., Stacy, A. W., & Valente, T. W. (2007). Structural comparison of cognitive associative networks in two populations. Journal of Applied Social Psychology, 37(9), 2097–2129.Google Scholar
  19. D’Mello, S., Hays, P., Williams, C., Cade, W., Brown, J., & Olney, A. (2010). Collaborative lecturing by human and computer tutors. In V. Aleven, J. Kay, & J. Mostow (Eds.), Intelligent tutoring systems (pp. 178–187). Berlin: Springer.Google Scholar
  20. Emig, J. (1977). Writing as a mode of learning. College Composition and Communication, 28(2), 122–128.Google Scholar
  21. Erk, K. (2012). Vector space models of word meaning and phrase meaning: A survey. Language and Linguistics Compass, 6(10), 635–653.Google Scholar
  22. Freeman, L. C. (1977). A set of measures of centrality based on betweenness. Sociometry, 40(1), 35–41.Google Scholar
  23. Garnham, A. (1987). Mental models as representations of discourse and text (1st ed.). Chichester: Ellis Horwood Ltd.Google Scholar
  24. Garnham, A. (2001). Mental models and the interpretation of anaphora. Hove: Psychology Press.Google Scholar
  25. Girju, R., Nakov, P., Nastase, V., Szpakowicz, S., Turney, P., & Yuret, D. (2009). Classification of semantic relations between nominals. Language Resources and Evaluation, 43(2), 105–121.Google Scholar
  26. Glaser, R., Chi, M. T., & Farr, M. J. (1988). The nature of expertise. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  27. Greeno, J. G. (1989). Situation, mental models, and generative knowledge. In D. Klahr & K. Kotovsky (Eds.), Complex information processing (1st ed., pp. 285–318). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  28. Hage, P., & Harary, F. (1983). Structural models in anthropology. Cambridge: Cambridge University Press.Google Scholar
  29. Hansen, D., Schneiderman, B., & Smith, M. (2010). Analyzing social media networks with NodeXL: Insights from a connected world. Burlington, MA: Morgan Kaufmann.Google Scholar
  30. Ifenthaler, D. (2010). Relational, structural, and semantic analysis of graphical representations and concept maps. Educational Technology Research and Development, 58(1), 81–97.Google Scholar
  31. Janssen, T. M. V. (2012). Montague semantics. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Winter 2012). Retrieved from
  32. Johnson-Laird, P. N. (2005a). Mental models and thoughts. In K. J. Holyoak (Ed.), The Cambridge handbook of thinking and reasoning (pp. 185–208). Cambridge: Cambridge University Press.Google Scholar
  33. Johnson-Laird, P. N. (2005b). The history of mental models. In K. I. Manktelow & M. C. Chung (Eds.), Psychology of reasoning: Theoretical and historical perspectives (pp. 179–212). New York: Psychology Press.Google Scholar
  34. Jonassen, D. H. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85.Google Scholar
  35. Jonassen, D. H., Beissner, K., & Yacci, M. (Eds.). (1993). Structural knowledge: Techniques for representing, conveying, and acquiring structural knowledge. Hillsdale: Lawrence Erlbaum Associates Inc.Google Scholar
  36. Jonassen, D. H., & Henning, P. (1996). Mental models: Knowledge in the head and knowledge in the world. In Proceedings of the 1996 international conference on learning sciences (pp. 433–438). International Society of the Learning Sciences.Google Scholar
  37. Kim, K. (2017). Visualizing first and second language interactions in science reading: A knowledge structure network approach. Language Assessment Quarterly, 14, 328–345.Google Scholar
  38. Kim, K. (2018). An automatic measure of cross-language text structures. Technology, Knowledge and Learning, 23(2), 301–314.Google Scholar
  39. Kim, K., Clarianay, R. B., & Kim, Y. (2018). Automatic representation of knowledge structure: Enhancing learning through knowledge structure reflection in an online course. Educational Technology Research and Development, 67(1), 105–122.Google Scholar
  40. Kim, M. (2012). Theoretically grounded guidelines for assessing learning progress: Cognitive changes in ill-structured complex problem-solving contexts. Educational Technology Research and Development, 60(4), 601–622. Scholar
  41. Kim, M. (2013). Concept map engineering: Methods and tools based on the semantic relation approach. Educational Technology Research and Development, 61(6), 951–978. Scholar
  42. Kim, M. (2015). Models of learning progress in solving complex problems: Expertise development in teaching and learning. Contemporary Educational Psychology, 42, 1–16. Scholar
  43. Kim, M., & Ayer, T. (2019). Learner participation profiles in an asynchronous online collaboration context. Internet and Higher Education, 41, 62–76. Scholar
  44. Kim, M., Zouaq, A., & Kim, S. (2016). Automatic detection of expert models: The exploration of expert modeling methods applicable to technology-based assessment and instruction. Computers & Education, 101, 55–69.Google Scholar
  45. Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction-integration model. Psychological Review, 95(2), 163.Google Scholar
  46. Knoke, D., & Kuklinski, J. H. (1982). Network analysis. Newbury Park: Sage Publications.Google Scholar
  47. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.Google Scholar
  48. Leydesdorff, L. (2007). Betweenness centrality as an indicator of the interdisciplinarity of scientific journals. Journal of the American Society for Information Science and Technology, 58(9), 1303–1319.Google Scholar
  49. Lintean, M., Rus, V., & Azevedo, R. (2012). Automatic detection of student mental models based on natural language student input during metacognitive skill training. International Journal of Artificial Intelligence in Education, 21(3), 169–190.Google Scholar
  50. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge: Cambridge University Press.Google Scholar
  51. Montague, R. (1974). In R. Thomason (Ed.), Formal philosophy: The selected papers of Richard Montague. New Haven: Yale University Press.Google Scholar
  52. Narayanan, V. K. (2005). Causal mapping: An historical overview. In V. K. Narayanan & D. J. Armstrong (Eds.), Causal mapping for research in information technology (pp. 1–19). Hershey: Idea Group Publishing.Google Scholar
  53. Newman, M. (2010). Networks: An introduction. Oxford: Oxford University Press.Google Scholar
  54. Newman, M. E. (2004). Analysis of weighted networks. Physical Review E, 70(5), 56–131.Google Scholar
  55. Nye, B. D., Graesser, A. C., & Hu, X. (2014). AutoTutor and Family: A review of 17 years of natural language tutoring. International Journal of Artificial Intelligence in Education, 24(4), 427–469.Google Scholar
  56. Partee, B. H. (1984). Compositionality. In F. Landman & F. Veltman (Eds.), Varieties of formal semantics: Proceedings of the 4th Amsterdam colloquium (Groningen-Amsterdam Studies in Semantics, No. 3) (pp. 281–311). Dordrecht: Foris.Google Scholar
  57. Pirnay-Dummer, P., Ifenthaler, D., & Spector, J. M. (2010). Highly integrated model assessment technology and tools. Educational Technology Research and Development, 58(1), 3–18.Google Scholar
  58. Powers, D. M. W. (2011). Evaluation: From precision, recall and F-measure to ROC, informedness, markedness and correlation. Journal of Machine Learning Technologies, 2(1), 37–63.Google Scholar
  59. Pretz, J. E., Naples, A. J., & Sternberg, R. J. (2003). Recognizing, defining, and representing problems. In J. E. Davidson & R. J. Sternberg (Eds.), The psychology of problem solving (pp. 3–30). New York, NY: Cambridge University Press.Google Scholar
  60. Quillian, M. R. (1985). Word concepts. A theory and simulation of some basic capabilities. Behavioral Science, 12(5), 410–430.Google Scholar
  61. Rupp, A. A., Gushta, M., Mislevy, R. J., & Shaffer, D. W. (2010). Evidence-centered design of epistemic games: Measurement principles for complex learning environments. The Journal of Technology, Learning and Assessment, 8(4), 4–47.Google Scholar
  62. Rus, V., D’Mello, S., Hu, X., & Graesser, A. (2013). Recent advances in conversational intelligent tutoring systems. AI Magazine, 34(3), 42–54.Google Scholar
  63. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organizations. Norwood, NJ: Ablex Publishing Corp.Google Scholar
  64. Seel, N. (2003). Model-centered learning and instruction. Technology, Instruction, Cognition, and Learning, 1(1), 59–85.Google Scholar
  65. Seel, N. M. (1999). Semiotics and structural leaning theory. Journal of Structural Learning and Intelligent Systems, 14(1), 11–28.Google Scholar
  66. Seel, N. M. (2001). Epistemology, situated cognition, and mental models: “Like a bridge over troubled water”. Instructional Science, 29(4/5), 403–427.Google Scholar
  67. Seel, N. M. (2004). Model-centered learning environments: Theory, instructional design, and effects. In N. M. Seel & S. Dijkstra (Eds.), Curriculum, plans, and processes in instruction design: International perspectives (1st ed., pp. 49–74). New York: Routledge.Google Scholar
  68. Seel, N. M., & Dinter, F. R. (1995). Instruction and mental model progression: Learner-dependent effects of teaching strategies on knowledge acquisition and analogical transfer. Educational Research and Evaluation, 1(1), 4–35.Google Scholar
  69. Shermis, M. D. (2010). Automated essay scoring in a high stakes testing environment. In Innovative assessment for the twenty-first century (pp. 167–185). Boston, MA: Springer US.Google Scholar
  70. Siemens, G., & Baker, R. (2012). Learning analytics and educational data mining. In Proceedings of the 2nd international conference on learning analytics and knowledgeLAK ’12 (pp. 252–254). New York, NY: ACM Press.Google Scholar
  71. Smith, J. P., III, diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A constructivist analysis of knowledge in transition. Journal of the Learning Sciences, 3(2), 115–163.Google Scholar
  72. Snow, R. E. (1990). New approaches to cognitive and conative assessment in education. International Journal of Educational Research, 14(5), 455–473.Google Scholar
  73. Spector, J. M. (2008). Complex domain learning. In H. H. Adelsberger, Kinshuk, & J. M. Pawlowski (Eds.), Handbook of information technologies for education and training (pp. 261–275). Heidelberg: Springer.Google Scholar
  74. Spector, J. M. (2010). Mental representations and their analysis: An epistemological perspective. In D. Ifenthaler, P. Pirnay-Dummer, & N. M. Seel (Eds.), Computer-based diagnostics and systematic analysis of knowledge (pp. 27–40). Boston: Springer US. Scholar
  75. Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141–188.Google Scholar
  76. Villalon, J., & Calvo, R. A. (2011). Concept maps as cognitive visualizations of writing assignments. Journal of Educational Technology & Society, 14(3), 16–27.Google Scholar
  77. Wasserman, S., & Faust, K. (1994). Social network analysis: Methods and applications. Cambridge: Cambridge University Press.Google Scholar
  78. Zimmerman, W. A., Kang, H. B., Kim, K., Gao, M., Johnson, G., Clariana, R., et al. (2018). Computer-automated approach for scoring short essays in an introductory statistics course. Journal of Statistics Education, 26(1), 40–47.Google Scholar
  79. Zouaq, A., Gagnon, M., & Ozell, B. (2010). Semantic analysis using dependency-based grammars and upper-level ontologies. International Journal of Computational Linguistics and Applications, 1(1–2), 85–101. Retrieved from
  80. Zouaq, A., Gasevic, D., & Hatala, M. (2011). Ontologizing concept maps using graph theory. In Proceedings of the 2011 ACM symposium on applied computing (pp. 1687–1692). New York, NY: ACM.Google Scholar
  81. Zouaq, A., Gasevic, D., & Hatala, M. (2012). Linguistic patterns for information extraction in Ontocmaps. In Proceedings of the 3rd international conference on ontology patternsVolume 929 (pp. 61–72).
  82. Zouaq, A., Joksimovic, S., & Gasevic, D. (2013). Ontology learning to analyze research trends in learning analytics publications. In CEUR WS Proceedings of the LAK Data Challenge, 974. Accessible under Accessed 19 Sept 2018.

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Department of Learning SciencesGeorgia State University, College of Education and Human DevelopmentAtlantaUSA
  2. 2.School of Information Science and Learning TechnologiesUniversity of MissouriColumbiaUSA

Personalised recommendations