An Evaluation Methodology for Concept Maps Mined from Lecture Notes: An Educational Perspective

  • Thushari AtapattuEmail author
  • Katrina Falkner
  • Nickolas Falkner
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 510)


Concept maps are effective tools that assist learners in organising and representing knowledge. Recent efforts in the area of concept mapping work toward semi- or fully automated approaches to extract concept maps from various text sources such as text books. The motivation for this research is twofold: novice learners require substantial assistance from experts in constructing their own maps, introducing additional hurdles, and alternatively, the workload required by academics in manually constructing expert maps is substantial and repetitive. A key limitation of an automated concept map generation is the lack of an evaluation framework to measure the quality of concept maps. The most common evaluation mechanism is measuring the overlap between machine-generated elements (e.g. concepts) with expert maps using relevancy measures such as precision and recall. However, in the educational context, the majority of knowledge presented is relevant to the learner, resulting in a large amount of information being retrieved for knowledge organisation. Therefore, this paper introduces a machine-based approach to evaluate the relative importance of knowledge by comparing with human judgment. We introduce three ranking models and conclude that the structural features are positively correlated with human experts (rs ~ 1) for courses with rich content and good structure (well-fitted).


Concept map mining Evaluation methodology Lecture notes 


  1. Atapattu, T., Falkner, K., Falkner, N.: Automated extraction of semantic concepts from semi-structured data: supporting computer-based education through the analysis of lecture notes. In: Liddle, S.W., Schewe, K.-D., Tjoa, A.M., Zhou, X. (eds.) DEXA 2012, Part I. LNCS, vol. 7446, pp. 161–175. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  2. Atapattu, T., Falkner, K., Falkner, N.: Acquisition of triples of knowledge from lecture notes: a natural language processing approach. In: Proceedings of the 7th International conference on Educational Data Mining, London, United Kingdom (2014)Google Scholar
  3. Alves, A., Pereira, F., Cardoso, F.: Automatic reading and learning from text. In: International Symposium on Artificial Intelligence (2002)Google Scholar
  4. Ausubel, D., Novak, J., Hanesian, H.: Educational Psychology: A Cognitive View. Holt, Rinehart and Winston, New York (1978)Google Scholar
  5. Chen, N., Kinshuk, Wei, C.: Mining e-learning domain concept map from academic articles. Comput. Educ. 50, 1009–1021 (2008)Google Scholar
  6. Coffey, J., Carnot, M., Feltovich, P., Feltovich, J., Hoffman, R., Canas, A., Novak, J.: A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. The Chief of Naval Education and Training (2003)Google Scholar
  7. Dali, L., Rusu, D., Fortuna, B., Mladenic, D., Grobelnik, M.: Question answering based on semantic graphs. In: Language and Technology Conference, Poznan, Poland (2009)Google Scholar
  8. Gouli, E., Gogoulou, A., Papanikolaou, K., Grigoriadou, M.: COMPASS: an adaptive web-based concept map assessment tool. In: Proceedings of the First International Conference on Concept Mapping (2004)Google Scholar
  9. Kinchin, I.: Developing PowerPoint handouts to support meaningful learning. Br. J. Educ. Technol. 37(4), 647–650 (2006)CrossRefGoogle Scholar
  10. Klein, D., Manning, C.: Accurate unlexicalized parsing. In: Proceedings of the 41st Meeting of the Association for Computational Linguistics, pp. 423–430 (2003)Google Scholar
  11. Leake, D., Maguitman, A., Reichherzer, T.: Understanding knowledge models: modelling assessment of concept importance in concept maps. In: Proceedings of CogSc (2004)Google Scholar
  12. Manning, C., Raghavan, P., Schutze, H.: Introduction to Information Retrieval. Cambridge University Press, Cambridge (2008)CrossRefzbMATHGoogle Scholar
  13. Novak, J., Gowin, D.: Learning How to Learn. Cambridge University Press, New York/Cambridge (1984)CrossRefGoogle Scholar
  14. Olney, A.M., Graesser, A., Person, N.: Question generation from Concept maps. In: Special issue on Question generation, Dialogue and Discourse (2012)Google Scholar
  15. Salton, G., McGill, M.: Introduction to Modern Information Retrieval. McGraw-Hill Inc., New York (1986)zbMATHGoogle Scholar
  16. Sleator, D., Temperly, D.: Parsing English with a links grammar. In: Third International Workshop on Parsing Technologies (1993)Google Scholar
  17. Villalon, J., Calvo, R.: Concept map mining: a definition and a framework for its evaluation. In: International Conference on Web Intelligence and Intelligent Agent Technology (2008)Google Scholar
  18. Zouaq, A., Nkabou, R.: Evaluating the generation of domain ontologies in the knowledge puzzle project. IEEE Trans. Knowl. Data Eng. 21, 1559–1572 (2009)CrossRefGoogle Scholar
  19. Zouaq, A., Gasevic, D., Hatala, M.: Voting theory for concept detection. In: Simperl, E., Cimiano, P., Polleres, A., Corcho, O., Presutti, V. (eds.) ESWC 2012. LNCS, vol. 7295, pp. 315–329. Springer, Heidelberg (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Thushari Atapattu
    • 1
    Email author
  • Katrina Falkner
    • 1
  • Nickolas Falkner
    • 1
  1. 1.School of Computer ScienceUniversity of AdelaideAdelaideAustralia

Personalised recommendations