A Technique for Automatically Scoring Open-Ended Concept Maps

  • Ellen M. Taricani
  • Roy B. Clariana


In this descriptive investigation, we seek to confirm and extend a technique for automatically scoring concept maps. Sixty unscored concept maps from a published dissertation were scored using a computer-based technique adapted from Schvaneveldt (1990) and colleague's Pathfinder network approach. The scores were based on link lines drawn between terms and on the geometric distances between terms. These concept map scores were compared to terminology and comprehension posttest scores. Concept map scores derived from link data were more related to terminology whereas concept map scores derived from distance data were more related to comprehension. A step-by-step description of the scoring technique is presented and the next steps in the development process are discussed.


computer-based assessment Pathfinder networks technology tools 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Ausubel, D. P. (1968). Educational psychology: a cognitive view. New York, NY: Holt, Rinehart, and Winston.Google Scholar
  2. Buzydlowski, J. W. (2002). A comparison of self-organizing maps and Pathfinder networks for the mapping of co-cited authors. Doctoral dissertation completed at Drexel University. Retrieved October 3, 2004, from Scholar
  3. Cañas, A. J., Coffey, J. W., Carnot, M. J., Feltovich, P., Hoffman, R. R., Feltovich, J., & Novak, J. D. (2003). A summary of literature pertaining to the use of concept mapping techniques and technologies for education and performance support. A report for The Chief of Naval Education and Training at Pensacola, Florida. Retrieved October 3, 2004, fromñas/Publications/ConceptMapLitReview/IHMC /LiteratureReviewonConceptMapping.pdf.Google Scholar
  4. Clariana, R. B. (2002). ALA-Mapper, version 1.0. Retrieved October 3, 2002, from Scholar
  5. Clariana, R. B. (2003). Latent semantic analysis writing prompt, used with the permission of Knowledge Analysis Technologies, LLC. Retrieved October 30, 2003 from Scholar
  6. Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a com-puter-based approach for scoring concept maps. International Journal of Instructional Media, 33(3), in press.Google Scholar
  7. Clariana, R. B., & Poindexter, M. T. (2004). The influence of relational and proposition-spe-cific processing on structural knowledge. Presented at the annual meeting of the American Educational Research Association, April 12–16, San Diego, CA. Retrieved April 15, 2004 from Scholar
  8. Crocker, L. A., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart, and Winston.Google Scholar
  9. Dearholt, D. W. & Schvaneveldt, R. W. (1990). Properties of Pathfinder Networks. In Schvaneveldt (Ed.), Pathfinder associative networks: studies in knowledge organization (pp. 1–30). Norwood, NJ: Ablex Publishing Corporation.Google Scholar
  10. Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: The Johns Hopkins Press.Google Scholar
  11. Durso, F. T., & Coggins, K. A. (1990). Graphs in the social and psychological sciences: empirical contributions of Pathfinder. In Schvaneveldt (Ed.), Pathfinder associative networks: Studies in knowledge organization (pp. 31–51). Norwood, NJ: Ablex Publishing Corporation.Google Scholar
  12. Dwyer, F. M. (1972). A guide for improving visualized instruction. State College, PA: Learning Services.Google Scholar
  13. Elman, J. L. (in press). An alternative view of the mental lexicon. Trends in Cognitive Science, in press. Retrieved December 10, 2004 from Scholar
  14. Frase, L. T. (1969). Structural analysis of the knowledge that results from thinking about text. Journal of Educational Psychology, 60(6, Part 2), 1–16.Google Scholar
  15. Goldsmith, T. E., & Davenport, D. M. (1990). Assessing structural similarity of graphs. In Schvaneveldt (Ed.), Pathfinder associative networks: studies in knowledge organization (pp. 75–87). Norwood, NJ: Ablex Publishing Corporation.Google Scholar
  16. Harper, M. E., Hoeft, R. M., Evans, A. W., & Jentsch, F. G. (2004). Scoring concepts maps: Can a practical method of scoring concept maps be used to assess trainee's knowledge structures? Paper presented at the Human Factors and Ergonomics Society 48th Annual Meeting.Google Scholar
  17. Herl, H. E., O'Neil, H. F. Jr., Chunga, G. K. W. K., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15, 315–333.CrossRefGoogle Scholar
  18. Humphreys, M. S., Bain, J. D., & Pike, R. (1989). Different ways to cue a coherent memory system: A theory for episodic, semantic and procedural tasks. Psychological Review, 96, 208–233.CrossRefGoogle Scholar
  19. Jonassen, D. H. (1987). Assessing cognitive structure: verifying a method using pattern notes. Journal of Research and Development in Education, 20(3), 1–13.Google Scholar
  20. Jonassen, D. H., Beissner, K., & Yacci, M. (1993). Structural knowledge: techniques for representing, conveying, and acquiring structural knowledge. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  21. Kinchin, I. M. (2000). Using concept maps to reveal understanding: A two-tier analysis. School Science Review, 81, 41–46.Google Scholar
  22. KNOT. (1998). Knowledge Network and Orientation Tool for the Personal Computer, version 4.3. Retrieved October 3, 2003 from Scholar
  23. Lambiotte, J., & Dansereau, D. (1992). Effects of knowledge maps and prior knowledge on recall of science lecture content. Journal of Experimental Education, 60, 189–201.Google Scholar
  24. Lee, Y., & Nelson, D. (2005). Viewing or visualizing'which concept map strategy works best on problem solving performance? British Journal of Educational Technology, 36(2), 193–203.Google Scholar
  25. Luckie, D. (2001). C-TOOLS: Concept-Connector Tools for online learning in science. A National Science Foundation grant. Retrieved October 3, 2003 from Scholar
  26. McClure, J., Sonak, B., & Suen, H. (1999). Concept map assessment of classroom learning: reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36, 475–492.CrossRefGoogle Scholar
  27. McClelland, J. L., McNaughton, B. L., & O'Reilly, R. C. (1995). Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models. Psychological Review, 102, 419–457.CrossRefGoogle Scholar
  28. Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. Cambridge, UK: Cambridge University Press.Google Scholar
  29. Pike, R. (1984). A comparison of convolution and matrix distributed memory systems. Psychological Review, 91, 281–294.Google Scholar
  30. Robinson, D. H., Corliss, S. B., Bush, A. M., Bera, S. J., & Tomberlin, T. (2003). Optimal presentation of graphic organizers and text: A case for large bites? Educational Technology Research and Development, 51(4), 25–41.Google Scholar
  31. Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33, 569–600.Google Scholar
  32. Ruiz-Primo, M. A., Schultz, S., Li, M., & Shavelson, R. J. (1999). On the cognitive validity of interpretations of scores from alternative concept mapping techniques. (CSE Tech. Rep. No. 503). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).Google Scholar
  33. Rye, J. A., & Rubba, P. A. (2002). Scoring concept maps: an expert map-based scheme weighted for relationships. School Science and Mathematics, 102, 33–44.Google Scholar
  34. Safayeni, F., Derbentseva, N., & Cañas, A. J. (2003). Concept maps: a theoretical note on concepts and the need for cyclic concept maps. Retrieved October 3, 2004 from pdf.Google Scholar
  35. Schau, C., Mattern, N., Zeilik, M., & Teague, K. W. (1999). Select-and-fill-in concept map scores as a measure of students' connected understanding of introductory astronomy. Paper presented at the Annual Meeting of the American Educational Research Association (Montreal, Canada, April). Available from ERIC ED434909.Google Scholar
  36. Shavelson, R. J., Lang, H., & Lewin, B. (1994). On concept maps as potential authentic assessment is science. (CSE Tech. Rep. No. 388). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).Google Scholar
  37. Schvaneveldt, R. W. (1990). Pathfinder associative networks: Studies in knowledge organization. Norwood, NJ: Ablex Publishing Corporation.Google Scholar
  38. Schvaneveldt, R. W., Dearholt, D. W., & Durso, F. T. (1988). Graph theoretic foundations of Pathfinder networks. Computers and Mathematics with Applications, 15, 337– 345.CrossRefGoogle Scholar
  39. Tan, S. C. (2000). The effects of incorporating concept mapping into computer assisted instruction. Journal of Educational Computing Research, 23, 113–131.CrossRefGoogle Scholar
  40. Taricani, E. M. (2002). Effects of the level of generativity in concept mapping with knowledge of correct response feedback on learning. Dissertation Abstracts International, ISBN # 0–493–67101–3.Google Scholar
  41. Yin, Y., Vanides, J., Ruiz-Primo, M. A., Ayala, C. C., & Shavelson, R. J. (2004). A comparison of two construct-a-concept-map science assessments: created linking phrases and selected linking phrases. (CSE Tech. Rep. No. 624). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2006

Authors and Affiliations

  1. 1.College of EducationThe Pennsylvania State University

Personalised recommendations