Abstract
The objective of the present study was to establish reliability and validity evidence of a simple procedure to measuring conceptual knowledge, the Concept Retrieval Technique (CRT). The CRT procedure requires test takers to freely retrieve concepts from memory they feel are relevant to a given topic, with raters awarding one mark for each correctly retrieved concept when matched against a target word list. Four studies were conducted. In Study 1 (N = 73), inter-rater agreement for the marking procedure of the CRT was determined by means of Cohen’s kappa. The kappa was κ = .85. Study 2 (N = 164) explored how consistent inter-rater agreement was across different subjects, age groups, and raters. Mean kappa was κ = .92 suggesting that the CRT can reliably be marked. Study 3 was concerned with establishing the convergent and discriminant validity of the CRT (N = 55). The correlation between the CRT scores and scores on essay-type items was r = .69 and between assessments that measured other ability domains r = .04 to .21 (n.s.). In Study 4, an experiment was conducted to determine the construct validity of the CRT (N = 45). Participants either acquired, or did not acquire, new knowledge of a particular topic and this manipulation was consistently reflected in their CRT scores. The results of the studies suggest that the CRT is a reliable and valid procedure to measure conceptual knowledge. Implications for education are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We acknowledge that not only students’ knowledge is subjected to educational assessment, there are other purposes as well, such as the assessment of one’s ability to apply knowledge or the assessment of one’s skills, e.g., the twenty-first century skills set (Chalkiadaki, 2018). However, this paper is concerned with knowledge and thus we limit our considerations to the acquisition of semantic knowledge.
It should be noted at this point, that strictly speaking it cannot be ruled out that students write down a concept (or concepts) of which its meaning is not well understood. The concept(s) could have been mentioned repeatedly in the learning materials, thus signifying its importance. However, in light of semantic network theory it is less likely that students can produce meaningful links to related and relevant concepts if the concept in question is not well understood, thus reducing the chance of guessing or rote reproduction of concepts. Nonetheless, this is a valid concern to be cognizant of, particularly when target word lists are relatively short.
References
Anatol, T., & Hariharan, S. (2009). Reliability of the evaluation of students' answers to essay-type questions. West Indian Medical Journal, 58(1), 13–16.
Bacon, D. R. (2003). Assessing learning outcomes: A comparison of multiple-choice and short-answer questions in a marketing context. Journal of Marketing Education, 25(1), 31–36.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111(4), 1061–1071. https://doi.org/10.1037/0033-295X.111.4.1061.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.
Chalkiadaki, A. (2018). A systematic literature review of 21st century skills and competencies in primary education. International Journal of Instruction, 11(3), 1–16.
Chiarello, C., Burgess, C., Richards, L., & Pollock, A. (1990). Semantic and associative priming in the cerebral hemispheres: Some words do, some words don't… sometimes, some places. Brain and Language, 38(1), 75–104.
Cicchetti, D. V. (1976). Assessing inter-rater reliability for rating scales: Resolving some basic issues. The British Journal of Psychiatry, 129(5), 452–456.
Collins, A. M., & Loftus, E. F. (1975). A spreading-activation theory of semantic processing. Psychological Review, 82(6), 407–428.
Collins, A. M., & Quillian, M. R. (1969). Retrieval time from semantic memory. Journal of Verbal Learning and Verbal Behavior, 8(2), 240–247.
Daley, B. J., & Torre, D. M. (2010). Concept maps in medical education: An analytical literature review. Medical Education, 44(5), 440–448. https://doi.org/10.1111/j.1365-2923.2010.03628.x.
Downing, S. M. (2012). Assessment of knowledge with written test forms. In G. R. Norman, C. P. Van der Vleuten, & D. I. Newble (Eds.), International handbook of research in medical education (pp. 647–672). Springer.
Edmondson, K. M. (2000). Assessing science understanding through concept maps. In J. J. Mintzes, J. H. Wandersee, & J. D. Novak (Eds.), Assessing science understanding: A human constructivist view (pp. 15–40). Elsevier.
Eppler, M. J. (2006). A comparison between concept maps, mind maps, conceptual diagrams, and visual metaphors as complementary tools for knowledge construction and sharing. Information Visualization, 5(3), 202–210. https://doi.org/10.1057/palgrave.ivs.9500131.
Ericsson, K. A., Charness, N., Feltovich, P. J., & Hoffman, R. R. (2013). The Cambridge handbook of expertise and expert performance. Cambridge University Press.
Falmagne, J. C., Cosyn, E., Doignon, J. P., & Thiéry, N. (2005). The assessment of knowledge, in theory and in practice. In B. Ganter, G. Stumme, & R. Wille (Eds.), Formal concept analysis: Foundations and applications (pp. 61–79). Springer.
Garson, J. (2015). Connectionism. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy (spring 2015 edition ed.). http://plato.stanford.edu/archives/spr2015/entries/connectionism/
Gay, L. R. (1980). The comparative effects of multiple-choice versus short-answer tests on retention. Journal of Educational Measurement, 17(1), 45–50.
Gronlund, N. E. (1998). Assessment of student achievement. Allyn & Bacon Publishing.
Hays, G. (2019). Developing a new measure for conceptual knowledge: The concept retrieval techniqueErasmus University Rotterdam]. Rotterdam. https://repub.eur.nl/pub/116394/.
Ho, V., Kumar, R. K., & Velan, G. (2014). Online testable concept maps: Benefits for learning about the pathogenesis of disease. Medical Education, 48(7), 687–697. https://doi.org/10.1111/medu.12422.
Joanisse, M. F., & McClelland, J. L. (2015). Connectionist perspectives on language learning, representation and processing. Wiley Interdisciplinary Reviews-Cognitive Science, 6(3), 235–247. https://doi.org/10.1002/wcs.1340.
Jones, M. N., Willits, J., & Dennis, S. (2015). Models of semantic memory. In J. R. Busemeyer, Z. Wang, J. T. Townsend, & A. Eidels (Eds.), Oxford Handbook of Mathematical and Computational Psychology (Vol. 1). Oxford University Press.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144.
Landauer, T. K., & Dumais, S. T. (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211–240.
Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 33, 363–374.
Leiva, F. M., Ríos, F. J. M., & Martínez, T. L. (2006). Assessment of interjudge reliability in the open-ended questions coding process. Quality & Quantity, 40(4), 519–537.
McClure, J. R., Sonak, B., & Suen, H. K. (1999). Concept map assessment of classroom learning: Reliability, validity, and logistical practicality. Journal of Research in Science Teaching, 36(4), 475–492.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Nicol, D. (2007). E-assessment by design: Using multiple-choice tests to good effect. Journal of Further and Higher Education, 31(1), 53–64.
Novak, J. D. (1990). Concept maps and Vee diagrams: Two metacognitive tools to facilitate meaningful learning. Instructional Science, 19(1), 29–52.
Novak, J. D., & Gowin, D. B. (1984). Learning how to learn. Cambridge University Press.
Plaut, D, C. (1995). Semantic and associative priming in a distributed attractor network. In Proceedings of the 17th Annual Conference of the Cognitive Science Society (pp. 37–42). Pittsburgh, PA, July 1995. Hillsdale, NJ: Lawrence Erlbaum Associates.
Raaijmakers, J. G., & Shiffrin, R. M. (1981). Search of associative memory. Psychological Review, 88(2), 93–134.
Resnick, L. (Ed.). (2018). Knowing, learning, and instruction: Essays in honor of Robert Glaser. Routledge.
Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15(1), 18–39.
Rosch, E., & Mervis, C. B. (1975). Family resemblances: Studies in the internal structure of categories. Cognitive Psychology, 7(4), 573–605.
Rotgans, J. I., & Schmidt, H. G. (2014). Situational interest and learning: Thirst for knowledge. Learning and Instruction, 32, 37–50. https://doi.org/10.1016/j.learninstruc.2014.01.002.
Ruiz-Primo, M. A., & Shavelson, R. J. (1996). Problems and issues in the use of concept maps in science assessment. Journal of Research in Science Teaching, 33(6), 569–600.
Schmidt, H. G., Van Der Molen, H. T., Te Winkel, W. W. R., & Wijnen, W. H. F. W. (2009). Constructivist, problem-based learning does work: A meta-analysis of curricular comparisons involving a single medical school. Educational Psychologist, 44(4), 227–249. https://doi.org/10.1080/00461520903213592.
Smith, E. E., Shoben, E. J., & Rips, L. J. (1974). Structure and process in semantic memory: A featural model for semantic decisions. Psychological Review, 81(3), 214–241.
Tulving, E. (1972). Episodic and semantic memory. In E. Tulving & W. Donaldson (Eds.), Organization of memory (Vol. 1, pp. 381–403). Academic Press.
West, D. C., Pomeroy, J. R., Park, J. K., Gerstenberger, E. A., & Sandoval, J. (2000). Critical thinking in graduate medical education: A role for concept mapping assessment? Journal of the American Medical Association, 284(9), 1105–1110.
Yew, E. H. J., Chng, E., & Schmidt, H. G. (2011). Is learning in problem-based learning cumulative? Advances in Health Sciences Education, 16(4), 449–464. https://doi.org/10.1007/s10459-010-9267-y.
Acknowledgements
The authors would like to thank Henk G. Schmidt, Erasmus University, the Netherlands, for useful suggestions concerning the manuscript.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethics Statement
Ethics approval was obtained from the Institutional Review Board of the University and the Catholic Education, Diocese of Parramatta, in conjunction with the Principal of Parramatta Marist is the granting authority for ethics. Informed written consent was obtained from the participants (and parents/guardians).
Conflict of Interest
The authors have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix 1 Target Word Lists Used in Study 1
Target List
Secondary School Science Topic: Acids
Admissible concepts | Marks |
|---|---|
Corrosive | 1 mark |
Strength | 1 mark |
Concentration | 1 mark |
Dissolve | 1 mark |
Hydrogen ion | 1 mark |
Acidity | 1 mark |
Dissociation | 1 mark |
Target List
Primary School Science Topic: Properties of Light.
Admissible concepts | Marks |
|---|---|
Particles | 1 mark |
Light sources | 1 mark |
Reflects off surfaces | 1 mark |
Pass through materials | 1 mark |
Shadow | 1 mark |
Target List
Secondary School Science Topic: Diffusion.
Admissible concepts | Marks |
|---|---|
Net movement | 1 mark |
Constant random motion | 1 mark |
Fast movement | 1 mark |
Concentration | 1 mark |
High to low concentration | 1 mark |
Equilibrium | 1 mark |
Passive process | 1 mark |
Concentration gradient | 1 mark |
No energy | 1 mark |
Target List
Secondary School Geography Topic: Benefits of Living Near Volcanoes.
Admissible concepts | Marks |
|---|---|
Fertile volcanic soil | 1 mark |
Minerals | 1 mark |
Sulphur | 1 mark |
Precious stones | 1 mark |
Building materials | 1 mark |
Diamonds | 1 mark |
Geothermal energy | 1 mark |
Turbines | 1 mark |
Electricity | 1 mark |
Tourism | 1 mark |
Pompeii | 1 mark |
Target List
Secondary School Chemistry: Periodic Table.
Admissible concepts | Marks |
|---|---|
Atomic number | 1 mark |
Atomic mass | 1 mark |
Electrons | 1 mark |
Reaction | 1 mark |
Formula | 1 mark |
Element | 1 mark |
Protons | 1 mark |
Atom | 1 mark |
Element example | 1 mark |
Shells | 1 mark |
Target List
Secondary School Biology: Infectious Diseases.
Admissible concepts | Marks |
|---|---|
Flu/Influenza | 1 mark |
Bacteria/Bacterial | 1 mark |
Protozoan | 1 mark |
Sporozoans | 1 mark |
Flagellates | 1 mark |
Virus | 1 mark |
Vaccine/Vaccines/Medicine | 1 mark |
Microscopic | 1 mark |
Amoeba | 1 mark |
Contamination/Contagious | 1 mark |
Antibiotics | 1 mark |
STD OR Herpes OR AIDS | 1 mark |
Appendix 2 Online Survey Software Qualtrics for the Topic Acids
Appendix 3 Scoring Example for CRT acids
Appendix 4 The Two Open-ended Questions Used to Assess Students’ Knowledge of the Topic of The Periodic Table in Study 3
Appendix 5 Rubrics Used in Study 3
Rubric for the Literary Task Assessment
Rubric for the Critical and Creative Thinking Assessment
Rubric for the Debate Assessment
Rights and permissions
About this article
Cite this article
Hays, G.J., Hendry, A.L., Tang, M. et al. Psychometric properties of a simple measure of conceptual knowledge: The concept retrieval technique. Curr Psychol 42, 3580–3595 (2023). https://doi.org/10.1007/s12144-021-01669-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-021-01669-9