Abstract
Besides learners’ awareness of their knowledge, a growing number of studies also emphasise the importance of teachers’ awareness of how well their students perform to adjust their teaching strategies accordingly. Therefore, proposing a multi-layered metacognitive regulatory model in teaching first, we investigated whether estimation type, item difficulty, and class performance affect teachers’ judgment accuracies ([JAs], i.e., score estimations). Teachers (N=38) of 86 classes made item-by-item and overall estimations of their classes’ test scores (N=2608 sixth-graders native in Turkish) at a PISA-equivalent mathematics test that was developed in the earliest phase of the current long-term research project. The results showed that teachers’ item-by-item estimations were below their classes’ actual performance, unlike their overall estimations. Teachers of low-performance classes were less accurate than those of high-performance classes. These teachers also showed the clearest underestimation for the easy questions, whereas teachers of high-performance classes overestimated their classes’ scores for the difficult questions. This dissociation implied that the teachers ‘must have’ primarily used their perceptions about their classes (e.g., classes’ existing performance) as a mnemonic judgment cue rather than item difficulty as an external cue when making their score estimations. The implications of the results were discussed in the light of existing literature and suggestions for prospective research were given.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Change history
26 January 2023
The abstract part of the paper seems having all of the apostrophes (?) replaced with the question marks (?). This should be corrected
Notes
The model can also involve -at least at a theoretical level- a fourth possible regulation, which is regarding the learners’ metacognitive regulation of their teachers and is also not mentioned in Thiede and his colleagues’ model (2019). However, we did not explicitly include this theoretically possible regulation loop that would emerge from the learner’s meta-level to the teacher’s object-object mainly because students do not have a clearly defined supervisory role in the educational settings just like their teachers. Nevertheless, learners may influence the teachers’ meta-levels (i.e., their metacognitive goals regarding their teaching objectives) by, for instance, course evaluations and/or various feedbacks; therefore, we show this as a bidirectional influence in the model rather than a fourth regulatory loop (see dashed arrows between learner’s and teacher’s meta-levels in Fig. 1).
The school administrators’ and parents’ collaborations were so high that the number of parents who did not give consent was almost none.
Being a part of this long-term research project, a separate study on the metacognitive monitoring performance of these students measured with type-2 signal detection theory’s calculation method along with their pre-test and immediate post-test score estimations has been reported elsewhere (see, Basokcu & Guzel 2022 for this earlier report). Since the present paper is primarily on teachers’ JAs, we didn’t repeat any of the findings regarding the students’ monitoring and score estimation performance in this current report. Therefore, we would like to refer the readers who would be curious about how sixth-grade mathematics learners behave metacognitively before, during, and immediately after the test is completed to the earlier report.
The students made algebraic calculations on the test booklet to reach the correct answers, e.g., volume calculation, for the short-answer questions. Hence, only the ‘final answers’ given to the short-answer questions were scored in terms of their accuracies. Also, as indicated in the test booklet, all of the three sub-items in the true-false question must have been answered correctly to count this true-false question was accurately responded.
The teachers in the present study confirmed that neither themselves nor their students who took part in the current study had already taken part in the test development phase of the research project earlier as an invigilator or as a participant.
Absenteeism on the test-taking day was almost none and the number of students who didn’t wish to complete the test was negligible.
Note that the test was not prepared by the teachers themselves and neither the students nor the teachers in the present study were instructed that the questions varied in terms of difficulty.
References
Bard, G., & Weinstein, Y. (2017). The effect of question order on evaluations of test performance: can the bias dissolve? Quarterly Journal of Experimental Psychology (2006), 70(10), 2130–2140. https://doi.org/10.1080/17470218.2016.1225108.
Basokcu, T. O., & Canpolat, A. (2018). Ankor Test Deseninde Bilissel Tanı Modeli Ortuk Yetenek Sınıfları ile Test Esitleme Calismasi (A Test Equation Study with Latent Classes following Cognitive Diagnostic Model as an Anchor Test Design). 6th International Congress on Measurement and Evaluation in Education and Psychology, 384–388.
Basokcu, T. O., & Guzel, M. A. (2022). Beyond counting the correct responses: metacognitive monitoring and score estimations in mathematics. Psychology in the Schools, 59(6), 1105–1121. https://doi.org/10.1002/pits.22665.
Bates, C., & Nettelbeck, T. (2001). Primary school teachers’ judgements of reading achievement. Educational Psychology, 21(2), 177–187. https://doi.org/10.1080/01443410020043878.
Birch, S. A. J. (2005). When knowledge is a curse: children’s and adults’ reasoning about mental states. Current Directions in Psychological Science, 14(1), 25–29. https://doi.org/10.1111/j.0963-7214.2005.00328.x.
Brendefur, J. L., Thiede, K., Strother, S., Jesse, D., & Sutton, J. (2016). The effects of professional development on elementary students’ mathematics achievement. Journal of Curriculum and Teaching, 5(2), 95. https://doi.org/10.5430/jct.v5n2p95.
Carr, M. (2010). The importance of metacognition for conceptual change and strategy use in mathematics. Metacognition, strategy use, and instruction (pp. 176–197). Guilford Press.
Conejo, R., Guzmán, E., Perez-de-la-Cruz, J. L., & Barros, B. (2014). An empirical study on the quantitative notion of task difficulty. Expert Systems with Applications, 41(2), 594–606. https://doi.org/10.1016/j.eswa.2013.07.084.
Darling-Hammond, L. (2000). Teacher quality and student achievement. Education Policy Analysis Archives, 8, 1. https://doi.org/10.14507/epaa.v8n1.2000.
Doherty, J., & Conolly, M. (1985). How accurately can primary school teachers predict the scores of their pupils in standardised tests of attainment? A study of some non-cognitive factors that influence specific judgements. Educational Studies, 11(1), 41–60. https://doi.org/10.1080/0305569850110105.
Duffy, G. G., Miller, S., Parsons, S., & Meloth, M. (2009). Teachers as metacognitive professionals. Handbook of Metacognition in Education. Routledge.
Dunlosky, J., & Metcalfe, J. (2009a). Judgments of learning. Metacognition. Sage Publications, Inc.
Dunlosky, J., & Metcalfe, J. (2009b). Metacognition. Sage Publications, Inc.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: a new area of cognitive-developmental inquiry. American Psychologist, 34(10), 906–911. https://doi.org/10.1037/0003-066X.34.10.906.
Fussell, S. R., & Krauss, R. M. (1992). Coordination of knowledge in communication: Effects of speakers’ assumptions about what others know. Journal of Personality and Social Psychology, 62(3), 378–391. https://doi.org/10.1037//0022-3514.62.3.378.
Gabriele, A. J., Joram, E., & Park, K. H. (2016). Elementary mathematics teachers’ judgment accuracy and calibration accuracy: do they predict students’ mathematics achievement outcomes? Learning and Instruction, 45, 49–60. https://doi.org/10.1016/j.learninstruc.2016.06.008.
Gralewski, J., & Karwowski, M. (2019). Are teachers’ ratings of students’ creativity related to students’ divergent thinking? A meta-analysis. Thinking Skills and Creativity, 33. https://doi.org/10.1016/j.tsc.2019.100583
Gronlund, N. E. (1982). Constructing achievement tests. Prentice-Hall.
Gronlund, N. E. (1993). Measurement and evaluation in teaching (10th ed.). New Jersey: The Macmillan Company.
Haladyna, T. M. (2004). Developing and validating multiple-choice test items, 3rd ed (pp. xi, 306). Lawrence Erlbaum Associates Publishers.
Harris, D. N., & Sass, T. R. (2011). Teacher training, teacher quality and student achievement. Journal of Public Economics, 95(7–8), 798–812. https://doi.org/10.1016/j.jpubeco.2010.11.009.
Hecht, S. A., & Greenfield, D. B. (2002). Explaining the predictive accuracy of teacher judgments of their students’ reading achievement: the role of gender, classroom behavior, and emergent literacy skills in a longitudinal sample of children exposed to poverty. Reading and Writing, 15(7), 789–809. https://doi.org/10.1023/A:1020985701556.
Impara, J. C., & Plake, B. S. (1998). Teachers’ ability to estimate item difficulty: a test of the assumptions in the Angoff standard setting method. Journal of Educational Measurement, 35(1), 69–81.
Jackson, A., & Greene, R. L. (2014). Impression formation of tests: retrospective judgments of performance are higher when easier questions come first. Memory & Cognition, 42(8), 1325–1332. https://doi.org/10.3758/s13421-014-0439-5.
Jenkins, L. N., & Demaray, M. K. (2016). Teachers’ judgments of the academic achievement of children with and without characteristics of inattention, impulsivity, and hyperactivity. Contemporary School Psychology, 20(2), 183–191. https://doi.org/10.1007/s40688-015-0073-7.
Kaiser, J., Retelsdorf, J., Südkamp, A., & Möller, J. (2013). Achievement and engagement: how student characteristics influence teacher judgments. Learning and Instruction, 28, 73–84. https://doi.org/10.1016/j.learninstruc.2013.06.001.
Kaiser, J., Südkamp, A., & Möller, J. (2017). The effects of student characteristics on teachers’ judgment accuracy: disentangling ethnicity, minority status, and achievement. Journal of Educational Psychology, 109(6), 871–888. https://doi.org/10.1037/edu0000156.
Koriat, A. (1997). Monitoring one’s own knowledge during study: a cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349–370. https://doi.org/10.1037/0096-3445.126.4.349.
Machts, N., Kaiser, J., Schmidt, F. T. C., & Möller, J. (2016). Accuracy of teachers’ judgments of students’ cognitive abilities: a meta-analysis. Educational Research Review, 19, 85–103. https://doi.org/10.1016/j.edurev.2016.06.003.
Martínez, J. F., Stecher, B., & Borko, H. (2009). Classroom assessment practices, teacher judgments, and student achievement in mathematics: evidence from the ECLS. Educational Assessment, 14(2), 78–102. https://doi.org/10.1080/10627190903039429.
Mazzoni, G., Cornoldi, C., & Marchitelli, G. (1990). Do memorability ratings affect study-time allocation? Memory & Cognition, 18(2), 196–204. https://doi.org/10.3758/BF03197095.
Nelson, T. O. (1990). Metamemory: A theoretical framework and new findings. In Psychology of Learning and Motivation—Advances in Research and Theory (pp. 125–173). https://doi.org/10.1016/S0079-7421(08)60053-5
Nelson, T. O., & Dunlosky, J. (1991). When people’s judgments of learning (JOLs) are extremely accurate at predicting subsequent recall: the “delayed-JOL effect”. Psychological Science, 2(4), 267–271. https://doi.org/10.1111/j.1467-9280.1991.tb00147.x.
Nelson, T. O., Kruglanski, A. W., & Jost, J. T. (1998). Knowing thyself and others: progress in metacognitive social psychology. Metacognition: cognitive and social dimensions (pp. 69–89). Sage Publications, Inc. https://doi.org/10.4135/9781446279212.n5.
Nelson, T. O., & Narens, L. (1994). Why investigate metacognition?. Metacognition: knowing about knowing (pp. 1–25). The MIT Press.
Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know: imputing one’s own knowledge to others. Psychological Bulletin, 125(6), 737–759. https://doi.org/10.1037/0033-2909.125.6.737.
Nickerson, R. S., Baddeley, A., & Freeman, B. (1987). Are people’s estimates of what other people know influenced by what they themselves know? Acta Psychologica, 64(3), 245–259. https://doi.org/10.1016/0001-6918(87)90010-2.
Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects? Educational Evaluation and Policy Analysis, 26(3), 237–257. https://doi.org/10.3102/01623737026003237.
Oudman, S., van de Pol, J., Bakker, A., Moerbeek, M., & van Gog, T. (2018). Effects of different cue types on the accuracy of primary school teachers’ judgments of students’ mathematical understanding. Teaching and Teacher Education, 76, 214–226. https://doi.org/10.1016/j.tate.2018.02.007.
Prohaska, V. (1994). “I know I’ll get an A”: confident overestimation of final course grades. Teaching of Psychology, 21(3), 141–143. https://doi.org/10.1177/009862839402100303.
Rausch, T., Karing, C., Dörfler, T., & Artelt, C. (2016). Personality similarity between teachers and their students influences teacher judgement of student achievement. Educational Psychology, 36(5), 863–878. https://doi.org/10.1080/01443410.2014.998629.
Ready, D. D., Wright, D. L., Ready, D. D., & Wright, D. L. (2011). Accuracy and inaccuracy in teachers’ perceptions of young children’s cognitive abilities: the role of child background and classroom context. American Educational Research Journal, 48(2), 335–360. https://doi.org/10.3102/0002831210374874.
Rhodes, M. G. (2016). Judgments of learning: methods, data, and theory. The Oxford handbook of metamemory (pp. 65–80). Oxford University Press.
Son, L. K., & Metcalfe, J. (2000). Metacognitive and control strategies in study-time allocation. Journal of Experimental Psychology: Learning Memory and Cognition, 26(1), 204–221. https://doi.org/10.1037/0278-7393.26.1.204.
Südkamp, A., Kaiser, J., & Möller, J. (2012). Accuracy of teachers’ judgments of students’ academic achievement: a meta-analysis. Journal of Educational Psychology, 104(3), 743–762. https://doi.org/10.1037/a0027627.
Svanum, S., & Bigatti, S. (2006). Grade expectations: informed or uninformed optimism, or both? Teaching of Psychology, 33(1), 14–18. https://doi.org/10.1207/s15328023top3301_4.
Thiede, K. W., Anderson, M. C. M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95(1), 66–73. https://doi.org/10.1037/0022-0663.95.1.66.
Thiede, K. W., Brendefur, J. L., Osguthorpe, R. D., Carney, M. B., Bremner, A., Strother, S., Oswalt, S., Snow, J. L., Sutton, J., & Jesse, D. (2015). Can teachers accurately predict student performance? Teaching and Teacher Education, 49, 36–44. https://doi.org/10.1016/j.tate.2015.01.012.
Thiede, K. W., Oswalt, S., Brendefur, J. L., Carney, M. B., & Osguthorpe, R. D. (2019). Teachers’ judgments of student learning of mathematics. In J. Dunlosky, & K. A. E. Rawson (Eds.), The Cambridge Handbook of Cognition and Education (pp. 678–695). Cambridge University Press. https://doi.org/10.1017/9781108235631.027.
Tullis, J. G. (2018). Predicting others’ knowledge: knowledge estimation as cue utilization. Memory & Cognition, 46(8), 1360–1375. https://doi.org/10.3758/s13421-018-0842-4.
Urhahne, D., & Wijnia, L. (2021). A review on the accuracy of teacher judgments. Educational Research Review, 32, 100374. https://doi.org/10.1016/j.edurev.2020.100374.
van de Watering, G., & van der Rijt, J. (2006). Teachers’ and students’ perceptions of assessments: a review and a study into the ability and accuracy of estimating the difficulty levels of assessment items. Educational Research Review, 1(2), 133–147. https://doi.org/10.1016/j.edurev.2006.05.001.
Wauters, K., Desmet, P., & Van Den Noortgate, W. (2012). Item difficulty estimation: an auspicious collaboration between data and judgment. Computers & Education, 58(4), 1183–1193. https://doi.org/10.1016/j.compedu.2011.11.020.
Weinstein, Y., & Roediger, H. L. (2010). Retrospective bias in test performance: providing easy items at the beginning of a test makes students believe they did better on it. Memory & Cognition, 38(3), 366–376. https://doi.org/10.3758/MC.38.3.366.
Zhu, M., Urhahne, D., & Rubie-Davies, C. M. (2018). The longitudinal effects of teacher judgement and different teacher treatment on students’ academic outcomes. Educational Psychology, 38(5), 648–668. https://doi.org/10.1080/01443410.2017.1412399.
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: historical background, Methodological Developments, and future prospects. American Educational Research Journal, 45(1), 166–183. https://doi.org/10.3102/0002831207312909.
Funding
This work was financed by The Scientific and Technological Research Council of Türkiye (TÜBİTAK), Grant No. 115K531.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
We have no known conflict of interest to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Güzel, M.A., Başokçu, T.O. Knowledge about others’ knowledge: how accurately do teachers estimate their students’ test scores?. Metacognition Learning 18, 295–312 (2023). https://doi.org/10.1007/s11409-023-09333-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11409-023-09333-2