Solving problems in educational settings, as in daily-life scenarios, involves constantly assessing one’s own confidence in each considered solution. Metacognitive research has exposed cues that may bias confidence judgments (e.g., familiarity with question terms). Typically, metacognitive research methodologies require examining misleading cues one-by-one, while recent research has revealed the integration of multiple cues stemming from the same stimuli. However, this research leaves open important questions about including the weight balance among cues and their changes across task design (e.g., instructions) and/or population characteristics (e.g., background knowledge). The present study presents the Bird’s-Eye View of Cue Integration (BEVoCI) methodology. It is based on hierarchical multiple regression models, allowing efficient exposure of multiple biases at once, their relative weights, and their malleability across task designs and populations. Notably, the BEVoCI can be applied both to planned studies and to existing datasets. I demonstrate its application in both ways. In Experiment 1 and Experiment 2, I introduce two nonverbal problem-solving tasks, the Comparison of Perimeters (CoP) and the novel Missing Tan Task (MTT), while Experiment 3 reanalyzes data collected by others, comprising algebra problems solved by children and adults. The experiments demonstrate exposing biases, their malleability across conditions, and the non-straightforward association between performance improvement and overcoming biases, and the results of Experiment 3 provide strong support for the generalizability of the methodology. Pinpointing sources of bias is essential for guiding educational design efforts.
This is a preview of subscription content,to check access.
Access this article
R command for Experiment 1: model.success<-lme(Accuracy_dv_c ~ 1 + Serial_order_iv_c + Perimeter_area_congruency_iv_c + Basic_shape_area_iv_c + Difference_in_edges_iv_c+ Response_time_iv_c, random = ~1|Username, data = raw_data). All predictors were centralized (denoted by c). The model for predicting confidence was the same with Confidence_dv_c instead of Accuracy_dv_c.
More details about the coding scheme can be found at https://www.jmap.org/JMAPArchives/CurrentVersion/JMAPAI_REGENTS_BOOK_BY_PI_TOPIC.pdf.
Ackerman, R. (2014). The Diminishing Criterion Model for metacognitive regulation of time investment. Journal of Experimental Psychology: General, 143(3), 1349–1368.
Ackerman, R. (2019). Heuristic cues for meta-reasoning judgments: Review and methodology. Psychological Topics, 28(1), 1–20.
Ackerman, R., & Beller, Y. (2017). Shared and distinct cue utilization for metacognitive judgments during reasoning and memorization. Thinking & Reasoning, 23(4), 376–408.
Ackerman, R., & Leiser, D. (2014). The effect of concrete supplements on metacognitive regulation during learning and open-book test taking. British Journal of Educational Psychology, 84(2), 329–348.
Ackerman, R., Leiser, D., & Shpigelman, M. (2013). Is comprehension of problem solutions resistant to misleading heuristic cues? Acta Psychologica, 143(1), 105–112.
Ackerman, R., Parush, A., Nassar, F., & Shtub, A. (2016). Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54, 101–113.
Ackerman, R., & Thompson, V. A. (2015). Meta-reasoning: What can we learn from meta-memory? In A. Feeney & V. Thompson (Eds.), Reasoning as memory (pp. 164–182). Psychology Press.
Ackerman, R., & Thompson, V. A. (2017). Meta-reasoning: Monitoring and control of thinking and reasoning. Trends in Cognitive Sciences, 21(8), 607–617.
Ackerman, R., Yom-Tov, E., & Torgovitsky, I. (2020). Using confidence and consensuality to predict time invested in problem solving and in real-life web searching. Cognition, 199, 104248.
Ackerman, R., & Zalmanov, H. (2012). The persistence of the fluency–confidence association in problem solving. Psychonomic Bulletin & Review, 19(6), 1187–1192.
Allaire-Duquette, G., Babai, R., & Stavy, R. (2019). Interventions aimed at overcoming intuitive interference: Insights from brain-imaging and behavioral studies. Cognitive processing, 20(1), 1–9.
Ariel, R., Karpicke, J. D., Witherby, A. E., & Tauber, S. K. (2021). Do judgments of learning directly enhance learning of educational materials? Educational Psychology Review, 33(2), 693–712.
Ariel, R., Lembeck, N. A., Moffat, S., & Hertzog, C. (2018). Are there sex differences in confidence and metacognitive monitoring accuracy for everyday, academic, and psychometrically measured spatial ability? Intelligence, 70, 42–51.
Baars, M., Vink, S., van Gog, T., de Bruin, A., & Paas, F. (2014). Effects of training self-assessment and using assessment standards on retrospective and prospective monitoring of problem solving. Learning and Instruction, 33, 92–107.
Baars, M., Visser, S., Van Gog, T., de Bruin, A., & Paas, F. (2013). Completion of partially worked-out examples as a generation strategy for improving monitoring accuracy. Contemporary Educational Psychology, 38(4), 395–406.
Baars, M., Wijnia, L., de Bruin, A., & Paas, F. (2020). The relation between students’ effort and monitoring judgments during learning: A meta-analysis. Educational Psychology Review, 32(4), 979–1002.
Bates, K. E., & Farran, E. K. (2021). Mental imagery and visual working memory abilities appear to be unrelated in childhood: Evidence for individual differences in strategy use. Cognitive Development, 60, 101120.
Beckstead, J. W. (2017). The bifocal lens model and equation: Examining the linkage between clinical judgments and decisions. Medical Decision Making, 37(1), 35–45.
bin Mohd Noh, M. F., & bin Mohd Matore, M. E. E. (2019). Brunswik’s Lens Model: This is how to inspire accurate raters. Creative Education, 10(12), 2859.
Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444.
Bohning, G., & Althouse, J. K. (1997). Using tangrams to teach geometry to young children. Early childhood education journal, 24(4), 239–242.
Bröder, A., & Undorf, M. (2019). Metamemory viewed through the judgment lens. Acta Psychologica, 197, 153–165.
Capraro, M. M., & Joffrion, H. (2006). Algebraic equations: Can middle-school students meaningfully translate from words to mathematical symbols? Reading Psychology, 27(2-3), 147–164.
Carney, R. N., & Levin, J. R. (2002). Pictorial illustrations still improve students’ learning from text. Educational Psychology Review, 14(1), 5–26.
Carpenter, J., Sherman, M. T., Kievit, R. A., Seth, A. K., Lau, H., & Fleming, S. M. (2019). Domain-general enhancements of metacognitive ability through adaptive training. Journal of Experimental Psychology: General, 148(1), 51–64.
Castel, A. D. (2008). Metacognition and learning about primacy and recency effects in free recall: The utilization of intrinsic and extrinsic cues when making judgments of learning. Memory & cognition, 36(2), 429–437.
Castro-Alonso, J. C., Ayres, P., & Paas, F. (2019). VAR: A battery of computer-based instruments to measure visuospatial processing. In Visuospatial processing for education in health and natural sciences (pp. 207–229). Springer.
Chen, O., Castro-Alonso, J. C., Paas, F., & Sweller, J. (2018). Undesirable difficulty effects in the learning of high-element interactivity materials. Frontiers in Psychology, 9, 1483.
Cooksey, R. W., Freebody, P., & Davidson, G. R. (1986). Teachers’ predictions of children’s early reading achievement: An application of social judgment theory. American Educational Research Journal, 23(1), 41–64.
de Bruin, A. B., Roelle, J., Carpenter, S. K., & Baars, M. (2020). Synthesizing cognitive load and self-regulation theory: A theoretical framework and research agenda. Educational Psychology Review, 32(4), 903–915.
De Neys, W., Rossi, S., & Houdé, O. (2013). Bats, balls, and substitution sensitivity: Cognitive misers are no happy fools. Psychonomic Bulletin & Review, 20, 269–273.
Delgado, P., Vargas, C., Ackerman, R., & Salmerón, L. (2018). Don’t throw away your printed books: A meta-analysis on the effects of reading media on reading comprehension. Educational Research, 25, 23–38.
Desender, K., Boldt, A., & Yeung, N. (2018). Subjective confidence predicts information seeking in decision making. Psychological Science, 29(5), 761–778.
Dunlosky, J., & Rawson, K. A. (2005). Why does rereading improve metacomprehension accuracy? Evaluating the levels-of-disruption hypothesis for the rereading effect. Discourse Processes, 40(1), 37–55.
Dunlosky, J., & Rawson, K. A. (2012). Overconfidence produces underachievement: Inaccurate self evaluations undermine students’ learning and retention. Learning and Instruction, 22(4), 271–280.
Engelhard, G., Jr., Wang, J., & Wind, S. A. (2018). A tale of two models: Psychometric and cognitive perspectives on rater-mediated assessments using accuracy ratings. Psychological Test and Assessment Modeling, 60(1), 33–52.
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G* Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160.
Fiedler, K., Ackerman, R., & Scarampi, C. (2019). Metacognition: Monitoring and controlling one’s own knowledge, reasoning and decisions. In R. J. Sternberg & J. Funke (Eds.), The Psychology of Human Thought: An Introduction (pp. 89–111). Heidelberg University Publishing.
Fleming, S. M., & Lau, H. C. (2014). How to measure metacognition. Frontiers in human neuroscience, 8, 443.
Galili, H., Babai, R., & Stavy, R. (2020). Intuitive interference in geometry: An eye-tracking study. Mind, Brain, and Education, 14(2), 155–166.
Golan, D. D., Barzillai, M., & Katzir, T. (2018). The effect of presentation mode on children’s reading preferences, performance, and self-evaluations. Computers & Education, 126, 346–358.
Halamish, V. (2018). Can very small font size enhance memory? Memory & cognition, 46(6), 979–993.
Händel, M., de Bruin, A. B., & Dresel, M. (2020). Individual differences in local and global metacognitive judgments. Metacognition and Learning, 15(1), 51–75.
Hart, Y., Mayo, A. E., Mayo, R., Rozenkrantz, L., Tendler, A., Alon, U., & Noy, L. (2017). Creative foraging: An experimental paradigm for studying exploration and discovery. PLoS One, 12(8), e0182133.
Higham, P. A., & Higham, D. P. (2019). New improved gamma: Enhancing the accuracy of Goodman–Kruskal’s gamma using ROC curves. Behavior Research Methods, 51(1), 108–125.
Hines, J. C., Hertzog, C., & Touron, D. R. (2015). Younger and older adults weigh multiple cues in a similar manner to generate judgments of learning. Aging, Neuropsychology, and Cognition, 22(6), 693–711.
Karelaia, N., & Hogarth, R. M. (2008). Determinants of linear judgment: A meta-analysis of lens model studies. Psychological bulletin, 134(3), 404.
Kaufmann, E. (2022). Lens model studies: Revealing teachers’ judgements for teacher education. Journal of Education for Teaching, 1–16.
Kenett, Y. N., Rosen, D. S., Tamez, E. R., & Thompson-Schill, S. L. (2021). Noninvasive brain stimulation to lateral prefrontal cortex alters the novelty of creative idea generation. Cognitive, Affective, & Behavioral Neuroscience, 21(2), 311–326.
Kleitman, S., & Moscrop, T. (2010). Self-confidence and academic achievements in primary-school children: Their relationships and links to parental bonds, intelligence, age, and gender. In A. Efklides & P. Misailidi (Eds.), Trends and Prospects in Metacognition Research. Part 2 (pp. 293–326). Springer.
Koriat, A. (1997). Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126, 349–370.
Koriat, A. (2018). When reality is out of focus: Can people tell whether their beliefs and judgments are correct or wrong? Journal of Experimental Psychology: General, 47(5), 613–631.
Koriat, A., & Ackerman, R. (2010). Metacognition and mindreading: Judgments of learning for Self and Other during self-paced study. Consciousness and Cognition, 19(1), 251–264.
Koriat, A., Ackerman, R., Adiv, S., Lockl, K., & Schneider, W. (2014). The effects of goal-driven and data-driven regulation on metacognitive monitoring during learning: A developmental perspective. Journal of Experimental Psychology: General, 143(1), 386–403.
Koriat, A., Ackerman, R., Lockl, K., & Schneider, W. (2009). The easily learned, easily remembered heuristic in children. Cognitive Development, 24(2), 169–182.
Koriat, A., & Bjork, R. A. (2006). Mending metacognitive illusions: A comparison of mnemonic-based and theory-based procedures. Journal of Experimental Psychology: Learning, Memory, and Cognition, 32(5), 1133–1145.
Koriat, A., & Levy-Sadot, R. (2001). The combined contributions of the cue-familiarity and accessibility heuristics to feelings of knowing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27(1), 34–53.
Koriat, A., Ma'ayan, H., & Nussinson, R. (2006). The intricate relationships between monitoring and control in metacognition: Lessons for the cause-and-effect relation between subjective experience and behavior. Journal of Experimental Psychology: General, 135(1), 36–68.
Koriat, A., Sheffer, L., & Ma'ayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology-General, 131(2), 147–162.
Kornell, N., & Hausman, H. (2017). Performance bias: Why judgments of learning are not affected by learning. Memory & cognition, 45(8), 1270–1280.
Lauterman, T., & Ackerman, R. (2014). Overcoming screen inferiority in learning and calibration. Computers in Human Behavior, 35, 455–463.
Leahy, W., & Sweller, J. (2016). Cognitive load theory and the effects of transient information on the modality effect. Instructional science, 44(1), 107–123.
Lee, J., Lee, J. O., & Collins, D. (2009). Enhancing children’s spatial sense using tangrams. Childhood Education, 86(2), 92–94.
Markovits, H., Thompson, V. A., & Brisson, J. (2015). Metacognition and abstract reasoning. Memory & cognition, 43(4), 681–693.
Metcalfe, J., & Finn, B. (2008). Evidence that judgments of learning are causally related to study choice. Psychonomic Bulletin & Review, 15(1), 174–179.
Michalsky, T. (2021). When to scaffold motivational self-regulation strategies for high school students’ science text comprehension. Frontiers in Psychology, 12, 658027.
Morsanyi, K., Prado, J., & Richland, L. E. (2018). The role of reasoning in mathematical thinking. Thinking & Reasoning, 24(2), 129–137.
Mueller, M. L., Tauber, S. K., & Dunlosky, J. (2013). Contributions of beliefs and processing fluency to the effect of relatedness on judgments of learning. Psychonomic Bulletin &, 20, 378–384.
Norman, G. R., & Eva, K. W. (2010). Diagnostic error and clinical reasoning. Medical education, 44(1), 94–100.
Oyserman, D., Elmore, K., Novin, S., Fisher, O., & Smith, G. C. (2018). Guiding people to interpret their experienced difficulty as importance highlights their academic possibilities and improves their academic performance. Frontiers in Psychology, 9, 781.
Parsons, L. M. (1995). Inability to reason about an object’s orientation using an axis and angle of rotation. Journal of Experimental Psychology: Human Perception and Performance, 21(6), 1259.
Pinheiro, J., Bates, D., DebRoy, S., & Sarkar, D. (2019). R Core Team. 2019. nlme: Linear and nonlinear mixed effects models. R package version 3.1-141. Available at h Ttp://CRAN. R-Project. Org/Package= Nlme.
Raaijmakers, S. F., Baars, M., Paas, F., van Merriënboer, J. J., & Van Gog, T. (2019). Effects of self-assessment feedback on self-assessment and task-selection accuracy. Metacognition and Learning, 14(1), 21–42.
Rau, M. A., & Matthews, P. G. (2017). How to make ‘more’ better? Principles for effective use of multiple representations to enhance students’ learning about fractions. ZDM, 49(4), 531–544.
Rhodes, M. G., & Castel, A. D. (2008). Memory predictions are influenced by perceptual information: evidence for metacognitive illusions. Journal of Experimental Psychology: General, 137(4), 615–625.
Roebers, C. M. (2017). Executive function and metacognition: Towards a unifying framework of cognitive self-regulation. Developmental review, 45, 31–51.
Roll, I., Baker, R. S., & d., Aleven, V., & Koedinger, K. R. (2014). On the benefits of seeking (and avoiding) help in online problem-solving environments. Journal of the Learning Sciences, 23(4), 537–560.
Scheiter, K., Ackerman, R., & Hoogerheide, V. (2020). Looking at mental effort appraisals through a metacognitive lens: Are they biased? Educational Psychology Review, 32(4), 1003–1027.
Serra, M. J., & Dunlosky, J. (2010). Metacomprehension judgements reflect the belief that diagrams improve learning from text. Memory, 18(7), 698–711.
Shepard, R. N., & Metzler, J. (1971). Mental rotation of three-dimensional objects. Science, 171(3972), 701–703.
Sidi, Y., Ophir, Y., & Ackerman, R. (2016). Generalizing screen inferiority-Does the medium, screen versus paper, affect performance even with brief tasks? Metacognition and Learning, 11(1), 15–33.
Sidi, Y., Shpigelman, M., Zalmanov, H., & Ackerman, R. (2017). Understanding metacognitive inferiority on screen by exposing cues for depth of processing. Learning and Instruction, 51, 61–73.
Sidi, Y., Torgovitsky, I., Soibelman, D., Miron-Spektor, E., & Ackerman, R. (2020). You may be more original than you think: Predictable biases in self-assessment of originality. Acta Psychologica, 203, 103002.
Smith, G. C., & Oyserman, D. (2015). Just not worth my time? Experienced difficulty and time investment. Social cognition, 33(2), 85–103.
Stanovich, K. E., West, R. F., & Toplak, M. E. (2016). Toward a rationality quotient (RQ): The comprehensive assessment of rational thinking (CART). In The Thinking Mind (pp. 216–236). Psychology Press.
Stavy, R., & Babai, R. (2008). Complexity of shapes and quantitative reasoning in geometry. Mind, Brain, and Education, 2(4), 170–176.
Suantak, L., Bolger, F., & Ferrell, W. R. (1996). The hard–easy effect in subjective probability calibration. Organizational Behavior and Human Decision Processes, 67(2), 201–221.
Sungkhasettee, V. W., Friedman, M. C., & Castel, A. D. (2011). Memory and metamemory for inverted words: Illusions of competency and desirable difficulties. Psychonomic Bulletin & Review, 18, 973–978.
Sweller, J., van Merriënboer, J. J., & Paas, F. (2019). Cognitive architecture and instructional design: 20 years later. Educational Psychology Review, 31(2), 261–292.
Tauber, S. K., Witherby, A. E., Dunlosky, J., Rawson, K. A., Putnam, A. L., & Roediger, H. L., III. (2018). Does covert retrieval benefit learning of key-term definitions? Journal of Applied Research in Memory and Cognition, 7(1), 106–115.
Thiede, K. W., Anderson, M. C. M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95(1), 66–73.
Thiede, K. W., Wright, K. L., Hagenah, S., Wenner, J., Abbott, J., & Arechiga, A. (2022). Drawing to improve metacomprehension accuracy. Learning and Instruction, 77, 101541.
Thompson, V. A., Prowse Turner, J. A., Pennycook, G., Ball, L., Brack, H., Ophir, Y., & Ackerman, R. (2013). The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition, 128, 237–251.
Tindall-Ford, S., Agostinho, S., & Sweller, J. (2020). Advances in cognitive load theory. Routledge.
Toplak, M. E., West, R. F., & Stanovich, K. E. (2014). Assessing miserly information processing: An expansion of the Cognitive Reflection Test. Thinking & Reasoning, 20(2), 147–168.
Träff, U., Olsson, L., Skagerlund, K., Skagenholt, M., & Östergren, R. (2019). Logical reasoning, spatial processing, and verbal working memory: Longitudinal predictors of physics achievement at age 12–13 years. Frontiers in Psychology, 10, 1929.
Undorf, M., & Ackerman, R. (2017). The puzzle of study time allocation for the most challenging items. Psychonomic Bulletin & Review, 24(6), 2003–2011.
Undorf, M., & Bröder, A. (2020). Cue integration in metamemory judgements is strategic. Quarterly Journal of Experimental Psychology, 73(4), 629–642.
Undorf, M., & Bröder, A. (2021). Metamemory for pictures of naturalistic scenes: Assessment of accuracy and cue utilization. Memory & cognition, 1–18.
Undorf, M., & Erdfelder, E. (2015). The relatedness effect on judgments of learning: A closer look at the contribution of processing fluency. Memory & cognition, 43(4), 647–658.
Undorf, M., Navarro-Báez, S., & Bröder, A. (2022). “You don’t know what this means to me”–Uncovering idiosyncratic influences on metamemory judgments. Cognition, 222, 105011.
Undorf, M., Söllner, A., & Bröder, A. (2018). Simultaneous utilization of multiple cues in judgments of learning. Memory & cognition, 46(4), 507–519.
Undorf, M., & Zimdahl, M. F. (2019). Metamemory and memory for a wide range of font sizes: What is the contribution of perceptual fluency? Journal of Experimental Psychology: Learning, Memory, and Cognition, 45(1), 97–109.
Vuorre, M., & Metcalfe, J. (2022). Measures of relative metacognitive accuracy are confounded with task performance in tasks that permit guessing. Metacognition and Learning, 17(2), 269–291.
Walkington, C., Clinton, V., Ritter, S. N., & Nathan, M. J. (2015). How readability and topic incidence relate to performance on mathematics story problems in computer-based curricula. Journal of Educational Psychology, 107(4), 1051.
Wiggins, N., & Kolen, E. S. (1971). Man versus model of man revisited: The forecasting of graduate school success. Journal of Personality and Social Psychology, 19(1), 100–106.
Yan, V. X., Bjork, E. L., & Bjork, R. A. (2016). On the difficulty of mending metacognitive illusions: A priori theories, fluency effects, and misattributions of the interleaving benefit. Journal of Experimental Psychology: General, 145(7), 918–933.
Zak, Y., Tapiro, H., Alicia, T. J., Parmet, Y., Rottem Hovev, M., Taylor, G. S., & Oron-Gilad, T. (2021). Rapid interpretation of temporal–spatial unmanned aerial vehicle (UAV) operational data–RITSUD: Aiding UAV operators with visualizations of patterns-of-life activities. Journal of Cognitive Engineering and Decision Making, 15(4), 135–154.
I would like to thank Meira Ben-Gad for editorial assistance and Paul Feigin for statistical consulting.
This work was supported by the Israel Science Foundation.
The research methodologies were approved by the Behavioral Sciences Research Ethics Committee of the Technion—Israel Institute of Technology (approval number 2020-015). Informed consent was collected from all participants at the outset of each experiment.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Ackerman, R. Bird’s-Eye View of Cue Integration: Exposing Instructional and Task Design Factors Which Bias Problem Solvers. Educ Psychol Rev 35, 55 (2023). https://doi.org/10.1007/s10648-023-09771-z