Skip to main content
SpringerLink
Log in

Does domain matter? Monitoring accuracy across domains

  • Published:
Metacognition and Learning Aims and scope Submit manuscript

Abstract

Confidence and its accuracy have been most commonly examined in domains such as general knowledge and learning, with less study of other domains, such as applied knowledge and problem solving. Monitoring accuracy in real-world competencies may depend on characteristics of the domain. In this study, we examined whether monitoring accuracy, both calibration (resistance to overconfidence) and resolution (discrimination) indices, are stable within individuals across tasks that represent highly diverse domains. We examined the well-established domain of general knowledge and three understudied applied domains of financial calculation, probability calculation, and the social skill of emotion recognition. In addition, we examined correlations between monitoring accuracy and cognitive abilities (intellectual ability and working memory) and several aggregated judgments regarding each task as a whole (ratings of predicted performance, task difficulty, and effort required) as well as the classic postdictive itemized confidence ratings. We found that resistance to overconfidence (calibration) was significantly positively correlated across tasks, reflecting a confidence trait, but not resolution. We also found that cognitive abilities were more consistently predictive of calibration than of resolution. Aggregated judgments and postdictive confidence were significant predictors of both calibration and resolution, but associations were task specific. Emotion recognition displayed the most unique profile of findings relative to other tasks. We conclude that when considering a wide range of domains, calibration displays domain generality, but resolution may display specificity across tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. For the general knowledge and emotion recognition tasks, due to a technical printing error, 38 participants missed completing two items on each of these tasks. We compared overall accuracy between those participants who missed the two items and the remaining 98 participants. There were no differences in overall accuracy. Thus, scores of these 38 participants were pro-rated for the statistical analyses.

  2. We also examined another resolution index, called the confidence-judgment accuracy quotient (CAQ; Jackson and Kleitman 2014; Schraw 2009) which provides a difference score between the average confidence assigned to correct and incorrect items. Across all of the analyses, we found parallel findings using the Gamma and CAQ indices.

References

  • Ackerman, R., & Thompson, V. A. (2017). Meta-reasoning: Monitoring and control of thinking and reasoning. Trends in Cognitive Sciences, 21(8), 607–617.

    Google Scholar 

  • Ackerman, R., & Zalmanov, H. (2012). The persistence of the fluency–confidence association in problem solving. Psychonomic Bulletin & Review, 19(6), 1187–1192.

    Google Scholar 

  • Ackerman, R., Parush, A., Nassar, F., & Shtub, A. (2016). Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior, 54, 101–113.

    Google Scholar 

  • Adolphs, R., Sears, L., & Piven, J. (2001). Abnormal processing of social information from faces in autism. Journal of Cognitive Neuroscience, 13(2), 232–240.

    Google Scholar 

  • Ais, J., Zylberberg, A., Barttfeld, P., & Sigman, M. (2016). Individual consistency in the accuracy and distribution of confidence judgments. Cognition, 146, 377–386.

    Google Scholar 

  • Anderson, M. C. M., & Thiede, K. W. (2008). Why do delayed summaries improve metacomprehension accuracy? Acta Psychologica, 128(1), 110–118.

    Google Scholar 

  • Baron-Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001). The “Reading the mind in the eyes” test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. Journal of Child Psychology and Psychiatry, 42(2), 241–251.

    Google Scholar 

  • Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64, 417–444.

    Google Scholar 

  • Bruine de Bruin, W., Parker, A. M., & Fischhoff, B. (2007). Individual differences in adult decision-making competence. Journal of Personality and Social Psychology, 92, 938–956.

    Google Scholar 

  • Chen, H., & Volpe, R. P. (1998). An analysis of personal financial literacy among college students. Financial Services Review, 7(2), 107–128.

    Google Scholar 

  • Destan, N., & Roebers, C. M. (2015). What are the metacognitive costs of young children’s overconfidence? Metacognition and Learning, 10(3), 347–374.

    Google Scholar 

  • Dunlosky, J., Mueller, M.L., & Thiede K.W. (2016) Methodology for investigating human metamemory: Problems and pitfalls: The Oxford handbook of metamemory.

  • Dunning, D., Heath, C., & Suls, J. M. (2004). Flawed self-assessment: Implications for health, education, and the workplace. Psychological Science in the Public Interest, 5(3), 69–106.

    Google Scholar 

  • Ekman, P. (1972). Universals and cultural differences in facial expressions of emotion. In J. Cole (Ed.), Nebraska symposium on motivation (pp. 207–283). Lincoln: University of Nebraska Press.

    Google Scholar 

  • Elfenbein, H. A., & Ambady, N. (2002). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128(2), 203–235.

    Google Scholar 

  • Engle, R. W. (2018). Working memory and executive attention: A revisit. Perspectives on Psychological Science, 13(2), 190–193.

    Google Scholar 

  • Erickson, S., & Heit, E. (2015). Metacognition and confidence: Comparing math to other academic subjects. Frontiers in Psychology, 6, 742.

    Google Scholar 

  • Finn, B. (2010). Ending on a high note: Adding a better end to effortful study. Journal of Experimental Psychology. Learning, Memory, and Cognition, 36(6), 1548–1553. https://doi.org/10.1037/a0020605.

    Article  Google Scholar 

  • Fleming, S. M., & Lau, H. C. (2014). How to measure metacognition. Frontiers in Human Neuroscience, 8, 443–451.

    Google Scholar 

  • Fukaya, T. (2013). Explanation generation, not explanation expectancy, improves metacomprehension accuracy. Metacognition and Learning, 8(1), 1–18.

    Google Scholar 

  • García, T., Rodríguez, C., González-Castro, P., González-Piendo, J., & Torrance, M. (2016). Elementary students’ metacognitive processes and post-performance calibration on mathematical problem-solving tasks. Metacognition and Learning, 11, 139–170. https://doi.org/10.1007/s11409-015-9139-1.

    Article  Google Scholar 

  • Gigerenzer, G., Gaissmaier, W., Kurz-Milcke, E., Schwartz, L. M., & Woloshin, S. (2007). Helping doctors and patients make sense of health statistics. Psychological Science in the Public Interest, 8, 53–96.

    Google Scholar 

  • Glaser, R. (1991). The maturing of the relationship between the science of learning and cognition and educational practice. Learning and Instruction, 1(2), 129–144.

    Google Scholar 

  • Glenberg, A. M., & Epstein, W. (1987). Inexpert calibration of comprehension. Memory & Cognition, 15(1), 84–93.

    Google Scholar 

  • Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (task load index): Results of empirical and theoretical research Advances in psychology (Vol. 52, pp. 139-183): Elsevier.

  • Hsu, C. F., Eastwood, J. D., & Toplak, M. E. (2017). Differences in perceived mental effort required and discomfort during a working memory task between individuals at-risk and not at-risk for ADHD. Frontiers in Psychology, 8, 407–415.

    Google Scholar 

  • Hsu, C. F., Propp, L., Panetta, L., Martin, S., Dentakos, S., Toplak, M. E., & Eastwood, J. D. (2018). Mental effort and discomfort: Testing the peak-end effect during a cognitively demanding task. PLoS One, 13(2), e0191479.

    Google Scholar 

  • Jackson, S. A., & Kleitman, S. (2014). Individual differences in decision-making and confidence: Capturing decision tendencies in a fictitious medical test. Metacognition and Learning, 9(1), 25–49.

    Google Scholar 

  • Juslin, P., Winman, A., & Olsson, H. (2000). Naive empiricism and dogmatism in confidence research: A critical examination of the hard–easy effect. Psychological Review, 107(2), 384–396.

    Google Scholar 

  • Kelemen, W. L., Frost, P. J., & Weaver, C. A. (2000). Individual differences in metacognition: Evidence against a general metacognitive ability. Memory & Cognition, 28(1), 92–107.

    Google Scholar 

  • Kelly, K. J., & Metcalfe, J. (2011). Metacognition of emotional face recognition. Emotion, 11(4), 896–906.

    Google Scholar 

  • Keren, G. (1991). Calibration and probability judgements: Conceptual and methodological issues. Acta Psychologica, 77(3), 217–273.

    Google Scholar 

  • Klayman, J., Soll, J. B., Gonzalez-Vallejo, C., & Barlas, S. (1999). Overconfidence: It depends on how, what, and whom you ask. Organizational Behavior and Human Decision Processes, 79(3), 216–247.

    Google Scholar 

  • Kleitman, S. (2008). Metacognition in the rationality debate. Self-confidence and its calibration. Saarbrucken: VDM Verlag Dr Muller.

    Google Scholar 

  • Kleitman, S., & Stankov, L. (2001). Ecological and person-oriented aspects of metacognitive processes in test-taking. Applied Cognitive Psychology, 15(3), 321–341.

    Google Scholar 

  • Kleitman, S., & Stankov, L. (2007). Self-confidence and metacognitive processes. Learning and Individual Differences, 17, 161–173.

    Google Scholar 

  • Kohler, C. G., Turner, T. H., Bilker, W. B., Brensinger, C. M., Siegel, S. J., Kanes, S. J., ... & Gur, R. C. (2003). Facial emotion recognition in schizophrenia: Intensity effects and error pattern. American Journal of Psychiatry, 160(10), 1768–1774.

  • Koriat, A. (1997). Monitoring one's own knowledge during study: A cue-utilization approach to judgments of learning. Journal of Experimental Psychology: General, 126(4), 349–370.

    Google Scholar 

  • Koriat, A. (2008). Easy comes, easy goes? The link between learning and remembering and its exploitation in metacognition. Memory & Cognition, 36(2), 416–428.

    Google Scholar 

  • Koriat, A. (2012a). The relationships between monitoring, regulation and performance. Learning and Instruction, 22(4), 296–298.

    Google Scholar 

  • Koriat, A. (2012b). The self-consistency model of subjective confidence. Psychological Review, 119(1), 80–113.

    Google Scholar 

  • Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Human Learning and Memory, 6(2), 107–118.

    Google Scholar 

  • Koriat, A., Sheffer, L., & Ma'ayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology: General, 131(2), 147–162.

    Google Scholar 

  • Koriat, A., Ackerman, R., Lockl, K., & Schneider, W. (2009). The memorizing effort heuristic in judgments of learning: A developmental perspective. Journal of Experimental Child Psychology, 102(3), 265–279.

    Google Scholar 

  • Lauterman, T., & Ackerman, R. (2014). Overcoming screen inferiority in learning and calibration. Computers in Human Behavior, 35, 455–463.

    Google Scholar 

  • Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? Organizational Behavior and Human Performance, 20(2), 159–183.

    Google Scholar 

  • Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1977). Calibration of probabilities: The state of the art. In Decision making and change in human affairs (pp. 275–324). Springer, Dordrecht.

  • Lin, L. M., & Zabrucky, K. M. (1998). Calibration of comprehension: Research and implications for education and instruction. Contemporary Educational Psychology, 23(4), 345–391.

    Google Scholar 

  • Lusardi, A., & Mitchell, O. S. (2014). The economic importance of financial literacy: Theory and evidence. Journal of Economic Literature, 52(1), 5–44.

    Google Scholar 

  • Maki, R. H., Shields, M., Wheeler, A. E., & Zacchilli, T. L. (2005). Individual differences in absolute and relative Metacomprehension accuracy. Journal of Educational Psychology, 97(4), 723–731.

    Google Scholar 

  • Malmendier, U., & Tate, G. (2008). Who makes acquisitions? CEO overconfidence and the market's reaction. Journal of Financial Economics, 89(1), 20–43.

    Google Scholar 

  • Mandell, L. (2008). Financial literacy of high school students. In Handbook of consumer finance research (pp. 163–183). Springer, New York, NY.

  • Masson, M. E., & Rotello, C. M. (2009). Sources of bias in the Goodman–Kruskal gamma coefficient measure of association: Implications for studies of metacognitive processes. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(2), 509–527.

    Google Scholar 

  • Miller, T. M., & Geraci, L. (2011). Unskilled but aware: Reinterpreting overconfidence in low-performing students. Journal of Experimental Psychology: Learning, Memory, and Cognition, 37(2), 502–506.

    Google Scholar 

  • Miron-Shatz, T., Hanoch, Y., Doniger, G. M., Omer, Z. B., & Ozanne, E. M. (2014). Subjective but not objective numeracy influences willingness to pay for BRCA1/2 genetic testing. Judgment and Decision making, 9(2), 152–158.

    Google Scholar 

  • Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95(1), 109–133.

    Google Scholar 

  • Nelson, T. O., & Narens, L. (1980). Norms of 300 general-information questions: Accuracy of recall, latency of recall, and feeling-of-knowing ratings. Journal of Verbal Learning and Verbal Behavior, 19(3), 338–368.

    Google Scholar 

  • Pallier, G., Wilkinson, R., Danthiir, V., Kleitman, S., Knezevic, G., Stankov, L., & Roberts, R. D. (2002). The role of individual differences in the accuracy of confidence judgments. The Journal of General Psychology, 129(3), 257–299.

    Google Scholar 

  • Parker, A. M., & Fischhoff, B. (2005). Decision-making competence: External validation through an individual differences approach. Journal of Behavioral Decision Making, 18, 1–27.

    Google Scholar 

  • Perfect, T. J. (2004). The role of self-rated ability in the accuracy of confidence judgements in eyewitness memory and general knowledge. Applied Cognitive Psychology, 18(2), 157–168.

    Google Scholar 

  • Rinne, L. F., & Mazzocco, M. M. (2014). Knowing right from wrong in mental arithmetic judgments: Calibration of confidence predicts the development of accuracy. PLoS One, 9(7), e98663.

    Google Scholar 

  • Ronis, D. L., & Yates, J. F. (1987). Components of probability judgment accuracy: Individual consistency and effects of subject matter and assessment method. Organizational Behavior and Human Decision Processes, 40(2), 193–218.

    Google Scholar 

  • Rozenblit, L., & Keil, F.C. (2002). The misunderstood limits of folk science: an illusion of explanatory depth. Cognitive Science 26(5), 521–562.

  • Schrand, C. M., & Zechman, S. L. (2012). Executive overconfidence and the slippery slope to financial misreporting. Journal of Accounting and Economics, 53(1–2), 311–329.

    Google Scholar 

  • Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning, 4(1), 33–45.

    Google Scholar 

  • Schraw, G., Dunkle, M. E., Bendixen, L. D., & Roedel, T. D. (1995). Does a general monitoring skill exist? Journal of Educational Psychology, 87(3), 433–444.

    Google Scholar 

  • Scott, B. M., & Berman, A. F. (2013). Examining the domain-specificity of metacognition using academic domains and task-specific individual differences. Australian Journal of Educational & Developmental Psychology, 13, 28–43.

    Google Scholar 

  • Shipley, W. C., Gruber, C. P., Martin, T. A., & Klein, A. M. (2009). Shipley-2. Los Angeles: Western Psychological Services.

    Google Scholar 

  • Shipstead, Z., Harrison, T. L., & Engle, R. W. (2016). Working memory capacity and fluid intelligence: Maintenance and disengagement. Perspectives on Psychological Science, 11(6), 771–799.

    Google Scholar 

  • Sidi, Y., Shpigelman, M., Zalmanov, H., & Ackerman, R. (2017). Understanding metacognitive inferiority on screen by exposing cues for depth of processing. Learning and Instruction, 51, 61–73.

    Google Scholar 

  • Soderstrom, N. C., Yue, C. L., & Bjork, E. L. (2015). Metamemory and education. In The Oxford Handbook of Metamemory.

  • Stankov, L., Kleitman, S., & Jackson, S. A. (2014). Measures of the trait of confidence. In G. J. Boyle, H. Saklofske, & G. Matthews (Eds.), Measures of personality and social psychological constructs. Academic Press (pp. 158-189).

  • Stanovich, K. E. (2009). What intelligence tests miss: The psychology of rational thought. New Haven: Yale University Press.

    Google Scholar 

  • Stanovich, K. (2011). Rationality and the reflective mind. New York: Oxford University Press.

    Google Scholar 

  • Stanovich, K. E., & West, R. F. (1998). Individual differences in rational thought. Journal of Experimental Psychology: General, 127(2), 161–188.

    Google Scholar 

  • Stanovich, K. E., West, R. F., & Toplak, M. E. (2011). Intelligence and rationality. In R. J. Sternberg & S. B. Kaufman (Eds.), Cambridge handbook of intelligence (pp. 784–826). New York: Cambridge University Press.

    Google Scholar 

  • Stanovich, K. E., West, R. F., & Toplak, M. E. (2016). The Rationality Quotient: Toward a test of rational thinking. MIT Press.

  • Tauber, S. K., Dunlosky, J., Rawson, K. A., Rhodes, M. G., & Sitzman, D. M. (2013). General knowledge norms: Updated and expanded from the Nelson and Narens (1980) norms. Behavior Research Methods, 45(4), 1115–1143.

    Google Scholar 

  • Thiede, K. W., Anderson, M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95(1), 66–73.

    Google Scholar 

  • Thiede, K. W., Dunlosky, J., Griffin, T. D., & Wiley, J. (2005). Understanding the delayed-keyword effect on metacomprehension accuracy. Journal of Experimental Psychology: Learning, Memory, and Cognition, 31(6), 1267–1280.

    Google Scholar 

  • Thiede, K. W., Griffin, T. D., Wiley, J., & Anderson, M. C. (2010). Poor metacomprehension accuracy as a result of inappropriate cue use. Discourse Processes, 47(4), 331–362.

    Google Scholar 

  • Thiede, K. W., Wiley, J., & Griffin, T. D. (2011). Test expectancy affects metacomprehension accuracy. British Journal of Educational Psychology, 81(2), 264–273.

    Google Scholar 

  • Thompson, V. A. (2009). Dual process theories: A metacognitive perspective. In J. ST. B. T. Evans & K. Frankish (Eds.), In two minds: Dual processes and beyond. Oxford University Press, Oxford.

  • Turner, M. L., & Engle, R. W. (1989). Is working memory capacity task dependent? Journal of Memory and Language, 28(2), 127–154.

    Google Scholar 

  • Veenman, M. V. J., & Verheij, J. (2003). Technical students' metacognitive skills: Relating general vs. specific metacognitive skills to study success. Learning and Individual Differences, 13(3), 259–272.

    Google Scholar 

  • Veenman, M. V., Van Hout-Wolters, B. H., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1(1), 3–14.

    Google Scholar 

  • West, R. F., & Stanovich, K. E. (1997). The domain specificity and generality of overconfidence: Individual differences in performance estimation bias. Psychonomic Bulletin & Review, 4(3), 387–392.

    Google Scholar 

  • Yates, J. F., Lee, J., & Bush, J. G. (1997). General knowledge overconfidence: Cross-national variations, response style, and "reality". Organizational Behavior and Human Decision Processes, 70, 87–94.

    Google Scholar 

  • Zacharakis, A. L., & Shepherd, D. A. (2001). The nature of information and overconfidence on venture capitalists' decision making. Journal of Business Venturing, 16(4), 311–332.

    Google Scholar 

Download references

Author information

Author notes

  1. Stella Dentakos and Wafa Saoud made equal contributions to the submission.

Authors and Affiliations

  1. LaMarsh Centre for Child and Youth Research, Department of Psychology, York University, 126 BSB, 4700 Keele St, Toronto, Ontario, M3J 1P3, Canada

    Stella Dentakos, Wafa Saoud & Maggie E. Toplak

  2. Faculty of Industrial Engineering & Management, Technion—Israel Institute of Technology, Haifa, Israel

    Rakefet Ackerman

Authors
  1. Stella Dentakos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wafa Saoud
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rakefet Ackerman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Maggie E. Toplak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maggie E. Toplak.

Ethics declarations

Human studies

The research reported in this study involving human participants was approved by the Research Ethics Board at York University.

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dentakos, S., Saoud, W., Ackerman, R. et al. Does domain matter? Monitoring accuracy across domains. Metacognition Learning 14, 413–436 (2019). https://doi.org/10.1007/s11409-019-09198-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11409-019-09198-4

Keywords

Search

Navigation