Abstract
In theoretical models of self-regulated learning, calibration is one important component in successful learning. Two issues of calibration are explored. First, Nelson (1987) suggested the G (gamma) coefficient is the most appropriate measure of calibration (judgment accuracy) and rejected signal detection theory’s d′ statistic because data commonly challenge distributional assumptions. We empirically examined this issue, comparing G and d′. Second, we examined whether a learner’s calibration varies across three domains of knowledge: general, word, and mathematics. A sample of 266 university students volunteered to participate in the study. Participants were selected from various undergraduate and graduate courses. Participants first answered demographic items. Then they completed three knowledge tests (general, word, and mathematics) and judged correctness for each answer provided. Order of domains was randomly counterbalanced among participants. Results show that d′ is a valid measure of calibration, that assumptions about underlying distributions can be tested, and that preliminary evidence suggests that d′ may be a superior measure of accuracy compared to G. Finally, calibration varied by domain.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ashby, F. G., & Townsend, T. (1986). Varieties of perceptual independence. Psychological Review, 95, 124–150.
Bornstein, B. H., & Zickafoose, D. J. (1999). “I know I know it, I know I saw it”: The stability of the confidence–accuracy relationship across domains. Journal of Experimental Psychology: Applied, 5, 76–88.
Burke, D. M., MacKay, D. G., Worthley, J. S., & Wade, E. (1991). On the tip of the tongue: what causes word finding failures in younger and older adults? Journal of Memory and Language, 30, 542–579.
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: a theoretical synthesis. Review of Educational Research, 65, 245–281.
Dunlosky, J., & Nelson, T. O. (1994). Does the sensitivity of judgments of learning (JOLs) to the effects of various study activities depend on when the JOLs occur? Journal of Memory and Language, 33, 545–565.
Glenberg, A. M., & Epstein, W. (1987). Inexpert calibration of comprehension. Memory & Cognition, 15, 84–93.
Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications. Journal of the American Statistical Association, 49, 732–764.
Gourevitch, V., & Galanter, E. (1967). A significance test for one parameter isosensitivity functions. Psychometrika, 32, 25–33.
Green, D. M., & Swets, J. A. (1966). Signal detection theory and psychophysics. New York: Wiley.
Kadlec, H. (1999). Statistical properties of d′ and β estimates of signal detection theory. Psychological Methods, 4, 22–43.
Kelemen, W. L., Frost, P. J., & Weaver, C. A. I. I. I. (2000). Individual differences in metacognition: evidence against a general metacognitive ability. Memory & Cognition, 28, 92–107.
Kim, J. O. (1971). Predictive measures of ordinal association. American Journal of Sociology, 76, 891–907.
Macmillan, A., & Creelman, D. (1996). Triangles in ROC space: history and theory of “nonparametric” measures of sensitivity and response bias. Psychonomic Bulletin & Review, 3, 164–170.
Maki, R. H., & Serra, M. (1992). The basis of test predictions for text materials. Journal of Experimental Psychology. Learning, Memory, and Cognition, 18, 116–126.
Masson, M. E. J., & Rotello, C. M. (2009). Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 509–527.
Matlin, M. W., & Foley, H. J. (1997). Sensation and perception (4th ed.). Needham Heights: Allyn and Bacon.
Nelson, T. O. (1984). A comparison of current measures of the accuracy of feeling-of-knowing predictions. Psychological Bulletin, 95, 109–133.
Nelson, T. O. (1986). ROC curves and measures of discrimination accuracy: a reply to Swets. Psychological Bulletin, 100, 128–132.
Nelson, T. O. (1987). The Goodman-Kruskal gamma coefficient as an alternative to signal-detection theory’s measures of absolute-judgment accuracy. In E. Roskam & R. Suck (Eds.), Progress in mathematical psychology, vol. 1 (pp. 299–306). Amsterdam: Elsevier/North-Holland.
Nelson, T. O. (1988). Predictive accuracy of the feeling of knowing across different criterion tasks and across different subject populations and individuals. In M. M. Gruenberg, P. E. Morris, & R. N. Sykes (Eds.), Practical aspects of memory: Current research and issues, vol. 1 (pp. 190–196). New York: Wiley.
Nelson, T. O., & Narens, L. (1980). Norms of 300 general-information questions: accuracy of recall, latency of recall, and feeling-of-knowing ratings. Journal of Verbal Learning and Verbal Behavior, 19, 338–368.
Pastore, R. E., & Scheirer, C. J. (1974). Signal detection theory: considerations for general application. Psychological Bulletin, 81, 945–958.
Pressley, M., & Ghatala, E. S. (1990). Self-regulated learning: monitoring learning from text. Educational Psychologist, 25, 19–33.
Robertson, C., & Morgan, T. (1990). Aspects of the design and analysis of signal detection experiments. Psychology, 42, 7–14.
Schraw, G. (2009). Measuring metacognitive judgments. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Handbook of metacognition in education (pp. 415–429). New York: Routledge.
Schraw, G., & Nietfeld, J. (1998). A further test of the general monitoring skill hypothesis. Journal of Educational Psychology, 90, 236–248.
Schraw, G., Dunkle, M. E., Bendixen, L. D., & Roedel, T. D. (1995). Does a general monitoring skill exist? Journal of Educational Psychology, 87, 433–444.
Schwartz, B. L., & Metcalfe, J. (1994). Methodological problems and pitfalls in the study of human metacognition. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 93–113). Cambridge: MIT Press.
Stone, N. J. (2000). Exploring the relationship between calibration and self-regulated learning. Educational Psychology Review, 12, 437–475.
Swets, J. A. (1973). The relative operating characteristic in psychology. Science, 182, 990–1000.
Swets, J. A. (1986). Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychological Bulletin, 99, 181–198.
Tobias, S., & Everson, H. (2000). Assessing metacognitive knowledge monitoring. In G. Schraw (Ed.), Issues in the measurement of metacognition (pp. 147–222). Lincoln: University of Nebraska Press.
Weaver, C. A., III, & Kelemen, W. L. (2002). Comparing processing-based, stimulus-based, and subject-based factors in metacognition. In P. Chambres, M. Izaute, & P. Marescaux (Eds.), Metacognition: Process, function, and use (pp. 49–60). Dordrecht: Kluwer Academic Publishers.
Wellman, H. M. (1977). Tip of the tongue and feeling of knowing experience: a developmental study of memory monitoring. Child Development, 48, 13–21.
Wilson, T. P. (1974). Measures of association for bivariate ordinal hypotheses. In H. M. Blalock (Ed.), Measurement in the social sciences (pp. 327–342). Chicago: Aldine.
Winne, P. H. (1995). Self regulation is ubiquitous but its forms vary with knowledge. Educational Psychologist, 30, 223–228.
Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. J. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp. 153–189). Hillsdale: Erlbaum.
Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45, 267–276.
Winne, P. H., & Jamieson-Noel, D. L. (2002). Exploring students’ calibration of self-reports about study tactics and achievement. Contemporary Educational Psychology, 27, 551–572.
Winne, P. H., & Perry, N. E. (2000). Measuring self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Orlando: Academic.
Winne, P. H., Jamieson-Noel, D., & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In P. R. Pintrich & M. L. Maehr (Eds.), Advances in motivation and achievement: New directions in measures and methods, vol. 12 (pp. 121–155). Greenwich: JAI Press.
Author information
Authors and Affiliations
Corresponding author
Additional information
Support for this research was provided by grants to Philip H. Winne from the Social Sciences and Humanities Research Council of Canada (410-2002-1787 and 512-2003-1012), the Canada Research Chair program, and Simon Fraser University.
Rights and permissions
About this article
Cite this article
Winne, P.H., Muis, K.R. Statistical estimates of learners’ judgments about knowledge in calibration of achievement. Metacognition Learning 6, 179–193 (2011). https://doi.org/10.1007/s11409-011-9074-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11409-011-9074-8