Skip to main content
Log in

Evaluating the metacognitive awareness inventory using empirical factor-structure evidence

  • Published:
Metacognition and Learning Aims and scope Submit manuscript

Abstract

Many scholars agree on the general theoretical structure of metacognition, which is what informed the development of the Metacognitive Awareness Inventory (MAI). Although self-report instruments such as the MAI suffer many threats to validity, they continue to be used in research and practice because of their convenience. With the MAI, studies have varied in the way they calculate scores and in their adherence to the intended theory. In this study, we address these shortcomings and propose modifications in calculating MAI scores. Using confirmatory factor analysis (CFA) and multidimensional random coefficients multinomial logit (MRCML) item-response modeling, we examined how well the intended functioning of the MAI matched the data from 622 undergraduate students. The results support scoring the MAI as two dimensions, knowledge and regulation of cognition, but indicate that the 52-item instrument has poor fit. Using iterative CFA and MRCML models, we tested subsets of items that represent the theory and had good fit. We followed up with tests of between-group and time invariance. The results support the use of a 19-item subset for between-group comparisons, with provisional evidence for its use in longitudinal studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Although treating the two separate time points as different groups violated the assumption of local independence, the DIF part of this procedure was only for diagnosing plausible problem items rather than for making inferential decisions.

References

  • Adams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23. doi:10.1177/0146621697211001.

    Article  Google Scholar 

  • Adams, R. J., Wu, M. L., Haldane, S. A., & Xun, S. X. (2012). ConQuest (version 3.0.1) [computer software]. Camberwell: Australian Council for Educational Research.

    Google Scholar 

  • AERA, APA, & NCME. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

  • Akin, A., Abaci, R., & Çetin, B. (2007). The validity and reliability of the Turkish version of the metacognitive awareness inventory. Educational Sciences: Theory & Practice, 7, 671–678.

    Google Scholar 

  • Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching (Vol. 32). San Francisco: John Wiley & Sons.

  • Berger, J.-L., & Karabenick, S. A. (2016). Construct validity of self-reported metacognitive learning strategies. Educational Assessment, 21, 19–33. doi:10.1080/10627197.2015.1127751.

    Article  Google Scholar 

  • Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54, 199–231. doi:10.1111/j.1464-0597.2005.00205.x.

    Article  Google Scholar 

  • Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience and school (expanded ed.). Washington, D.C.: National Academy Press.

    Google Scholar 

  • Briggs, D. C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4, 87–100.

    Google Scholar 

  • Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. doi:10.1037/a0030641.

    Article  Google Scholar 

  • Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. Weinert & R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65–116). Mahwah: Erlbaum.

    Google Scholar 

  • Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.

    Google Scholar 

  • Couper, M. P., Tourangeau, R., Conrad, F. G., & Singer, E. (2006). Evaluating the effectiveness of visual analog scales a web experiment. Social Science Computer Review, 24, 227–245. doi:10.1177/0894439305281503.

    Article  Google Scholar 

  • Coutinho, S. A. (2007). The relationship between goals, metacognition, and academic success. Educate~: The Journal of Doctoral Research in Education, 7(1), 39–47.

    Google Scholar 

  • Cromley, J. G., & Azevedo, R. (2006). Self-report of reading comprehension strategies: What are we measuring? Metacognition and Learning, 1, 229–247. doi:10.1007/s11409-006-9002-5.

    Article  Google Scholar 

  • Dent, A. L., & Koenka, A. C. (2015). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, Advance online publication, 150. doi:10.1007/s10648-015-9320-8.

  • Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391–409. doi:10.1007/s10648-008-9083-6.

    Article  Google Scholar 

  • Duncan, T. G., & McKeachie, W. J. (2005). The making of the motivated strategies for learning questionnaire. Educational Psychologist, 40, 117–128. doi:10.1207/s15326985ep4002_6.

    Article  Google Scholar 

  • Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence (pp. 231–235). Hillsdale: Lawrence Erlbaum.

  • Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34, 906–911. doi:10.1037/0003-066X.34.10.906.

    Article  Google Scholar 

  • Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45, 203–209. doi:10.1080/00461520.2010.515935.

    Article  Google Scholar 

  • Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Disposition, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449–455. doi:10.1037/0003-066X.53.4.449.

    Article  Google Scholar 

  • Hartley, K., & Bendixen, L. D. (2003). The use of comprehension aids in a hypermedia environment: Investigating the impact of metacognitive awareness and epistemological beliefs. Journal of Educational Multimedia and Hypermedia, 12, 275–289.

    Google Scholar 

  • Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16, 235–266. doi:10.1023/B:EDPR.0000034022.16470.f3.

    Article  Google Scholar 

  • Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118.

    Article  Google Scholar 

  • Hughs, J. A. (2015). Impact of online self-regulated professional development on technology and engineering educators metacognitive awareness. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3710627).

  • Jacobs, J. E., & Paris, S. G. (1987). Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22, 255–278.

    Article  Google Scholar 

  • Jacobse, A. E., & Harskamp, E. G. (2012). Towards efficient measurement of metacognition in mathematical problem solving. Metacognition and Learning, 7, 133–149. doi:10.1007/s11409-012-9088-x.

    Article  Google Scholar 

  • Khosa, D. K., & Volet, S. E. (2014). Productive group engagement in cognitive activity and metacognitive regulation during collaborative learning: Can it explain differences in students’ conceptual understanding? Metacognition and Learning, 9, 287–307. doi:10.1007/s11409-014-9117-z.

    Article  Google Scholar 

  • Kleitman, S., & Stankov, L. (2007). Self-confidence and metacognitive processes. Learning and Individual Differences, 17, 161–173. doi:10.1016/j.lindif.2007.03.004.

    Article  Google Scholar 

  • Kline, R. B. (2011). Principles and practice of structural equation modeling (third ed.). New York: Guilford Press.

    Google Scholar 

  • Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (Second ed., pp. 263–313). Bingley: Emerald.

    Google Scholar 

  • Ku, K. Y. L., & Ho, I. T. (2010). Metacognitive strategies that enhance critical thinking. Metacognition and Learning, 5, 251–267. doi:10.1007/s11409-010-9060-6.

    Article  Google Scholar 

  • Kuhn, D., & Dean Jr., D. (2004). Metacognition: A bridge between cognitive psychology and educational practice. Theory Into Practice, 43, 268–273. doi:10.1207/s15430421tip4304_4.

    Article  Google Scholar 

  • Lima Filho, R. N., & Bruni, A. L. (2015). Metacognitive awareness inventory: Translation and validation from a confirmatory analysis. Psicologia: Ciência e Profissão, 35, 1275–1293. doi:10.1590/1982-3703002292013.

    Google Scholar 

  • Magno, C. (2008). Reading strategy, amount of writing, metacognition, metamemory, and apprehension as predictors of English written proficiency. Asian EFL Journal, 29, 15–48.

    Google Scholar 

  • Magno, C. (2010). The role of metacognitive skills in developing critical thinking. Metacognition and Learning, 5, 137–156. doi:10.1007/s11409-010-9054-4.

    Article  Google Scholar 

  • Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.

    Article  Google Scholar 

  • Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-category measures. Multivariate Behavioral Research, 39, 479–515.

    Article  Google Scholar 

  • Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77, 177–195. doi:10.1348/000709905X90876.

    Article  Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (2012). MPlus user’s guide (Seventh ed.). Los Angeles: Muthén & Muthén.

    Google Scholar 

  • Olejnik, S., & Nist, S. L. (1992). Identifying latent variables measured by the learning and study strategies inventory (LASSI). The Journal of Experimental Education, 60, 151–159.

    Article  Google Scholar 

  • Peverly, S. T., Brobst, K. E., Graham, M., & Shaw, R. (2003). College adults are not good at self-regulation: A study on the relationship of self-regulation, note taking, and test taking. Journal of Educational Psychology, 95, 335–346. doi:10.1037/0022-0663.95.2.335.

    Article  Google Scholar 

  • Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, 385–407. doi:10.1007/s10648-004-0006-x.

    Article  Google Scholar 

  • Pintrich, P. R., & de Groot, E. V. (1990). Motivated strategies for learning questionnaire. Retrieved from PsycTESTS. doi:10.1037/t09161-000.

  • Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). Assessing metacognition and self-regulated learning. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 43–97). Lincoln: Buros Institute.

    Google Scholar 

  • Pucheu, P. M. (2008). An investigation of the relationships between the scoring rubrics inventory and the metacognitive awareness inventory as reported by secondary school core -subject teachers. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3313868).

  • RincónGallardo, T. J. (2009). The effect of the use of learning journals on the development of metacognition in undergraduate students. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3389888).

  • Rozencwajg, P. (2003). Metacognitive factors in scientific problem-solving strategies. European Journal of Psychology of Education, 18, 281–294. doi:10.1007/BF03173249.

    Article  Google Scholar 

  • Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph No. 17.

  • Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students’ strategic processing. British Journal of Educational Psychology, 77, 351–378. doi:10.1348/000709906X106147.

    Article  Google Scholar 

  • Schellings, G., & Van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6, 83–90. doi:10.1007/s11409-011-9081-9.

    Article  Google Scholar 

  • Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.

    Google Scholar 

  • Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113–125. doi:10.1023/a:1003044231033.

    Article  Google Scholar 

  • Schraw, G. (2010). Measuring self-regulation in computer-based learning environments. Educational Psychologist, 45, 258–266. doi:10.1080/00461520.2010.515936.

    Article  Google Scholar 

  • Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460–475. doi:10.1006/ceps.1994.1033.

    Article  Google Scholar 

  • Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111–139. doi:10.1007/s11165-005-3917-8.

  • Schunk, D. H. (2008). Metacognition, self-regulation, and self-regulated learning: Research recommendations. Educational Psychology Review, 20, 463–467. doi:10.1007/s10648-008-9086-3.

    Article  Google Scholar 

  • Smith, E. V. (2004). Evidence for the reliability of measures and validity of measure interpretation: A Rasch measurement perspective. In E. V. Smith & R. M. Smith (Eds.), Introduction to Rasch measurement: Theory, models and applications (pp. 93–122). Maple Grove: JAM Press.

    Google Scholar 

  • Sperling, R. A., Howard, B. C., Staley, R., & DuBois, N. (2004). Metacognition and self-regulated learning constructs. Educational Research & Evaluation, 10, 117–139.

    Article  Google Scholar 

  • Sternberg, R. J. (1998). Metacognition, abilities, and developing expertise: What makes an expert student? Instructional Science, 26, 127–140. doi:10.1023/a:1003096215103.

    Article  Google Scholar 

  • Stewart, P. W., Cooper, S. S., & Moulding, L. R. (2007). Metacognitive development in professional educators. The Researcher, 21(1), 32–40.

    Google Scholar 

  • Teo, T., & Lee, C. B. (2012). Assessing the factorial validity of the metacognitive awareness inventory (MAI) in an Asian country: A confirmatory factor analysis. The International Journal of Educational and Psychological Assessment, 10(2), 92–103.

    Google Scholar 

  • Tobias, S., & Everson, H. (2000). Assessing metacognitive knowledge monitoring. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 147–222). Lincoln: Buros Institute.

    Google Scholar 

  • Tock, J. L., & Moxley, J. H. (2017). A comprehensive reanalysis of the metacognitive self-regulation scale from the MSLQ. Metacognition and Learning, 12, 79–111. doi:10.1007/s11409-016-9161-y.

    Article  Google Scholar 

  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.

    Book  Google Scholar 

  • Turan, S., Demirel, Ö., & Sayek, İ. (2009). Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Medical Teacher, 31(10), 477–483. doi:10.3109/01421590903193521.

    Article  Google Scholar 

  • Umino, A., & Dammeyer, J. (2016). Effects of a non-instructional prosocial intervention program on children’s metacognition skills and quality of life. International Journal of Educational Research, 78, 24–31. doi:10.1016/j.ijer.2016.05.004.

    Article  Google Scholar 

  • Vallin, L. M. (2017). Metacognition as a pedagogical approach to improve students’ use of metacognitive strategies. Honolulu: (Unpublished doctoral dissertation), University of Hawai‘i at Mānoa.

  • Vauras, M., Iiskala, T., Kajamies, A., Kinnunen, R., & Lehtinen, E. (2003). Shared-regulation and motivation of collaborating peers: A case analysis. Psychologia, 46, 19–37.

    Article  Google Scholar 

  • Veenman, M. V. J. (2011). Learning to self-monitor and self-regulate. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 197–218). New York: Taylor & Francis Group.

    Google Scholar 

  • Veenman, M. V. J., Kok, R., & Blöte, A. W. (2005). The relation between intellectual and metacognitive skills in early adolescence. Instructional Science, 33, 193–211. doi:10.1007/s11251-004-2274-8.

    Article  Google Scholar 

  • Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14. doi:10.1007/s11409-006-6893-0.

    Article  Google Scholar 

  • Vrugt, A., & Oort, F. J. (2008). Metacognition, achievement goals, study strategies and academic achievement: Pathways to achievement. Metacognition and Learning, 3, 123–146. doi:10.1007/s11409-008-9022-4.

    Article  Google Scholar 

  • Weinstein, C.E., Schulte, A., & Palmer, D.R. (1987). The learning and study strategies inventory Clearwater: H&H Publishing.

  • Wilson, M. (2005). Constructing measures: An item response modeling approach. New York: Psychology Press.

    Google Scholar 

  • Winne, P. H., & Perry, N. E. (2005). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Burlingtton: Elsevier Academic Press.

    Google Scholar 

  • Winston, K. A., Van der Vleuten, C. P. M., & Scherpbier, A. J. J. A. (2010). An investigation into the design and effectiveness of a mandatory cognitive skills programme for at-risk medical students. Medical Teacher, 32, 236–243. doi:10.3109/01421590903197035.

    Article  Google Scholar 

  • Young, A., & Fry, J. D. (2008). Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2), 1–10.

    Google Scholar 

  • Yuan, K.-H., Chan, W., Marcoulides, G. A., & Bentler, P. M. (2016). Assessing structural equation models by equivalence testing with adjusted fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 23, 319–330. doi:10.1080/10705511.2015.1065414.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to George M. Harrison.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendix

Appendix

Table 9 The MAI prompts and the factor assignments for the Study 1 model comparisons
Table 10 Studies’ scoring and response-scale formatting of the MAI

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Harrison, G.M., Vallin, L.M. Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition Learning 13, 15–38 (2018). https://doi.org/10.1007/s11409-017-9176-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11409-017-9176-z

Keywords

Navigation