Abstract
Many scholars agree on the general theoretical structure of metacognition, which is what informed the development of the Metacognitive Awareness Inventory (MAI). Although self-report instruments such as the MAI suffer many threats to validity, they continue to be used in research and practice because of their convenience. With the MAI, studies have varied in the way they calculate scores and in their adherence to the intended theory. In this study, we address these shortcomings and propose modifications in calculating MAI scores. Using confirmatory factor analysis (CFA) and multidimensional random coefficients multinomial logit (MRCML) item-response modeling, we examined how well the intended functioning of the MAI matched the data from 622 undergraduate students. The results support scoring the MAI as two dimensions, knowledge and regulation of cognition, but indicate that the 52-item instrument has poor fit. Using iterative CFA and MRCML models, we tested subsets of items that represent the theory and had good fit. We followed up with tests of between-group and time invariance. The results support the use of a 19-item subset for between-group comparisons, with provisional evidence for its use in longitudinal studies.
Similar content being viewed by others
Notes
Although treating the two separate time points as different groups violated the assumption of local independence, the DIF part of this procedure was only for diagnosing plausible problem items rather than for making inferential decisions.
References
Adams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1–23. doi:10.1177/0146621697211001.
Adams, R. J., Wu, M. L., Haldane, S. A., & Xun, S. X. (2012). ConQuest (version 3.0.1) [computer software]. Camberwell: Australian Council for Educational Research.
AERA, APA, & NCME. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
Akin, A., Abaci, R., & Çetin, B. (2007). The validity and reliability of the Turkish version of the metacognitive awareness inventory. Educational Sciences: Theory & Practice, 7, 671–678.
Ambrose, S. A., Bridges, M. W., DiPietro, M., Lovett, M. C., & Norman, M. K. (2010). How learning works: Seven research-based principles for smart teaching (Vol. 32). San Francisco: John Wiley & Sons.
Berger, J.-L., & Karabenick, S. A. (2016). Construct validity of self-reported metacognitive learning strategies. Educational Assessment, 21, 19–33. doi:10.1080/10627197.2015.1127751.
Boekaerts, M., & Corno, L. (2005). Self-regulation in the classroom: A perspective on assessment and intervention. Applied Psychology, 54, 199–231. doi:10.1111/j.1464-0597.2005.00205.x.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience and school (expanded ed.). Washington, D.C.: National Academy Press.
Briggs, D. C., & Wilson, M. (2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4, 87–100.
Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18, 36–52. doi:10.1037/a0030641.
Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. Weinert & R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65–116). Mahwah: Erlbaum.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: Guilford Press.
Couper, M. P., Tourangeau, R., Conrad, F. G., & Singer, E. (2006). Evaluating the effectiveness of visual analog scales a web experiment. Social Science Computer Review, 24, 227–245. doi:10.1177/0894439305281503.
Coutinho, S. A. (2007). The relationship between goals, metacognition, and academic success. Educate~: The Journal of Doctoral Research in Education, 7(1), 39–47.
Cromley, J. G., & Azevedo, R. (2006). Self-report of reading comprehension strategies: What are we measuring? Metacognition and Learning, 1, 229–247. doi:10.1007/s11409-006-9002-5.
Dent, A. L., & Koenka, A. C. (2015). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, Advance online publication, 1–50. doi:10.1007/s10648-015-9320-8.
Dinsmore, D. L., Alexander, P. A., & Loughlin, S. M. (2008). Focusing the conceptual lens on metacognition, self-regulation, and self-regulated learning. Educational Psychology Review, 20, 391–409. doi:10.1007/s10648-008-9083-6.
Duncan, T. G., & McKeachie, W. J. (2005). The making of the motivated strategies for learning questionnaire. Educational Psychologist, 40, 117–128. doi:10.1207/s15326985ep4002_6.
Flavell, J. H. (1976). Metacognitive aspects of problem solving. In L. B. Resnick (Ed.), The nature of intelligence (pp. 231–235). Hillsdale: Lawrence Erlbaum.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American Psychologist, 34, 906–911. doi:10.1037/0003-066X.34.10.906.
Greene, J. A., & Azevedo, R. (2010). The measurement of learners’ self-regulated cognitive and metacognitive processes while using computer-based learning environments. Educational Psychologist, 45, 203–209. doi:10.1080/00461520.2010.515935.
Halpern, D. F. (1998). Teaching critical thinking for transfer across domains: Disposition, skills, structure training, and metacognitive monitoring. American Psychologist, 53, 449–455. doi:10.1037/0003-066X.53.4.449.
Hartley, K., & Bendixen, L. D. (2003). The use of comprehension aids in a hypermedia environment: Investigating the impact of metacognitive awareness and epistemological beliefs. Journal of Educational Multimedia and Hypermedia, 12, 275–289.
Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16, 235–266. doi:10.1023/B:EDPR.0000034022.16470.f3.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6, 1–55. doi:10.1080/10705519909540118.
Hughs, J. A. (2015). Impact of online self-regulated professional development on technology and engineering educators metacognitive awareness. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3710627).
Jacobs, J. E., & Paris, S. G. (1987). Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22, 255–278.
Jacobse, A. E., & Harskamp, E. G. (2012). Towards efficient measurement of metacognition in mathematical problem solving. Metacognition and Learning, 7, 133–149. doi:10.1007/s11409-012-9088-x.
Khosa, D. K., & Volet, S. E. (2014). Productive group engagement in cognitive activity and metacognitive regulation during collaborative learning: Can it explain differences in students’ conceptual understanding? Metacognition and Learning, 9, 287–307. doi:10.1007/s11409-014-9117-z.
Kleitman, S., & Stankov, L. (2007). Self-confidence and metacognitive processes. Learning and Individual Differences, 17, 161–173. doi:10.1016/j.lindif.2007.03.004.
Kline, R. B. (2011). Principles and practice of structural equation modeling (third ed.). New York: Guilford Press.
Krosnick, J. A., & Presser, S. (2010). Question and questionnaire design. In P. V. Marsden & J. D. Wright (Eds.), Handbook of survey research (Second ed., pp. 263–313). Bingley: Emerald.
Ku, K. Y. L., & Ho, I. T. (2010). Metacognitive strategies that enhance critical thinking. Metacognition and Learning, 5, 251–267. doi:10.1007/s11409-010-9060-6.
Kuhn, D., & Dean Jr., D. (2004). Metacognition: A bridge between cognitive psychology and educational practice. Theory Into Practice, 43, 268–273. doi:10.1207/s15430421tip4304_4.
Lima Filho, R. N., & Bruni, A. L. (2015). Metacognitive awareness inventory: Translation and validation from a confirmatory analysis. Psicologia: Ciência e Profissão, 35, 1275–1293. doi:10.1590/1982-3703002292013.
Magno, C. (2008). Reading strategy, amount of writing, metacognition, metamemory, and apprehension as predictors of English written proficiency. Asian EFL Journal, 29, 15–48.
Magno, C. (2010). The role of metacognitive skills in developing critical thinking. Metacognition and Learning, 5, 137–156. doi:10.1007/s11409-010-9054-4.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
Millsap, R. E., & Yun-Tein, J. (2004). Assessing factorial invariance in ordered-category measures. Multivariate Behavioral Research, 39, 479–515.
Muis, K. R., Winne, P. H., & Jamieson-Noel, D. (2007). Using a multitrait-multimethod analysis to examine conceptual similarities of three self-regulated learning inventories. British Journal of Educational Psychology, 77, 177–195. doi:10.1348/000709905X90876.
Muthén, L. K., & Muthén, B. O. (2012). MPlus user’s guide (Seventh ed.). Los Angeles: Muthén & Muthén.
Olejnik, S., & Nist, S. L. (1992). Identifying latent variables measured by the learning and study strategies inventory (LASSI). The Journal of Experimental Education, 60, 151–159.
Peverly, S. T., Brobst, K. E., Graham, M., & Shaw, R. (2003). College adults are not good at self-regulation: A study on the relationship of self-regulation, note taking, and test taking. Journal of Educational Psychology, 95, 335–346. doi:10.1037/0022-0663.95.2.335.
Pintrich, P. R. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16, 385–407. doi:10.1007/s10648-004-0006-x.
Pintrich, P. R., & de Groot, E. V. (1990). Motivated strategies for learning questionnaire. Retrieved from PsycTESTS. doi:10.1037/t09161-000.
Pintrich, P. R., Wolters, C. A., & Baxter, G. P. (2000). Assessing metacognition and self-regulated learning. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 43–97). Lincoln: Buros Institute.
Pucheu, P. M. (2008). An investigation of the relationships between the scoring rubrics inventory and the metacognitive awareness inventory as reported by secondary school core -subject teachers. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3313868).
RincónGallardo, T. J. (2009). The effect of the use of learning journals on the development of metacognition in undergraduate students. (doctoral dissertation). ProQuest Dissertations & Theses Global database. (accession no. 3389888).
Rozencwajg, P. (2003). Metacognitive factors in scientific problem-solving strategies. European Journal of Psychology of Education, 18, 281–294. doi:10.1007/BF03173249.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph No. 17.
Samuelstuen, M. S., & Bråten, I. (2007). Examining the validity of self-reports on scales measuring students’ strategic processing. British Journal of Educational Psychology, 77, 351–378. doi:10.1348/000709906X106147.
Schellings, G., & Van Hout-Wolters, B. (2011). Measuring strategy use with self-report instruments: Theoretical and empirical considerations. Metacognition and Learning, 6, 83–90. doi:10.1007/s11409-011-9081-9.
Schön, D. A. (1983). The reflective practitioner: How professionals think in action. New York: Basic Books.
Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113–125. doi:10.1023/a:1003044231033.
Schraw, G. (2010). Measuring self-regulation in computer-based learning environments. Educational Psychologist, 45, 258–266. doi:10.1080/00461520.2010.515936.
Schraw, G., & Dennison, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460–475. doi:10.1006/ceps.1994.1033.
Schraw, G., Crippen, K. J., & Hartley, K. (2006). Promoting self-regulation in science education: Metacognition as part of a broader perspective on learning. Research in Science Education, 36, 111–139. doi:10.1007/s11165-005-3917-8.
Schunk, D. H. (2008). Metacognition, self-regulation, and self-regulated learning: Research recommendations. Educational Psychology Review, 20, 463–467. doi:10.1007/s10648-008-9086-3.
Smith, E. V. (2004). Evidence for the reliability of measures and validity of measure interpretation: A Rasch measurement perspective. In E. V. Smith & R. M. Smith (Eds.), Introduction to Rasch measurement: Theory, models and applications (pp. 93–122). Maple Grove: JAM Press.
Sperling, R. A., Howard, B. C., Staley, R., & DuBois, N. (2004). Metacognition and self-regulated learning constructs. Educational Research & Evaluation, 10, 117–139.
Sternberg, R. J. (1998). Metacognition, abilities, and developing expertise: What makes an expert student? Instructional Science, 26, 127–140. doi:10.1023/a:1003096215103.
Stewart, P. W., Cooper, S. S., & Moulding, L. R. (2007). Metacognitive development in professional educators. The Researcher, 21(1), 32–40.
Teo, T., & Lee, C. B. (2012). Assessing the factorial validity of the metacognitive awareness inventory (MAI) in an Asian country: A confirmatory factor analysis. The International Journal of Educational and Psychological Assessment, 10(2), 92–103.
Tobias, S., & Everson, H. (2000). Assessing metacognitive knowledge monitoring. In G. Schraw & J. C. Impara (Eds.), Issues in the measurement of metacognition (pp. 147–222). Lincoln: Buros Institute.
Tock, J. L., & Moxley, J. H. (2017). A comprehensive reanalysis of the metacognitive self-regulation scale from the MSLQ. Metacognition and Learning, 12, 79–111. doi:10.1007/s11409-016-9161-y.
Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.
Turan, S., Demirel, Ö., & Sayek, İ. (2009). Metacognitive awareness and self-regulated learning skills of medical students in different medical curricula. Medical Teacher, 31(10), 477–483. doi:10.3109/01421590903193521.
Umino, A., & Dammeyer, J. (2016). Effects of a non-instructional prosocial intervention program on children’s metacognition skills and quality of life. International Journal of Educational Research, 78, 24–31. doi:10.1016/j.ijer.2016.05.004.
Vallin, L. M. (2017). Metacognition as a pedagogical approach to improve students’ use of metacognitive strategies. Honolulu: (Unpublished doctoral dissertation), University of Hawai‘i at Mānoa.
Vauras, M., Iiskala, T., Kajamies, A., Kinnunen, R., & Lehtinen, E. (2003). Shared-regulation and motivation of collaborating peers: A case analysis. Psychologia, 46, 19–37.
Veenman, M. V. J. (2011). Learning to self-monitor and self-regulate. In R. E. Mayer & P. A. Alexander (Eds.), Handbook of research on learning and instruction (pp. 197–218). New York: Taylor & Francis Group.
Veenman, M. V. J., Kok, R., & Blöte, A. W. (2005). The relation between intellectual and metacognitive skills in early adolescence. Instructional Science, 33, 193–211. doi:10.1007/s11251-004-2274-8.
Veenman, M. V. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14. doi:10.1007/s11409-006-6893-0.
Vrugt, A., & Oort, F. J. (2008). Metacognition, achievement goals, study strategies and academic achievement: Pathways to achievement. Metacognition and Learning, 3, 123–146. doi:10.1007/s11409-008-9022-4.
Weinstein, C.E., Schulte, A., & Palmer, D.R. (1987). The learning and study strategies inventory Clearwater: H&H Publishing.
Wilson, M. (2005). Constructing measures: An item response modeling approach. New York: Psychology Press.
Winne, P. H., & Perry, N. E. (2005). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531–566). Burlingtton: Elsevier Academic Press.
Winston, K. A., Van der Vleuten, C. P. M., & Scherpbier, A. J. J. A. (2010). An investigation into the design and effectiveness of a mandatory cognitive skills programme for at-risk medical students. Medical Teacher, 32, 236–243. doi:10.3109/01421590903197035.
Young, A., & Fry, J. D. (2008). Metacognitive awareness and academic achievement in college students. Journal of the Scholarship of Teaching and Learning, 8(2), 1–10.
Yuan, K.-H., Chan, W., Marcoulides, G. A., & Bentler, P. M. (2016). Assessing structural equation models by equivalence testing with adjusted fit indexes. Structural Equation Modeling: A Multidisciplinary Journal, 23, 319–330. doi:10.1080/10705511.2015.1065414.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Harrison, G.M., Vallin, L.M. Evaluating the metacognitive awareness inventory using empirical factor-structure evidence. Metacognition Learning 13, 15–38 (2018). https://doi.org/10.1007/s11409-017-9176-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11409-017-9176-z