Skip to main content

Development and Validation of the Test Takers’ Metacognitive Awareness Reading Questionnaire (TMARQ)

Abstract

This article reports the development of the Test Takers’ Metacognitive Awareness Reading Questionnaire (TMARQ) which measures test takers’ metacognition in reading comprehension tests. The TMARQ comprises seven subscales: planning strategies, evaluating strategies, monitoring strategies, strategies for identifying important information, inference-making strategies, integrating strategies, and supporting strategies. In this article, a validity argument is laid out for the questionnaire by presenting content-referenced, substantive, and structural evidence of validity, which is primarily yielded through Rasch measurement and structural equation modeling.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

Notes

  1. The full score of English subject on the NTME is 150.

References

  • Anderson, N. (1991). Individual differences in strategy use in second language reading and testing. Modern Language Journal, 75, 460–472.

    Article  Google Scholar 

  • Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 255–292). White Plains, NY: Longman.

    Google Scholar 

  • Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.

    Article  Google Scholar 

  • Arbuckle, J. L. (2011). IBM SPSS Amos 20.0 [computer program]. New York: IBM.

    Google Scholar 

  • Aryadoust, V. (2013). Building a validity argument for a listening test of academic proficiency. Newcastle: Cambridge Scholars.

  • Aryadoust, V., Goh, C., & Lee, O. K. (2011). An investigation of differential item functioning in the MELAB listening test. Language Assessment Quarterly, 8(4), 361–385.

    Article  Google Scholar 

  • Aryadoust, V., Goh, C., & Lee, O. K. (2012). Developing an academic listening self-assessment questionnaire: A study of modeling academic listening. Psychological Test and Assessment Modeling, 54(3), 227–256.

    Google Scholar 

  • Bachman, L. F. (1990). Fundamental consideration in language testing. Oxford: Oxford University Press.

    Google Scholar 

  • Bachman, L. F., & Cohen, A. D. (1998). Interface between second language acquisition and language testing research. Cambridge: Cambridge University Press.

    Google Scholar 

  • Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice. Oxford: Oxford University Press.

    Google Scholar 

  • Bachman, L. F., & Palmer, A. S. (2010). Language testing in practice. Oxford: Oxford University Press.

    Google Scholar 

  • Baker, L., & Brown, A. L. (1984). Metacognitive skills and reading. In R. Barr, M. L. Kamil, P. Mosenthal, & P. D. Pearson (Eds.), Handbook of reading research (Vol. 2, pp. 353–394). White Plains, NY: Longman.

    Google Scholar 

  • Bandalos, D. L., & Finney, S. J. (2001). Item parceling issues in structural equation modeling. Mahwah, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Brown, A. L. (1978). Knowing when, where, and how to remember: A problem of metacognition. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 1, pp. 77–165). Hillsdale, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Byrne, B. M. (2011). Structural equation modeling with Mplus: Basic concepts, applications, and programming. New York: Routledge.

    Google Scholar 

  • Cain, K., Oakhill, J., & Bryant, P. (2004). Children’s reading comprehension ability: Concurrent prediction by working memory, verbal ability, and component skills. Journal of Educational Psychology, 96, 31–42.

    Article  Google Scholar 

  • Carrell, P. L. (1989). Metacognitive awareness and second language reading. Modern Language Journal, 73(2), 121–131.

    Article  Google Scholar 

  • Chapelle, C. A., Chung, Y.-R., Hegelheimer, V., Pendar, N., & Xu, J. (2010). Towards a computer-delivered test of productive grammatical ability. Language Testing, 27(4), 443–469.

    Article  Google Scholar 

  • Cohen, A. D. (2006). The coming age of research on test-taking strategies. Language Assessment Quarterly, 3(4), 307–331.

    Article  Google Scholar 

  • Cohen, A.D., & Upton, T.A. (2006). Strategies in responding to the new TOEFL reading tasks (TOEFL Monograph Series Report No. 33). Princeton, NJ: Educational Testing Service. Retrieved July 20, 2012 from http://www.ets.org/Media/Research/pdf/RR-06-06.pdf.

  • Dörnyei, Z. (2003). Questionnaires in second language research: Construction, administration and processing. Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data (Rev ed.). Cambridge, MA: MIT Press.

    Google Scholar 

  • Flavell, J. H. (1978). Metacognitive development. In J. M. Scadura & C. J. Brainerd (Eds.), Structural Process theories of complex human behavior. Alphen aan den Rijn: Sijthoff and Noordhoff.

    Google Scholar 

  • Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive- developmental inquiry. American Psychologist, 34, 906–911.

    Article  Google Scholar 

  • Gagnè, E. D., Yekovich, C. W., & Yekovich, F. R. (1993). The cognitive psychology of school learning. New York: Harper Collins.

    Google Scholar 

  • Hair, J. F., Black, W. C., Babin, B. J., Anderson, R. E., & Tatham, R. L. (2010). Multivariate data analysis (6th ed.). Upper Saddle River, NJ: Pearson Education.

    Google Scholar 

  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternative. Structural Equation Modeling, 6(1), 1–55.

    Article  Google Scholar 

  • Jacobs, J. E., & Paris, S. G. (1987). Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist, 22, 255–278.

    Article  Google Scholar 

  • Kintsch, W., & van Dijk, (1978). Toward a model of text comprehension and production. Psychological Review, 85, 363–394.

    Article  Google Scholar 

  • Kline, R. B. (2011). Principles and practices of structural equation modeling (2nd ed.). New York: Guilford.

    Google Scholar 

  • Kunnan, A. J. (1998). An introduction to structural equation modeling for language assessment research. Language Testing, 15(3), 295–332.

    Google Scholar 

  • Lee, C. B. (2013). Examining intentional knowing among secondary school students: Through the lens of metacognition. The Asia-Pacific Education Researcher, 22(1), 79–90.

    Google Scholar 

  • Linacre, J. M. (2010a). A users’ guide to WINSTEPS ® MINISTEPS Rasch-model computer programs. Chicago, IL: Winsteps.

    Google Scholar 

  • Linacre, J. M. (2010b). WINSTEPS: Rasch model computer programs. Chicago, IL: Winsteps.

    Google Scholar 

  • Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–103). New York: Macmillan.

    Google Scholar 

  • Mokhtari, K., Sheorey, R., & Reichard, C. A. (2009). Measuring the reading strategies of first and second language readers. In K. Mokhtari & R. Sheorey (Eds.), Reading strategies of first- and second language learners. Norwood, MA: Christopher-Gordon Publishers, Inc.

    Google Scholar 

  • Paris, S. G., Lipson, M. Y., & Wixson, K. K. (1983). Becoming a strategic reader. Contemporary Educational Psychology, 8, 248–293.

    Article  Google Scholar 

  • Paris, S. G., & Winograd, P. (1990). How metacognition can promote academic learning and instruction. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction (pp. 15–51). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Park, G. (1997). Language learning strategies and English proficiency in Korean university students. Foreign Language Annals, 30, 211–221.

    Article  Google Scholar 

  • Pearson, P. D. (2009). The roots of reading comprehension instruction. In S. E. Isreal & G. G. Duffy (Eds.), Handbook of research on reading comprehension. New York: Routledge.

    Google Scholar 

  • Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing, 20(1), 26–56.

    Article  Google Scholar 

  • Phakiti, A. (2008). Construct validation of Bachman and Palmer’s (1996) strategic competence model over time in EFL reading tests. Language Testing, 25(2), 237–272.

    Article  Google Scholar 

  • Pintrich, P. R., Smith, D. A. F., Garcia, T., & McKeachie, W. J. (1993). Reliability and predictive validity of the motivated strategies for learning questionnaire (MSLQ). Educational and Psychological Measurement, 53, 801–803.

    Article  Google Scholar 

  • Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Purpura, J. E. (1997). An analysis of the relationships between test takers’ cognitive and metacognitive strategy use and second language test performance. Language Learning, 47, 289–294.

    Google Scholar 

  • Purpura, J. E. (1999). Learner strategy use and performance on language tests: A structural equation modeling approach. Cambridge: Cambridge University Press.

    Google Scholar 

  • Roeschl-Heils, A., Schneider, W., & van Kraayenoord, C. E. (2003). Reading, metacognition, and motivation: A follow-up study of German students 7 and 8. European Journal of Psychology of Education, 18, 75–86.

    Article  Google Scholar 

  • Smith, E. V. Jr. (2005). Effect of redundancy on Rasch item and person estimates. Journal of Applied Measurement, 6(2), 147–163.

    Google Scholar 

  • Song, X. (2005). Language learner strategy use and English proficiency on the Michigan English Language Assessment Battery. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 3, 1–26.

    Google Scholar 

  • Song, X., & Cheng, L. (2006). Language learner strategy use and test performance of Chinese learners of English. Language Assessment Quarterly, 3(3), 243–266.

    Article  Google Scholar 

  • Tarricone, P. (2011). The taxonomy of metacognition. New York: The Psychology Press.

  • van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York: Academic.

  • Wang, W. C. (2004). Direct estimation of correlation as a measure of association strength using multidimensional item response models. Educational and Psychological Measurement, 64, 937–955.

    Article  Google Scholar 

  • Wen, Q., & Johnson, R. (1997). L2 learner variables and English achievement: A study of tertiary-level English majors in China. Applied Linguistics, 18, 27–48.

    Article  Google Scholar 

  • Wenden, A. L. (1998). Metacognitive knowledge and language learning. Applied Linguistics, 19(4), 515–537.

    Article  Google Scholar 

  • Wolfe, E. W., & Smith, E. V. Jr. (2007a). Instrument development tools and activities for measure validation using Rasch models: Part I—Instrument development tools. Journal of Applied Measurement, 8, 97–123.

    Google Scholar 

  • Wolfe, E. W., & Smith, E. V. Jr. (2007b). Instrument development tools and activities for measure validation using Rasch models: Part II—Validation activities. Journal of Applied Measurement, 8, 204–233.

    Google Scholar 

  • Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370.

    Google Scholar 

  • Wu, M. L., Adams, R., Wilson, M., & Haldane, S. A. (2007). ACER ConQuest, version 2.0 [computer program]. Melbourne: Australian Council for Educational Research.

    Google Scholar 

  • Zhang, L. J. (2010). A dynamic metacognitive systems account of Chinese university students’ knowledge about EFL reading. TESOL Quarterly, 44(2), 320–353.

    Google Scholar 

  • Zhang, L. (in press). A structural equation modeling approach to investigating test takers’ strategy use and reading test performance. Asian EFL Journal.

  • Zhang, L., & Zhang, L. J. (2013). Relationships between Chinese college test takers’ strategy use and EFL reading test performance: A structural equation modelling approach. RECL Journal, 44(1), 35–57.

    Google Scholar 

  • Zheng, Y., & Cheng, L. (2008). Test review: College English test (CET) in China. Language Testing, 25(3), 408–417.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Limei Zhang.

Appendices

Appendix 1

Test Takers’ Metacognitive Awareness Reading Questionnaire (TMARQ)

The purpose of this survey is to collect information about various strategies you use when taking reading comprehension tests. Each statement is followed by six numbers, 0, 1, 2, 3, 4, and 5, and each number means the following:

  • 0 means that “I never do this.”

  • 1 means that “I almost never do this.”

  • 2 means that “I do this only occasionally.”

  • 3 means that “I sometimes do this.” (about 50 % of the time)

  • 4 means that “I usually do this.”

  • 5 means that “I always or almost always do this.”

After reading each statement, circle the number (0, 1, 2, 3, 4, or 5) which applies to you. Note that there are no right or wrong responses to any of the items on this survey.

Strategy Never Always
1. I take a general view of the text to see what it is about before reading. 0 1 2 3 4 5
2. I review the text first by noting its characteristics like length and structure. 0 1 2 3 4 5
3. I have a purpose as to what to read and in what order. 0 1 2 3 4 5
4. I plan what to do before the reading test. 0 1 2 3 4 5
5. I make sure I am clear about the goals of the reading test tasks. 0 1 2 3 4 5
6. I think over essential steps to complete the reading test. 0 1 2 3 4 5
7. I know what to do if my plan does not work out while completing this reading test. 0 1 2 3 4 5
8. I flip through the reading test before I actually start it. 0 1 2 3 4 5
9. I use test questions to establish purposes. 0 1 2 3 4 5
10. I critically evaluate the information presented in the text. 0 1 2 3 4 5
11. I know when I understand something and when I do not. 0 1 2 3 4 5
12. I am aware of my loss of concentration in reading the text. 0 1 2 3 4 5
13. I make affective responses to the text content. 0 1 2 3 4 5
14. I check if I understand the text and reading tasks regularly in completing the test. 0 1 2 3 4 5
15. I evaluate my plan of test completion constantly. 0 1 2 3 4 5
16. I know when I should complete the test more carefully. 0 1 2 3 4 5
17. When the text becomes difficult, I reread the problematic part to increase my understanding. 0 1 2 3 4 5
18. I consider whether the content of the text fits my reading purpose. 0 1 2 3 4 5
19. I adjust my reading speed to increase comprehension. 0 1 2 3 4 5
20. I am aware when and where I am confused in the text. 0 1 2 3 4 5
21. I try to get back on track when I lose concentration. 0 1 2 3 4 5
22. I know when I should complete the test more quickly. 0 1 2 3 4 5
23. I budget my time wisely on the test. 0 1 2 3 4 5
24. I adjust my pace in answering questions. 0 1 2 3 4 5
25. I adjust overall strategies in completing the test. 0 1 2 3 4 5
26. I correct my misunderstanding or mistakes immediately. 0 1 2 3 4 5
27. I check my own performance and progress as I complete the test. 0 1 2 3 4 5
28. I pay attention to the main points when reading the text. 0 1 2 3 4 5
29. I read the first sentence of each paragraph for the main idea. 0 1 2 3 4 5
30. I look for how the text is organized and ignore details. 0 1 2 3 4 5
31. I try to get the gist of each paragraph. 0 1 2 3 4 5
32. I preview key sentences to understand the text better. 0 1 2 3 4 5
33. I skip unknown words when reading. 0 1 2 3 4 5
34. I skim reading materials for a general understanding. 0 1 2 3 4 5
35. I scan reading materials for specific words or phrases. 0 1 2 3 4 5
36. I use typographical features like boldface and italics to identify key information. 0 1 2 3 4 5
37. I use context clues to help me understand the text better. 0 1 2 3 4 5
38. I try to make inferences of the meanings of new words from context clues. 0 1 2 3 4 5
39. I guess meanings of unknown words using root words. 0 1 2 3 4 5
40. I infer what will happen next when reading the text. 0 1 2 3 4 5
41. I make inferences beyond the information presented in the text. 0 1 2 3 4 5
42. I try to use my prior knowledge to help my understanding. 0 1 2 3 4 5
43. I use my past experience to increase my understanding. 0 1 2 3 4 5
44. I make summaries of new information to understand the text better. 0 1 2 3 4 5
45. I try to get the overall meanings of the text when reading. 0 1 2 3 4 5
46. I go back and forth in the text to find relationships among ideas in it. 0 1 2 3 4 5
47. I read the text not only for a surface understanding but also for its implied meaning. 0 1 2 3 4 5
48. I reread texts or tasks several times when I feel I do not understand them. 0 1 2 3 4 5
49. If I do not understand one part, I go back to a point before that part and reread from there. 0 1 2 3 4 5
50. I take notes to increase my understanding. 0 1 2 3 4 5
51. I underline or circle information in the text to understand the text better. 0 1 2 3 4 5
52. I paraphrase (restate in my own words) to understand the text better. 0 1 2 3 4 5
53. I try to visualize information to help my understanding. 0 1 2 3 4 5
54. I translate key words from English into Chinese as I read and answer the questions. 0 1 2 3 4 5

Appendix 2

See Table 9.

Table 9 Summary of the rating category measures of the seven subscales

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Zhang, L., Aryadoust, V. & Zhang, L.J. Development and Validation of the Test Takers’ Metacognitive Awareness Reading Questionnaire (TMARQ). Asia-Pacific Edu Res 23, 37–51 (2014). https://doi.org/10.1007/s40299-013-0083-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40299-013-0083-z

Keywords

  • Metacognitive awareness
  • Reading test
  • Rasch measurement
  • Structural equation modeling
  • Validity