Abstract
Different tests measure text comprehension, including the cloze gap-filling test, often used for language learning. Different studies hypothesized cognitive strategies in this type of test and their relationship with working memory and performance. However, no study investigated the cloze test, working memory, and possible cognitive strategies, while performing the test. Therefore, this study aimed to identify cognitive visual strategies in the cloze test by applying an unsupervised algorithm and to analyze the relationship between these strategies with working memory and performance in the cloze test. Our sample consisted of 51 university students, the largest sample in studies of cognitive strategies with cloze tests. Participants answered an 11-item cloze test in a computer with eye-tracking, a verbal working memory test, and a visuospatial working memory test. Our analysis of participants’ scanpath identified two main strategies: one with fewer toggles between text and word bank and fewer fixations than the other one, indicating the existence of a global strategy. Furthermore, a model predicting the efficiency of participants in the cloze test found that item complexity, using a global strategy, and higher scores of working memory were the most significant predictors. These results confirm the hypothesis of a global strategy being related to successfully achieving higher-order reading processes.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alderson JC (2000) Assessing reading. Cambridge University Press, Cambridge
Alderson JC, Cseresznyés M (2003) Into Europe: reading and use of English. Budapest: Teleki László Foundation
Anderson NJ, Bachman L, Perkins K, Cohen A (1991) An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Lang Test 8:41–66. https://doi.org/10.1177/026553229100800104
Baddeley A (2012) Working memory: theories, models, and controversies. Annu Rev Psychol 63:1–29
Bertolino NP (2018) Avaliação de compreensão de leitura: evidências a partir da análise dos movimentos oculares em teste do tipo Cloze [Reading comprehension assessment: evidence from the analysis of eye movements in a Cloze test] (Undergraduate research). Mackenzie Presbyterian University, São Paulo, Brazil. http://eventoscopq.mackenzie.br/index.php/jornada/xvjornada/paper/download/1650/1011
Brown JD (2003) Twenty-five years of cloze testing research: So what? In Poedjosoedarmo GR (Ed.), Teaching and assessing language proficiency, pp 79–111. Singapore: SEAMEO Regional Language Centre
Brunfaut T, McCray G (2015) Looking into test-takers’ cognitive processes whilst completing reading tasks: a mixed-method eye-tracking and stimulated recall study. (ARAGs Research Reports Online; Vol. AR/2015/001). London: The British Council. Retrieved from https://www.britishcouncil.org/sites/default/files/brunfaut_and_mccray_report_final_0.pdf
Bruyer R, Brysbaert M (2011) Combining speed and accuracy in cognitive psychology: Is the inverse efficiency score (IES) a better dependent variable than the mean reaction time (RT) and the percentage of errors (PE)? Psychol Belgica 51:5–13. https://doi.org/10.5334/pb-51-1-5
Chicco D, Warrens MJ, Jurman G (2021) The coefficient of determination R-squared is more informative than SMAPE, MAE, MAPE, MSE and RMSE in regression analysis evaluation. PeerJ Comput Sci 7:e623. https://doi.org/10.7717/peerj-cs.623
Corsi PM (1973) Human memory and the medial temporal region of the brain. Dissertation Abstracts International, 34(2-B), 891
Cule E, Moritz S, Frankowski D (2021) Ridge Regression with Automatic Selection of the Penalty Parameter. R Package version 2.9. https://CRAN.R-project.org/package=ridge
Davies A, Brown A, Elder C, Hill K, Lumley T, McNamara T (1999) Dictionary of language testing. Cambridge University Press, Cambridge
Donolato E, Giofrè D, Mammarella IC (2017) Differences in verbal and visuospatial forward and backward order recall: a review of the literature. Front Psychol 8:663. https://doi.org/10.3389/fpsyg.2017.00663
Dungan BJ, Vogel EK (2015) Short-term memory. In Toga AW (Ed.) Brain mapping, pp 481–485. Academic Press. https://doi.org/10.1016/B978-0-12-397025-1.00273-6
Ellwood-Lowe ME, Whitfield-Gabrieli S, Bunge SA (2021) Brain network coupling associated with cognitive performance varies as a function of a child’s environment in the ABCD study. Nat Commun 12(1):7183. https://doi.org/10.1038/s41467-021-27336-y
Engelhard G (2001) Historical views of the influences of measurement and reading theories on the assessment of reading. J Appl Meas 2:1–28
Figueiredo VL, Nascimento ED (2007) Performance in the two digit span subtest tasks of the WISC-III and WAIS-III. Psicologia teoria e pesquisa 23(3):313–318
Friedman J, Hastie T, Tibshirani R (2010) Regularization paths for generalized linear models via coordinate descent. J Stat Softw 33:1–22
Gao X, Gu X (2008) An introspective study on test-taking process for banked cloze. CELEA J 31(4):3–16
Gonthier C, Roulin J-L (2020) Intraindividual strategy shifts in Raven’s matrices, and their dependence on working memory capacity and need for cognition. J Exp Psychol Gen 149(3):564–579. https://doi.org/10.1037/xge0000660
Hale J (2001) A probabilistic Earley parser as a psycholinguistic model. In: Second meeting of the north american chapter of the association for computational linguistics
Hering E (1879) Über Muskelgeräusche des Auges [About eye muscle noises]. Sitzberichte der kaiserlichen Akademie der Wissenschaften in Wien. Mathematischnaturwissenschaftliche Klasse 79:137–154
Ilc G, Stopar A (2015) Validating the Slovenian national alignment to CEFR: the case of the B2 reading comprehension examination in English. Lang Test 32(4):443–462. https://doi.org/10.1177/0265532214562098
Javal E (1878) Essai sur la physiologie de la lecture [Essay on the Physiology of Reading]. Annales D’ocilistique 80:240–274
Javal (1879) E. Essai sur la physiologie de la lecture [Essay on the Physiology of Reading]. Annales D’ocilistique 82:242–253
Juhasz BJ, Pollatsek A (2011) Lexical influences on eye movements in reading. In: Liversedge SP, Gilchrist ID, Everling S (eds) The Oxford handbook of eye movements. Oxford University Press, pp 873–893
Khalifa H, Weir CJ (2009) Examining reading: Research and practice in assessing second language reading. Cambridge University Press, Cambridge
Kleijn S, Pander Maat H, Sanders T (2019) Cloze testing for comprehension assessment: the HyTeC-cloze. Lang Test 36(4):553–572. https://doi.org/10.1177/0265532219840382
Kucharský Š, Visser I, Truțescu GO, Laurence PG, Zaharieva M, Raijmakers ME (2020) Cognitive strategies revealed by clustering eye movement transitions. J Eye Mov Res. https://doi.org/10.16910/jemr.13.1.1
Lamare M (1892) Des mouvements des yeux dans la lecture [About the eye movements in reading]. Bulletins Et Mémoires De La Société Française D’ophthalmologie 10:354–364
Laurence PG, Macedo EC (2023) Cognitive strategies in matrix-reasoning tasks: state of the art. Psychon Bull Rev 30:147–159. https://doi.org/10.3758/s13423-022-02160-7
Laurence PG, Pinto TM, Rosa ATF, Macedo EC (2018) Can a lexical decision task predict efficiency in the judgment of ambiguous sentences? Psicologia Reflexão e Crítica 31:13. https://doi.org/10.1186/s41155-018-0093-0
Laurence PG, Jana TA, Bunge SA, Macedo EC (2023) Eye gaze patterns during reasoning provide insights regarding individual differences in underlying cognitive abilities. J Intell 11(4):75. https://doi.org/10.3390/jintelligence11040075
Levy R (2008) Expectation-based syntactic comprehension. Cognition 106(3):1126–1177. https://doi.org/10.1016/j.cognition.2007.05.006
Liesefeld HR, Janczyk M (2018) Combining speed and accuracy to control for speed-accuracy trade-offs(?). Behav Res Methods 51:40–60. https://doi.org/10.3758/s13428-018-1076-x
McCray G, Brunfaut T (2018) Investigating the construct measured by banked gap-fill items: evidence from eye-tracking. Lang Test 35:51–73. https://doi.org/10.1177/0265532216677105
Mertzen D, Paape D, Dillon B, Engbert R, Vasishth S, Mertzen D, Vasishth S (2023) Syntactic and semantic interference in sentence comprehension: Support from English and German eye-tracking data. Glossa Psycholing. https://doi.org/10.5070/G60111266
Mézière DC, Yu L, Reichle ED, Von Der Malsburg T, McArthur G (2023) Using eye-tracking measures to predict reading. Compreh Read Res Q 58(3):425–449. https://doi.org/10.1002/rrq.498
Michelino MS (2019) Processamento Fonológico e Velocidade de Nomeação Automática Rápida em Jovens e Adultos Analfabetos Funcionais [Phonological Processing and Rapid Automatic Naming Speed in Functional Illiterate Young People and Adults] (Master’s thesis, Mackenzie Presbyterian University). Adelpha Repositório Digital. São Paulo, Brazil. https://dspace.mackenzie.br/items/4f3a52e2-e91f-49bb-b0b7-47706aac44a4
Oller JW, Jonz J (1994) Why cloze Procedure? In Oller JW, Jonz J (Eds.), Cloze and coherence. Cranbury, NJ: Bucknell University Press
Pinheiro AMV (1996) Contagem de frequência de ocorrência de palavras expostas a crianças de 1ª à 4ª série do Ensino Fundamental [Frequency count of occurrence of words exposed to children from the 1st to the 4th grade of Elementary School]. São Paulo, SP: Associação Brasileira de Dislexia
Płużyczka M (2018) The first hundred years: a history of eye tracking as a research method. Appl Ling Pap 45(4):101–116. https://doi.org/10.32612/uw.25449354.2018
R Core Team (2021) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria
Revelle W (2020) Psych: procedures for personality and psychological research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psych Version = 2.0.12
Rosa ATF (2016) Análise do padrão dos movimentos oculares em tarefas de leitura: relações entre processamento lexical, sintático e semântico [Analysis of eye movement pattern in reading tasks: relationships between lexical, syntactic and semantic processing] (Doctoral dissertation). Mackenzie Presbyterian University, São Paulo, Brazil. https://dspace.mackenzie.br/items/aee67c1a-5fb7-4f88-8001-e87febcff91b
Santos FH, Mello CB, Bueno OFA, Dellatolas G (2005) Cross-cultural differences for three visual memory tasks in Brazilian children. Percept Motor Skills 101(2):421–433. https://doi.org/10.2466/pms.101.2.421-433
Snow RE (1980) Aptitude processes. In: Snow RE, Federick P-A, Montague WE (eds) Aptitude learning, and instruction: cognitive process analysis of aptitude. Erlbaum, Hillsdale, NJ, pp 27–63
Steiger JH (1980) Tests for comparing elements of a correlation matrix. Psychol Bull 87(2):245–251. https://doi.org/10.1037/0033-2909.87.2.245
Trace J (2020) Clozing the gap: How far do cloze items measure? Lang Test 37(2):235–253. https://doi.org/10.1177/0265532219888617
Tremblay A (2011) Proficiency assessment standards in second language acquisition research: “Clozing” the gap. Stud Second Lang Acquis 33(3):339–372. https://doi.org/10.1017/S0272263111000015
Vabalas A, Gowen E, Poliakoff E, Casson AJ (2019) Machine learning algorithm validation with a limited sample size. PLoS ONE 14(11):e0224365. https://doi.org/10.1371/journal.pone.0224365
Vinegrad M (1994) A revised adult dyslexia checklist. Educare 48:21–23
Von der Malsburg T, Vasishth S (2011) What is the scanpath signature of syntactic reanalysis? J Mem Lang 65(2):109–127. https://doi.org/10.1016/j.jml.2011.02.004
Wechsler D (1997) WAIS-III: administration and scoring manual. Psychological Corporation, San Antonio. https://doi.org/10.1037/t49755-000
Wechsler D (2003) Wechsler intelligence scale for children, fourth edition (WISC-IV). Psychological Corporation, San Antonio. https://doi.org/10.1037/t15174-000
Weir C, Hawkey R, Green A, Devi S (2012) The cognitive processes underlying the academic reading construct as measured by IELTS. IELTS Res Rep 9:157–189
Yamashita J (2003) Processes of taking a gap-filling test: comparison of skilled and less skilled EFL readers. Lang Test 20(3):267–293. https://doi.org/10.1191/0265532203lt257oa
Yang C (2023) Using eye-tracking and retrospective verbal reports to explore the cognitive processes of banked gap-filling: a case study featuring methodological triangulation. Lang Test Asia 13(1):22. https://doi.org/10.1186/s40468-023-00234-4
Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B (stat Methodol) 67(2):301–320
Acknowledgements
We want to thank Šimon Kucharský for publishing the algorithm’s code openly at https://osf.io/wvzs9/ and for answering our questions regarding the implementation of the algorithm. The text was partially crafted using GPT-3.5, OpenAI's extensive language-generation model. Following the generation of the draft, the author meticulously reviewed, edited, and refined the text according to personal preferences, assuming full responsibility for the content of this publication.
Funding
This study was supported by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP; Grant Numbers: 2018/08069-3 and 2018/09654-7) and Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq; grant number: 309159/2019-9).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors have no competing interests to declare relevant to this article’s content.
Ethical approval
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institution and with the 1964 Helsinki Declaration and its later amendments. The study was approved by the Ethics and Research Committee of the University (CAAE 98599618.2.0000.0084).
Open practices
The study was preregistered in AsPredicted and can be accessed at https://aspredicted.org/blind.php?x=az5nn3. The data and analysis code are available at https://osf.io/sgxkh/.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Editors: Valerio Santangelo (University of Perugia), Moreno I. Coco (Sapienza University of Rome); Reviewers: Elena Allegretti (Sapienza University of Rome), Dario Paape (University of Potsdam).
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Laurence, P.G., Bassetto, S.A., Bertolino, N.P. et al. Differences in scanpath pattern and verbal working memory predicts efficient reading in the Cloze gap-filling test. Cogn Process (2024). https://doi.org/10.1007/s10339-024-01189-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10339-024-01189-x