Abstract
Assuming that behavior observed during functional magnetic resonance imaging (fMRI) is comparable with behavior outside the scanner appears to be a basic tenet in cognitive neuroscience. Nevertheless, this assumption has rarely been tested directly. Here, we examined the reliability and validity of speeded performance during fMRI scanning by having the same 30 participants perform a battery of five reaction time (RT) tasks in two separate fMRI sessions and a standard laboratory (i.e., outside-scanner) session. Medium-to-high intra-class correlations between the three sessions showed that individual RT differences were conserved across sessions. Thus, for the range of tasks used, test–retest reliability and criterion validity of performance during scanning were satisfactory. Further, the pattern of between-task relations did not change within the scanner, attesting to the construct validity of performance measurements during scanning. In some tasks, however, RTs obtained from fMRI conditions were significantly shorter than those observed under normal laboratory conditions. In summary, RTs obtained during fMRI scanning appear to be largely reliable and valid measures of behavior. The observed RT speed-up during scanning might reflect task-specific interactions with a slightly different neuro-cognitive state, indicating some limits to generalizing brain–behavior relations observed with fMRI. These findings encourage further efforts in fMRI research to establish the external validity of within-scanner task performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aron AR, Gluck MA, Poldrack RA (2006) Long-term test-retest reliability of functional MRI in a classification learning task. NeuroImage 29(3):1000–1006. doi:10.1016/j.neuroimage.2005.08.010
Assecondi S, Vanderperren K, Novitskiy N, Ramautar JR, Fias W, Staelens S, Stiers P, Sunaert S, Van Huffel S, Lemahieu I (2010) Effect of the static magnetic field of the MR-scanner on ERPs: evaluation of visual, cognitive and motor potentials. Clin Neurophysiol 121(5):672–685. doi:10.1016/j.clinph.2009.12.032
Chong RK, Mills B, Dailey L, Lane E, Smith S, Lee KH (2010) Specific interference between a cognitive task and sensory organization for stance balance control in healthy young adults: visuospatial effects. Neuropsychologia 48(9):2709–2718. doi:10.1016/j.neuropsychologia.2010.05.018
Chouinard PA, Morrissey BF, Kohler S, Goodale MA (2008) Repetition suppression in occipital-temporal visual areas is modulated by physical rather than semantic features of objects. NeuroImage 41(1):130–144. doi:10.1016/j.neuroimage.2008.02.011
Clément F, Belleville S (2009) Test-retest reliability of fMRI verbal episodic memory paradigms in healthy older adults and in persons with mild cognitive impairment. Hum Brain Mapp 30(12):4033–4047. doi:10.1002/hbm.20827
Fernández G, Specht K, Weis S, Tendolkar I, Reuber M, Fell J, Klaver P, Ruhlmann J, Reul J, Elger CE (2003) Intrasubject reproducibility of presurgical language lateralization and mapping using fMRI. Neurology 60(6):969–975
Fischer T, Langner R, Birbaumer N, Brocke B (2008) Arousal and attention: self-chosen stimulation optimizes cortical excitability and minimizes compensatory effort. J Cogn Neurosci 20(8):1443–1453. doi:10.1162/jocn.2008.20101
Fliessbach K, Rohe T, Linder NS, Trautner P, Elger CE, Weber B (2010) Retest reliability of reward-related BOLD signals. NeuroImage 50(3):1168–1176. doi:10.1016/j.neuroimage.2010.01.036
Harley C, Boyd JE, Cockburn J, Collin C, Haggard P, Wann JP, Wade DT (2006) Disruption of sitting balance after stroke: influence of spoken output. J Neurol Neurosurg Psychiatry 77(5):674–676. doi:10.1136/jnnp.2005.074138
Hockey GRJ (1984) Varieties of attentional state: The effects of the environment. In: Parasuraman R, Davies DR (eds) Varieties of attention. Academic Press, Orlando, pp 449–483
Hommel B, Fischer R, Colzato LS, van den Wildenberg WP, Cellini C (2012) The effect of fMRI (noise) on cognitive control. J Exp Psychol Hum Percept Perform 38(2):290–301. doi:10.1037/a0026353
Jackson DN (1984) Multidimensional Aptitude Battery. Research Psychologists Press, Port Huron
Kiehl KA, Liddle PF (2003) Reproducibility of the hemodynamic response to auditory oddball stimuli: a six-week test-retest study. Hum Brain Mapp 18(1):42–52. doi:10.1002/hbm.10074
Koch I, Ruge H, Brass M, Rubin O, Meiran N, Prinz W (2003) Equivalence of cognitive processes in brain imaging and behavioral studies: evidence from task switching. NeuroImage 20(1):572–577
Koten JW Jr, Wood G, Hagoort P, Goebel R, Propping P, Willmes K, Boomsma DI (2009) Genetic contribution to variation in cognitive function: an fMRI study in twins. Science 323(5922):1737–1740. doi:10.1126/science.1167371
Krinzinger H, Koten JW, Hennemann J, Schueppen A, Sahr K, Arndt D, Konrad K, Willmes K (2011) Sensitivity, reproducibility, and reliability of self-paced versus fixed stimulus presentation in an fMRI study on exact, non-symbolic arithmetic in typically developing children aged between 6 and 12 years. Developmental neuropsychology 36(6):721–740. doi:10.1080/87565641.2010.549882
Langner R, Eickhoff SB (2012) Sustaining attention to simple tasks: a meta-analytic review of the neural mechanisms of vigilant attention. Psychol Bull. doi:10.1037/a0030694
Langner R, Willmes K, Chatterjee A, Eickhoff SB, Sturm W (2010) Energetic effects of stimulus intensity on prolonged simple reaction-time performance. Psychol Res 74(5):499–512. doi:10.1007/s00426-010-0275-6
Manoach DS, Halpern EF, Kramer TS, Chang Y, Goff DC, Rauch SL, Kennedy DN, Gollub RL (2001) Test-retest reliability of a functional MRI working memory paradigm in normal and schizophrenic subjects. Am J Psychiatry 158(6):955–958
McGraw KO, Wong SP (1996) Forming inferences about some intraclass correlation coefficients. Psychol Meth 1:30–46
Newell A, Rosenbloom PS (1981) Mechanisms of skill acquisition and the law of practice. In: Anderson JR (ed) Cognitive skills and their acquisition. Erlbaum, Hillsdale, pp 1–55
Snow JC, Pettypiece CE, McAdam TD, McLean AD, Stroman PW, Goodale MA, Culham JC (2011) Bringing the real world into the fMRI scanner: repetition effects for pictures versus real objects. Scientific reports 1:130. doi:10.1038/srep00130
Stark R, Schienle A, Walter B, Kirsch P, Blecker C, Ott U, Schäfer A, Sammer G, Zimmermann M, Vaitl D (2004) Hemodynamic effects of negative emotional pictures - a test-retest analysis. Neuropsychobiology 50(1):108–118. doi:10.1159/000077948
Vernon PA (1989) The heritability of measures of speed of information processing. Personality Individ Differ 10:573–576
Wagner K, Frings L, Quiske A, Unterrainer J, Schwarzwald R, Spreer J, Halsband U, Schulze-Bonhage A (2005) The reliability of fMRI activations in the medial temporal lobes in a verbal episodic memory task. NeuroImage 28(1):122–131. doi:10.1016/j.neuroimage.2005.06.005
Author information
Authors and Affiliations
Corresponding authors
Additional information
Jan Willem Koten and Robert Langner contributed equally.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Koten, J.W., Langner, R., Wood, G. et al. Are reaction times obtained during fMRI scanning reliable and valid measures of behavior?. Exp Brain Res 227, 93–100 (2013). https://doi.org/10.1007/s00221-013-3488-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00221-013-3488-2