Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies?
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
Neuroscience is a rapidly expanding field in which complex studies and equipment setups are the norm. Often these push boundaries in terms of what technology can offer, and increasingly they make use of a wide range of stimulus materials and interconnected equipment (e.g., magnetic resonance imaging, electroencephalography, magnetoencephalography, eyetrackers, biofeedback, etc.). The software that bonds the various constituent parts together itself allows for ever more elaborate investigations to be carried out with apparent ease. However, research over the last decade has suggested a growing, yet underacknowledged, problem with obtaining millisecond-accurate timing in some computer-based studies. Crucially, timing inaccuracies can affect not just response time measurements, but also stimulus presentation and the synchronization between equipment. This is not a new problem, but rather one that researchers may have assumed had been solved with the advent of faster computers, state-of-the-art equipment, and more advanced software. In this article, we highlight the potential sources of error, their causes, and their likely impact on replication. Unfortunately, in many applications, inaccurate timing is not easily resolved by utilizing ever-faster computers, newer equipment, or post-hoc statistical manipulation. To ensure consistency across the field, we advocate that researchers self-validate the timing accuracy of their own equipment whilst running the actual paradigm in situ.
- Basser, PJ, Jones, DK (2002) Diffusion-tensor MRI: Theory, experimental design and data analysis—A technical review. NMR in Biomedicine 15: pp. 456-467 CrossRef
- Brown, W, Malveau, R, McCormick, S, Mowbray, T (1998) AntiPatterns: Refactoring software, architectures, and projects in crisis. Wiley, New York, NY
- Damian, MF (2010) Does variability in human performance outweigh imprecision in response devices such as computer keyboards?. Behavior Research Methods 42: pp. 205-211 CrossRef
- Jaekl, P, Soto-Faraco, S, Harris, LR (2012) Perceived size change induced by audiovisual temporal delays. Experimental Brain Research 216: pp. 457-462 CrossRef
- Kessler, B, Treiman, R, Mullennix, J (2002) Phonetic biases in voice key response time measurements. Journal of Memory and Language 47: pp. 145-171 CrossRef
- Killion, MC (1984) New insert headphones for audiometry. Hearing Instruments 35: pp. 46
- McDonnell, S (2004) Code complete: A practical handbook of software construction. Microsoft Press, Redmond, WA
- Morgante, JD, Zolfaghari, R, Johnson, SP (2011) A critical test of temporal and spatial accuracy of the Tobii T60XL eye tracker. Infancy 17: pp. 9-32 CrossRef
- Neath, I, Earle, A, Hallett, D, Surprenant, AM (2011) Response timing accuracy in Apple Macintosh computers. Behavior Research Methods 43: pp. 353-362 CrossRef
- Pashler, H, Wagenmakers, E-J (2012) Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?. Perspectives on Psychological Science 7: pp. 528-530 CrossRef
- Plant, R, Hammond, NV (2001) Benchmarking the timing characteristics of tools used by behavioural scientists. Abstracts of the Psychonomic Society (42nd Annual Meeting) 6: pp. 109a
- Plant, R., & Hammond, N. V. (2001b, November). Towards an experimental timing standards laboratory. Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Orlando, Florida.
- Plant, RR, Hammond, NV (2002) Towards an experimental timing standards lab: Benchmarking precision in the real world. Behavior Research Methods, Instruments, & Computers 34: pp. 218-226 CrossRef
- Plant, RR, Hammond, N, Whitehouse, T (2003) How choice of mouse may effect response timing in psychological studies. Behavior Research Methods, Instruments, & Computers 35: pp. 276-284 CrossRef
- Plant, RR, Turner, G (2004) Self-validating presentation and response timing in cognitive paradigms: How and why?. Behavior Research Methods, Instruments, & Computers 36: pp. 291-303 CrossRef
- Plant, RR, Turner, G (2009) Millisecond precision psychological research in a world of commodity computers: New hardware, new problems?. Behavior Research Methods 41: pp. 598-614 CrossRef
- Plant, R., & Turner, G. (2012, November). Could your equipment account for your experimental effect? Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Minneapolis, Minnesota.
- Reimers, S, Stewart, N (2007) Adobe Flash as a medium for online experimentation: A test of RT measurement capabilities. Behavior Research Methods 39: pp. 365-370 CrossRef
- Reimers, S, Stewart, N (2008) Using Adobe Flash Lite on mobile phones for psychological research: Reaction time measurement reliability and interdevice variability. Behavior Research Methods 40: pp. 1170-1176 CrossRef
- Rosenthal, R (1979) The file draw problem and the tolerance for null results. Psychological Bulletin 83: pp. 638-641 CrossRef
- Thurgood, C, Whitfield, TW, Patterson, J (2011) Towards a visual recognition threshold: New instrument shows humans identify animals with only 1 ms of visual exposure. Vision Research 51: pp. 1966-1971 CrossRef
- Tyler, MD, Tyler, L, Burnham, DK (2005) The delayed trigger voice key: An improved analogue voice key for psycholinguistic research. Behavior Research Methods 37: pp. 139-147 CrossRef
- Tyson, M. (2011, September 7). Experiments with precise timing in iOS [Web log post]. Retrieved from http://atastypixel.com/blog/experiments-with-precise-timing-in-ios/
- Ulrich, R, Giray, M (1989) Time resolution of clocks. Effects on reaction time measurement—Good news for bad clocks. British Journal of Mathematical and Statistical Psychology 42: pp. 1-12 CrossRef
- Wang, D., Vaidyanathan, P., Haake, A., & Pelz, J. (2012, November). Are eye trackers always as accurate as we assume? Paper presented at the annual meeting of the Society for Computers in Psychology (SCip), Minneapolis, Minnesota.
- Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies?
Cognitive, Affective, & Behavioral Neuroscience
Volume 13, Issue 3 , pp 598-614
- Cover Date
- Print ISSN
- Online ISSN
- Springer US
- Additional Links
- Millisecond timing accuracy
- Millisecond timing error
- Experiment generators
- Equipment error
- Industry Sectors