Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies?
First Online: 03 May 2013 DOI:
10.3758/s13415-013-0166-6 Cite this article as: Plant, R.R. & Quinlan, P.T. Cogn Affect Behav Neurosci (2013) 13: 598. doi:10.3758/s13415-013-0166-6 Abstract
Neuroscience is a rapidly expanding field in which complex studies and equipment setups are the norm. Often these push boundaries in terms of what technology can offer, and increasingly they make use of a wide range of stimulus materials and interconnected equipment (e.g., magnetic resonance imaging, electroencephalography, magnetoencephalography, eyetrackers, biofeedback, etc.). The software that bonds the various constituent parts together itself allows for ever more elaborate investigations to be carried out with apparent ease. However, research over the last decade has suggested a growing, yet underacknowledged, problem with obtaining millisecond-accurate timing in some computer-based studies. Crucially, timing inaccuracies can affect not just response time measurements, but also stimulus presentation and the synchronization between equipment. This is not a new problem, but rather one that researchers may have assumed had been solved with the advent of faster computers, state-of-the-art equipment, and more advanced software. In this article, we highlight the potential sources of error, their causes, and their likely impact on replication. Unfortunately, in many applications, inaccurate timing is not easily resolved by utilizing ever-faster computers, newer equipment, or post-hoc statistical manipulation. To ensure consistency across the field, we advocate that researchers self-validate the timing accuracy of their own equipment whilst running the actual paradigm in situ.
Keywords Replication Millisecond timing accuracy Millisecond timing error Experiment generators Equipment error References
Basser, P. J., & Jones, D. K. (2002). Diffusion-tensor MRI: Theory, experimental design and data analysis—A technical review.
NMR in Biomedicine, 15,
10.1002/nbm.783 PubMed CrossRef Google Scholar
Brown, W., Malveau, R., McCormick, S., & Mowbray, T. (1998).
AntiPatterns: Refactoring software, architectures, and projects in crisis
. New York, NY: Wiley.
Damian, M. F. (2010). Does variability in human performance outweigh imprecision in response devices such as computer keyboards?
Behavior Research Methods, 42,
10.3758/BRM.42.1.205 PubMed CrossRef Google Scholar
Jaekl, P., Soto-Faraco, S., & Harris, L. R. (2012). Perceived size change induced by audiovisual temporal delays.
Experimental Brain Research, 216,
10.1007/s00221-011-2948-9 CrossRef Google Scholar
Kessler, B., Treiman, R., & Mullennix, J. (2002). Phonetic biases in voice key response time measurements.
Journal of Memory and Language, 47,
10.1006/jmla.2001.2835 CrossRef Google Scholar
Killion, M. C. (1984). New insert headphones for audiometry.
Hearing Instruments, 35,
McDonnell, S. (2004).
Code complete: A practical handbook of software construction
. Redmond, WA: Microsoft Press.
Morgante, J. D., Zolfaghari, R., & Johnson, S. P. (2011). A critical test of temporal and spatial accuracy of the Tobii T60XL eye tracker.
10.1111/j.1532-7078.2011.00089.x CrossRef Google Scholar
Neath, I., Earle, A., Hallett, D., & Surprenant, A. M. (2011). Response timing accuracy in Apple Macintosh computers.
Behavior Research Methods, 43,
10.3758/s13428-011-0069-9 PubMed CrossRef Google Scholar
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?
Perspectives on Psychological Science, 7,
10.1177/1745691612465253 CrossRef Google Scholar
Plant, R., & Hammond, N. V. (2001a). Benchmarking the timing characteristics of tools used by behavioural scientists.
Abstracts of the Psychonomic Society (42nd Annual Meeting), 6,
Plant, R., & Hammond, N. V. (2001b, November).
Towards an experimental timing standards laboratory. Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Orlando, Florida.
Plant, R. R., & Hammond, N. V. (2002). Towards an experimental timing standards lab: Benchmarking precision in the real world.
Behavior Research Methods, Instruments, & Computers, 34,
CrossRef Google Scholar
Plant, R. R., Hammond, N., & Whitehouse, T. (2003). How choice of mouse may effect response timing in psychological studies.
Behavior Research Methods, Instruments, & Computers, 35,
10.3758/BF03202553 CrossRef Google Scholar
Plant, R. R., & Turner, G. (2004). Self-validating presentation and response timing in cognitive paradigms: How and why?
Behavior Research Methods, Instruments, & Computers, 36,
CrossRef Google Scholar
Plant, R. R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: New hardware, new problems?
Behavior Research Methods, 41,
10.3758/BRM.41.3.598 PubMed CrossRef Google Scholar
Plant, R., & Turner, G. (2012, November).
Could your equipment account for your experimental effect? Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Minneapolis, Minnesota.
Reimers, S., & Stewart, N. (2007). Adobe Flash as a medium for online experimentation: A test of RT measurement capabilities.
Behavior Research Methods, 39,
10.3758/BF03193004 PubMed CrossRef Google Scholar
Reimers, S., & Stewart, N. (2008). Using Adobe Flash Lite on mobile phones for psychological research: Reaction time measurement reliability and interdevice variability.
Behavior Research Methods, 40,
10.3758/BRM.40.4.1170 PubMed CrossRef Google Scholar
Rosenthal, R. (1979). The file draw problem and the tolerance for null results.
Psychological Bulletin, 83,
10.1037/0033-2909.86.3.638 CrossRef Google Scholar
Thurgood, C., Whitfield, T. W., & Patterson, J. (2011). Towards a visual recognition threshold: New instrument shows humans identify animals with only 1 ms of visual exposure.
Vision Research, 51,
10.1016/j.visres.2011.07.008 PubMed CrossRef Google Scholar
Tyler, M. D., Tyler, L., & Burnham, D. K. (2005). The delayed trigger voice key: An improved analogue voice key for psycholinguistic research.
Behavior Research Methods, 37,
10.3758/BF03206408 PubMed CrossRef Google Scholar
Tyson, M. (2011, September 7). Experiments with precise timing in iOS [Web log post]. Retrieved from
Ulrich, R., & Giray, M. (1989). Time resolution of clocks. Effects on reaction time measurement—Good news for bad clocks.
British Journal of Mathematical and Statistical Psychology, 42,
CrossRef Google Scholar
Wang, D., Vaidyanathan, P., Haake, A., & Pelz, J. (2012, November).
Are eye trackers always as accurate as we assume? Paper presented at the annual meeting of the Society for Computers in Psychology (SCip), Minneapolis, Minnesota. Copyright information
© Psychonomic Society, Inc. 2013