Could millisecond timing errors in commonly used equipment be a cause of replication failure in some neuroscience studies?
Neuroscience is a rapidly expanding field in which complex studies and equipment setups are the norm. Often these push boundaries in terms of what technology can offer, and increasingly they make use of a wide range of stimulus materials and interconnected equipment (e.g., magnetic resonance imaging, electroencephalography, magnetoencephalography, eyetrackers, biofeedback, etc.). The software that bonds the various constituent parts together itself allows for ever more elaborate investigations to be carried out with apparent ease. However, research over the last decade has suggested a growing, yet underacknowledged, problem with obtaining millisecond-accurate timing in some computer-based studies. Crucially, timing inaccuracies can affect not just response time measurements, but also stimulus presentation and the synchronization between equipment. This is not a new problem, but rather one that researchers may have assumed had been solved with the advent of faster computers, state-of-the-art equipment, and more advanced software. In this article, we highlight the potential sources of error, their causes, and their likely impact on replication. Unfortunately, in many applications, inaccurate timing is not easily resolved by utilizing ever-faster computers, newer equipment, or post-hoc statistical manipulation. To ensure consistency across the field, we advocate that researchers self-validate the timing accuracy of their own equipment whilst running the actual paradigm in situ.
KeywordsReplication Millisecond timing accuracy Millisecond timing error Experiment generators Equipment error
- Brown, W., Malveau, R., McCormick, S., & Mowbray, T. (1998). AntiPatterns: Refactoring software, architectures, and projects in crisis. New York, NY: Wiley.Google Scholar
- Killion, M. C. (1984). New insert headphones for audiometry. Hearing Instruments, 35, 46.Google Scholar
- McDonnell, S. (2004). Code complete: A practical handbook of software construction. Redmond, WA: Microsoft Press.Google Scholar
- Plant, R., & Hammond, N. V. (2001a). Benchmarking the timing characteristics of tools used by behavioural scientists. Abstracts of the Psychonomic Society (42nd Annual Meeting), 6, 109a.Google Scholar
- Plant, R., & Hammond, N. V. (2001b, November). Towards an experimental timing standards laboratory. Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Orlando, Florida.Google Scholar
- Plant, R., & Turner, G. (2012, November). Could your equipment account for your experimental effect? Paper presented at the annual meeting of the Society for Computers in Psychology (SCiP), Minneapolis, Minnesota.Google Scholar
- Tyson, M. (2011, September 7). Experiments with precise timing in iOS [Web log post]. Retrieved from http://atastypixel.com/blog/experiments-with-precise-timing-in-ios/
- Wang, D., Vaidyanathan, P., Haake, A., & Pelz, J. (2012, November). Are eye trackers always as accurate as we assume? Paper presented at the annual meeting of the Society for Computers in Psychology (SCip), Minneapolis, Minnesota.Google Scholar