Cognitive, Affective, & Behavioral Neuroscience

, Volume 13, Issue 3, pp 615–626 | Cite as

Number of events and reliability in fMRI



Relatively early in the history of fMRI, research focused on issues of power and reliability, with an important line concerning the establishment of optimal procedures for experimental design in order to maximize the various statistical properties of such designs. However, in recent years, tasks wherein events are defined only a posteriori, on the basis of behavior, have become increasingly common. Although these designs enable a much wider array of questions to be answered, they are not amenable to the tight control afforded by designs with events defined entirely a priori, and little work has assessed issues of power and reliability in such designs. We demonstrate how differences in numbers of events—as can occur with a posteriori event definition—affect reliability, both through simulation and in real data.


Neuroimaging Design and analysis Statistical power 


Author note

This research was supported by the Institute for Collaborative Biotechnologies through Grant No. W911NF-09-D-0001 from the U.S. Army Research Office. The content of the information does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.


  1. Bennett, C. M., & Miller, M. B. (2010). How reliable are the results from functional magnetic resonance imaging? Annals of the New York Academy of Sciences, 1191, 133–155.PubMedCrossRefGoogle Scholar
  2. Bennett, C. M., & Miller, M. B. (2013). Differences in reliability of block versus event-related designs. Manuscript submitted for publication.Google Scholar
  3. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. doi: 10.1038/nrn3475 PubMedCrossRefGoogle Scholar
  4. Caceres, A., Hall, D. L., Zelaya, F. O., Williams, S. C. R., & Mehta, M. A. (2009). Measuring fMRI reliability with the intra-class correlation coefficient. NeuroImage, 45, 758–768.PubMedCrossRefGoogle Scholar
  5. Carp, J. (2012). The secret lives of experiments: Methods reporting in the fMRI literature. NeuroImage, 63, 289–300.PubMedCrossRefGoogle Scholar
  6. Dale, A. M. (1999). Optimal experimental design for event-related fMRI. Human Brain Mapping, 8, 109–114.PubMedCrossRefGoogle Scholar
  7. Desmond, J. E., & Glover, G. H. (2002). Estimating sample size in functional MRI (fMRI) neuroimaging studies: Statistical power analyses. Journal of Neuroscience Methods, 118, 115–128. doi: 10.1016/S0165-0270(02)00121-8 PubMedCrossRefGoogle Scholar
  8. Eklund, A., Andersson, M., Josephson, C., Johannesson, M., & Knutsson, H. (2012). Does parametric fMRI analysis with SPM yield valid results? An empirical study of 1484 rest datasets. NeuroImage, 61, 565–578.PubMedCrossRefGoogle Scholar
  9. Friston, K. J., Zarahn, E., Josephs, O., Henson, R. N. A., & Dale, A. M. (1999). Stochastic designs in event-related fMRI. NeuroImage, 10, 607–619.PubMedCrossRefGoogle Scholar
  10. Gorgolewski, K. J., Storkey, A. J., Bastin, M. E., Whittle, I., & Pernet, C. (2013). Single subject fMRI test–retest reliability metrics and confounding factors. NeuroImage, 69, 231–243.PubMedCrossRefGoogle Scholar
  11. Green, D. M., & Swets, J. A. (1974). Signal detection theory and psychophysics (Rev. ed). Huntington, NY: Krieger.Google Scholar
  12. Killeen, P. R. (2005a). An alternative to null hypothesis significance tests. Psychological Science, 16, 345–353.PubMedCrossRefGoogle Scholar
  13. Killeen, P. R. (2005b). Replicability, confidence, and priors. Psychological Science, 16, 1009–1012.PubMedCrossRefGoogle Scholar
  14. Liu, T. T., & Frank, L. R. (2004). Efficiency, power, and entropy in event-related fMRI with multiple trial types: Part I. Theory. NeuroImage, 21, 387–400.PubMedCrossRefGoogle Scholar
  15. Liu, T. T., Frank, L. R., Wong, E. C., & Buxton, R. B. (2001). Detection power, estimation efficiency, and predictability in event-related fMRI. NeuroImage, 13, 759–773.PubMedCrossRefGoogle Scholar
  16. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user’s guide. Mahwah, NJ: Erlbaum.Google Scholar
  17. McGonigle, D. J. (2012). Test–retest reliability in fMRI: or how I learned to stop worrying and love the variability. NeuroImage, 62, 1116–1120.PubMedCrossRefGoogle Scholar
  18. McGonigle, D. J., Howseman, A., Athwal, B., Friston, K., Frackowiak, R., & Holmes, A. (2000). Variability in fMRI: An examination of intersession differences. NeuroImage, 11, 708–734.PubMedCrossRefGoogle Scholar
  19. Miller, M. B., Donovan, C. L., Bennett, C. M., Aminoff, E. M., & Mayer, R. E. (2012). Individual differences in cognitive style and strategy predict similarities in the patterns of brain activity between individuals. NeuroImage, 59, 83–93.PubMedCrossRefGoogle Scholar
  20. Miller, M. B., Donovan, C.-L., Van Horn, J. D., German, E., Sokol-Hessner, P., & Wolford, G. L. (2009). Unique and persistent individual patterns of brain activity across different memory retrieval tasks. NeuroImage, 48(3), 625–635.PubMedCrossRefGoogle Scholar
  21. Miller, M. B., Van Horn, J. D., Wolford, G. L., Handy, T. C., Valsangkar-Smyth, M., Inati, S., . . . Gazzaniga, M. S. (2002). Extensive individual differences in brain activations associated with episodic retrieval are reliable over time. Journal of Cognitive Neuroscience, 14, 1200–1214.Google Scholar
  22. Mumford, J. A., & Nichols, T. E. (2008). Power calculation for group fMRI studies accounting for arbitrary design and temporal autocorrelation. NeuroImage, 39, 261–268.PubMedCrossRefGoogle Scholar
  23. Mumford, J. A., Turner, B. O., Ashby, F. G., & Poldrack, R. A. (2012). Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses. NeuroImage, 59, 2636–2643. doi: 10.1016/j.neuroimage.2011.08.076 PubMedCrossRefGoogle Scholar
  24. Power, J. D., Barnes, K. A., Snyder, A. Z., Schlaggar, B. L., & Petersen, S. E. (2011). Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion. NeuroImage, 59, 2142–2154.PubMedCrossRefGoogle Scholar
  25. Turner, B. O., Donovan, C.-L., & Miller, M. B. (2013). Dissociating memory processes by their differences in neural stability. Poster presented at the annual meeting of the Cognitive Neuroscience Society.Google Scholar
  26. Van Dijk, K. R. A., Sabuncu, M. R., & Buckner, R. L. (2012). The influence of head motion on intrinsic functional connectivity MRI. NeuroImage, 59, 431–438.PubMedCrossRefGoogle Scholar
  27. Vul, E., Harris, C., Winkielman, P., & Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality, and social cognition. Perspectives on Psychological Science, 4, 274–290. doi: 10.1111/j.1745-6924.2009.01125.x CrossRefGoogle Scholar
  28. Wager, T. D., & Nichols, T. E. (2003). Optimization of experimental design in fMRI: A general framework using a genetic algorithm. NeuroImage, 18, 293–309.PubMedCrossRefGoogle Scholar
  29. Worsley, K. J., & Friston, K. J. (1995). Analysis of fMRI time-series revisited—again. NeuroImage, 2, 173–181.PubMedCrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  1. 1.Department of Psychological & Brain SciencesUniversity of California, Santa BarbaraSanta BarbaraUSA

Personalised recommendations