Cognitive, Affective, & Behavioral Neuroscience

, Volume 13, Issue 4, pp 714–724

Quantifying the reliability of image replication studies: The image intraclass correlation coefficient (I2C2)

  • H. Shou
  • A. Eloyan
  • S. Lee
  • V. Zipunnikov
  • A. N. Crainiceanu
  • M. B. Nebel
  • B. Caffo
  • M. A. Lindquist
  • C. M. Crainiceanu
Article

DOI: 10.3758/s13415-013-0196-0

Cite this article as:
Shou, H., Eloyan, A., Lee, S. et al. Cogn Affect Behav Neurosci (2013) 13: 714. doi:10.3758/s13415-013-0196-0

Abstract

This article proposes the image intraclass correlation (I2C2) coefficient as a global measure of reliability for imaging studies. The I2C2 generalizes the classic intraclass correlation (ICC) coefficient to the case when the data of interest are images, thereby providing a measure that is both intuitive and convenient. Drawing a connection with classical measurement error models for replication experiments, the I2C2 can be computed quickly, even in high-dimensional imaging studies. A nonparametric bootstrap procedure is introduced to quantify the variability of the I2C2 estimator. Furthermore, a Monte Carlo permutation is utilized to test reproducibility versus a zero I2C2, representing complete lack of reproducibility. Methodologies are applied to three replication studies arising from different brain imaging modalities and settings: regional analysis of volumes in normalized space imaging for characterizing brain morphology, seed-voxel brain activation maps based on resting-state functional magnetic resonance imaging (fMRI), and fractional anisotropy in an area surrounding the corpus callosum via diffusion tensor imaging. Notably, resting-state fMRI brain activation maps are found to have low reliability, ranging from .2 to .4. Software and data are available to provide easy access to the proposed methods.

Keywords

RAVENS DTI fMRI Replication studies Intraclass correlation coefficient 

Copyright information

© Psychonomic Society, Inc. 2013

Authors and Affiliations

  • H. Shou
    • 1
  • A. Eloyan
    • 1
  • S. Lee
    • 2
  • V. Zipunnikov
    • 1
  • A. N. Crainiceanu
    • 3
  • M. B. Nebel
    • 4
  • B. Caffo
    • 1
  • M. A. Lindquist
    • 1
  • C. M. Crainiceanu
    • 1
  1. 1.Department of Biostatistics, Bloomberg School of Public HealthJohns Hopkins UniversityBaltimoreUSA
  2. 2.Department of Psychiatry and the Department of BiostatisticsColumbia UniversityNew YorkUSA
  3. 3.Computer Science DepartmentUnited States Naval AcademyMDUSA
  4. 4.Laboratory for Neurocognitive and Imaging ResearchKennedy Krieger InstituteBaltimoreUSA

Personalised recommendations