Permutation Tests for Classification: Towards Statistical Significance in Image-Based Studies

  • Polina Golland
  • Bruce Fischl
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2732)


Estimating statistical significance of detected differences between two groups of medical scans is a challenging problem due to the high dimensionality of the data and the relatively small number of training examples. In this paper, we demonstrate a non-parametric technique for estimation of statistical significance in the context of discriminative analysis (i.e., training a classifier function to label new examples into one of two groups). Our approach adopts permutation tests, first developed in classical statistics for hypothesis testing, to estimate how likely we are to obtain the observed classification performance, as measured by testing on a hold-out set or cross-validation, by chance. We demonstrate the method on examples of both structural and functional neuroimaging studies.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bullmore, E.T., et al.: Global, Voxel, and Cluster Tests, by Theory and Permutation, for a Difference Between Two Groups of Structural MR Images of the Brain. IEEE Transactions on Medical Imaging 18(1), 32–42 (1999)CrossRefGoogle Scholar
  2. 2.
    Csernansky, J.G., et al.: Hippocampal Morphometry in Schizophrenia by High Dimensional Brain Mapping. Proceedings of National Academy of Science 95(19), 11406–11411 (1998)CrossRefGoogle Scholar
  3. 3.
    Dale, A.M., et al.: Cortical Surface-Based Analysis I: Segmentation and Surface Reconstruction. NeuroImage 9, 179–194 (1999)CrossRefGoogle Scholar
  4. 4.
    Efron, B.: The Jacknife, The Bootstrap, and Other Resampling Plans. SIAM, Philadelphia, PA (1982)Google Scholar
  5. 5.
    Fischl, B., et al.: Cortical Surface-Based Analysis II: Inflation, Flattening, a Surface- Based Coordinate System. NeuroImage 9, 195–207 (1999)CrossRefGoogle Scholar
  6. 6.
    Fischl, B., et al.: High-resolution intersubject averaging and a coordinate system for the cortical surface. Human Brain Mapping 8, 272–284 (1999)CrossRefGoogle Scholar
  7. 7.
    Fischl, B., et al.: Measuring the thickness of the human cerebral cortex from magnetic resonance images. PNAS 26, 11050–11055 (2000)CrossRefGoogle Scholar
  8. 8.
    Fischl, B., Liu, A., Dale, A.M.: Automated Manifold Surgery: Constructing Geometrically Accurate and Topologically Correct Models of the Human Cerebral Cortex. IEEE Transactions on Medical Imaging 20(1), 70–80 (2001)CrossRefGoogle Scholar
  9. 9.
    Gerig, G., et al.: Shape versus Size: Improved Understanding of the Morphology of Brain Structures. In: Niessen, W.J., Viergever, M.A. (eds.) MICCAI 2001. LNCS, vol. 2208, pp. 24–32. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  10. 10.
    Golland, P., et al.: Discriminative Analysis for Image-Based Studies. In: Dohi, T., Kikinis, R. (eds.) MICCAI 2002. LNCS, vol. 2488, pp. 508–515. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  11. 11.
    Good, P.: Permutation Tests: A Practical guide to Resampling Methods for Testing Hypothesis. Springer, Heidelberg (1994)Google Scholar
  12. 12.
    Guyon, I., et al.: What Size Test Set Gives Good Error Rate Estimates? IEEE Trans. Pattern Analysis and Machine Intelligence 20(1), 52–64 (1998)CrossRefGoogle Scholar
  13. 13.
    Haxby, J.V., et al.: Distributed and Overlapping Representations of Faces and Objects In Ventral Temporal Cortex. Science 293, 2425–2430 (2001)CrossRefGoogle Scholar
  14. 14.
    Martin, J., Pentland, A., Kikinis, R.: Shape Analysis of Brain Structures Using Physical and Experimental Models. In: Proceedings of CVPR 1994, pp. 752–755 (1994)Google Scholar
  15. 15.
    Nichols, T.E., Holmes, A.P.: Nonparametric Permutation Tests For Functional Neuroimaging: A Primer with Examples. Human Brain Mapping 15, 1–25 (2001)CrossRefGoogle Scholar
  16. 16.
    Thomson, P.M., et al.: Dynamics of Gray Matter Loss in Alzheimer’s Disease. Journal of Neuroscience 23(3) (2003)Google Scholar
  17. 17.
    Sachs, L.: Applied Statistics: A Handbook of Techniques. Springer, Heidelberg (1984)MATHGoogle Scholar
  18. 18.
    Spiridon, M., Kanwisher, N.: How distributed is visual category information in human occipito-temporal cortex? An fMRI study. Neuron 35(6), 1157–1165 (2002)CrossRefGoogle Scholar
  19. 19.
    Vapnik, V.N.: Statistical Learning Theory. John Wiley & Sons, Chichester (1998)MATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Polina Golland
    • 1
  • Bruce Fischl
    • 2
  1. 1.Artificial Intelligence LaboratoryMassachusetts Institute of TechnologyCambridge
  2. 2.Athinoula A. Martinos Center for Biomedical ImagingMassachusetts General Hospital, Harvard Medical SchoolBoston

Personalised recommendations