Permutation Tests for Classification: Towards Statistical Significance in Image-Based Studies
Estimating statistical significance of detected differences between two groups of medical scans is a challenging problem due to the high dimensionality of the data and the relatively small number of training examples. In this paper, we demonstrate a non-parametric technique for estimation of statistical significance in the context of discriminative analysis (i.e., training a classifier function to label new examples into one of two groups). Our approach adopts permutation tests, first developed in classical statistics for hypothesis testing, to estimate how likely we are to obtain the observed classification performance, as measured by testing on a hold-out set or cross-validation, by chance. We demonstrate the method on examples of both structural and functional neuroimaging studies.
Unable to display preview. Download preview PDF.
- 4.Efron, B.: The Jacknife, The Bootstrap, and Other Resampling Plans. SIAM, Philadelphia, PA (1982)Google Scholar
- 11.Good, P.: Permutation Tests: A Practical guide to Resampling Methods for Testing Hypothesis. Springer, Heidelberg (1994)Google Scholar
- 14.Martin, J., Pentland, A., Kikinis, R.: Shape Analysis of Brain Structures Using Physical and Experimental Models. In: Proceedings of CVPR 1994, pp. 752–755 (1994)Google Scholar
- 16.Thomson, P.M., et al.: Dynamics of Gray Matter Loss in Alzheimer’s Disease. Journal of Neuroscience 23(3) (2003)Google Scholar