Machine Learning

, Volume 57, Issue 1–2, pp 145–175 | Cite as

Learning to Decode Cognitive States from Brain Images

  • Tom M. Mitchell
  • Rebecca Hutchinson
  • Radu S. Niculescu
  • Francisco Pereira
  • Xuerui Wang
  • Marcel Just
  • Sharlene Newman


Over the past decade, functional Magnetic Resonance Imaging (fMRI) has emerged as a powerful new instrument to collect vast quantities of data about activity in the human brain. A typical fMRI experiment can produce a three-dimensional image related to the human subject's brain activity every half second, at a spatial resolution of a few millimeters. As in other modern empirical sciences, this new instrumentation has led to a flood of new data, and a corresponding need for new data analysis methods. We describe recent research applying machine learning methods to the problem of classifying the cognitive state of a human subject based on fRMI data observed over a single time interval. In particular, we present case studies in which we have successfully trained classifiers to distinguish cognitive states such as (1) whether the human subject is looking at a picture or a sentence, (2) whether the subject is reading an ambiguous or non-ambiguous sentence, and (3) whether the word the subject is viewing is a word describing food, people, buildings, etc. This learning problem provides an interesting case study of classifier learning from extremely high dimensional (105 features), extremely sparse (tens of training examples), noisy data. This paper summarizes the results obtained in these three case studies, as well as lessons learned about how to successfully apply machine learning methods to train classifiers in such settings.

scientific data analysis functional Magnetic Resonance Imaging high dimensional data feature selection Bayesian classifier Support Vector Machine nearest neighbor brain image analysis 


  1. Aguirre, G. K., Zarahn, E., & D'Esposito, M. (1998). An area within human ventral cortex sensitive to building stimuli: Evidence and implications. Neuron, 21, 373–383.Google Scholar
  2. Anderson, J. R. et al. (2004). An information-processing model of the BOLD response in symbol manipulation tasks. Psychonomic Bulletin and Review (in press).Google Scholar
  3. Battig, W. F., & Montague, W. E. (1968). Category norms for verbal items in 56 categories: A replication and extension of the Connecticut norms. Journal of Experimental Psychology Monograph, 80:3, 1–46.Google Scholar
  4. Blankertz, B., Curio, G., & Mller, K. R. (2002). Classifying single trial EEG: Towards brain computer interfacing. Advances in Neural Inf. Proc. Systems (NIPS 2001), 14, 157–164.Google Scholar
  5. Bly, B. M. (2001). When you have a General Linear Hammer, every fMRI time-series looks like independent identically distributed nails. Concepts and Methods in NeuroImaging Workshop, NIPS.Google Scholar
  6. Burges, C. (1998). A tutorial on support vector machines for pattern recognition. Journal of data Mining and Knowledge Discovery, 2:2, 121–167.Google Scholar
  7. Caviness, V. S., Kennedy, D. N., Bates, J., & Makris, N. J. (1996). MRI-based parcellation of human neocortex: An anatomically specified method with estimate of reliability. Journal of Cognitive Neuroscience, 8, 566–588.Google Scholar
  8. Chao, L., Haxby, J. V., & Martin, A. (1999). Attribute-based neural substrates in temporal cortex for perceiving and knowing about objects. Nature Neuroscience, 2, 913–919.Google Scholar
  9. Chao, L., Weisberg, J., & Martin, A. (2002). Experience-dependent modulation of category-related cortical activity. Cerebral Cortex, 12, 545–551.Google Scholar
  10. Cover, T., & Thomas, J. (1991). Elements of information theory. Wiley and Sons.Google Scholar
  11. Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI) "brain reading": Detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage, 19, 261–270.Google Scholar
  12. Eddy, W. et al. (1998). The challenge of functional magnetic resonance imaging. Journal of Computational and Graphical Statistics, 8:3, 545–558.Google Scholar
  13. Friston, K. J. et al. (1995a). Statistical parametric maps in functional imaging: A general linear approach. Human Brain Mapping, 2, 189–210Google Scholar
  14. Friston, K. J. et al. (1995b). Analysis of fMRI time-series revisited. NeuroImage, 2, 45–53.Google Scholar
  15. Genovese, C. (1999). Statistical inference in functional magnetic resonance imaging. CMU Statistics Tech Report 674.Google Scholar
  16. Goutte, C., Toft, P., Rostrup, E., Nielsen, F. A., & Hansen, L. K. (1998). On clustering fMRi time series. Technical Report IMM-REP-1998-11.Google Scholar
  17. Haxby, J. et al. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293, 2425–2430.Google Scholar
  18. Hojen-Sorensen, P., Hansen, L. K., & Rasmussen, C. E. (1999). Bayesian modeling of fMRI time series. NIPS*99. Denver.Google Scholar
  19. Ishai, A., Ungerleider, L. G., Martin, A., Schouten, J. L., & Haxby, J. V. (1999). Distributed Representation of Objects in the Human Ventral Visual Pathway. Proc. Nat. Acad. Sci. USA, 96, 9379–9384.Google Scholar
  20. Joachims, T. (2001).Astatistical learning model of text classification with support vector machines. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM.Google Scholar
  21. Just, M. A., Carpenter, P.A., & Varma, S. (1999). Computational modeling of high-level cognition and brain function. Human Brain Mapping, 8, 128–136.Google Scholar
  22. Keller, T. A., Just, M. A., & Stenger, V. A. (2001). Reading span and the time-course of cortical activation in sentence-picture verification. Annual Convention of the Psychonomic Society, Orlando, FL.Google Scholar
  23. Kjems, U., Hansen, L., Anderson, J., Frutiger, S., Muley, S., Sidtis, J., Rottenberg, D., & Strother, S. C. (2002). The quantitative evalutation of functional neuroimaging experiments: Mutual information learning curves, NeuroImage, 15, 772–786.Google Scholar
  24. Mason, R., Just, M., Keller, T., & Carpenter, P. (2004). Ambiguity in the Brain: What brain imaging reveals about the processing of syntactically ambiguous sentences, {tiJounal of Experimental Psychology: Learning, Memory, and Cognition} (in press).Google Scholar
  25. McKeown, M. J. et al. (1998). Analysis of fMRI data by blind separation into independent spatial components. {tiHuman Brain Mapping}, 6:3, 160–188.Google Scholar
  26. Menon, R. S., Luknowsky, D. L., & Gati, J. S. (1998). Mental chronometry using latency-resolved functional magnetic resonance imaging. {tiProc. Natl. Acad. Sci (U.S.A.)}, 95, 10902–10907.Google Scholar
  27. Mitchell, T. M. (1997). {atMachine learning}. McGraw-Hill.Google Scholar
  28. Mitchell, T. M., Hutchinson, R., Just, M., Niculescu, S. R., Pereira, F., & Wang, X. (2003). Classifying instantaneous cognitive states from fMRI data. In {tiProceedings of the 2003 Americal Medical Informatics Association Annual Symposium}. Washington D.C.Google Scholar
  29. Nigam, K., McCallum, A., Thrun, S., & Mitchell, T. (2000). Text classification from labeled and unlabeled documents using EM. Machine Learning, 39, 103–134.Google Scholar
  30. Ng, A. Y., & Jordan, M. (2002). On discriminative vs. generative classifiers: A comparison of logistic regression and Naive Bayes., 14.Neural Information Processing Systems Google Scholar
  31. Penny,W. (2001). Mixture models with adaptive spatial priors. Concepts and Methods in NeuroImagingworkshop at NIPS*01, Vancouver, British Columbia, Canada.Google Scholar
  32. Rademacher, J., Galaburda, A. M., Kennedy, D. N., Filipek, P. A., & Caviness, V. S. (1992). Human cerebral cortex: Localization, parcellation, and morphometry with magnetic resonance imaging. Journal of Cognitive Neuroscience, 4, 352–374.Google Scholar
  33. Strother S. C., Anderson, J., Hansen, L., Kjems, U., Kustra, R., Siditis, J., Frutiger, S., Muley, S., LaConte, S., & Rottenberg, D. (2002). The quantitative evaluation of functional neuroimaging experiments: The NPAIRS data analysis framework. Neuroimage, 15, 747–771.Google Scholar
  34. Talairach, J., & Tournoux, P. (1988). Co-planar stereotaxic atlas of the human brain. Thieme, New York.Google Scholar
  35. Wagner, A. D. et al. (1998). Building memories: Remembering and forgetting of verbal experiences as predicted by brain activity. Science, 281, 1188–1191.Google Scholar
  36. Wang, X., Hutchinson, R., & Mitchell, T. M. (2003). Training fMRI classifiers to detect cognitive states across multiple human subjects. In Proceedings of the 2003 Conference on Neural Information Processing Systems, Vancouver.Google Scholar
  37. Yang, Y. (1999). An evaluation of statistical approaches to text categorization. Journal of Information Retrieval, 1:1/2, 67–88.Google Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • Tom M. Mitchell
    • 1
  • Rebecca Hutchinson
    • 1
  • Radu S. Niculescu
    • 1
  • Francisco Pereira
    • 1
  • Xuerui Wang
    • 1
  • Marcel Just
    • 2
  • Sharlene Newman
    • 2
  1. 1.School of Computer ScienceCarnegie Mellon UniversityPortugal
  2. 2.Psychology DepartmentCarnegie Mellon UniversityPortugal

Personalised recommendations