Abstract
Most neuroimaging experiments are under-powered, limited by the number of subjects and cognitive processes that an individual study can investigate. Nonetheless, over decades of research, neuroscience has accumulated an extensive wealth of results. It remains a challenge to digest this growing knowledge base and obtain new insights since existing meta-analytic tools are limited to keyword queries. In this work, we propose Text2Brain, a neural network approach for coordinate-based meta-analysis of neuroimaging studies to synthesize brain activation maps from open-ended text queries. Combining a transformer-based text encoder and a 3D image generator, Text2Brain was trained on variable-length text snippets and their corresponding activation maps sampled from 13,000 published neuroimaging studies. We demonstrate that Text2Brain can synthesize anatomically-plausible neural activation patterns from free-form textual descriptions of cognitive concepts. Text2Brain is available at https://braininterpreter.com as a web-based tool for retrieving established priors and generating new hypotheses for neuroscience research.
Keywords
- Coordinate-based meta-analysis
- Transformers
- Information retrieval
- Image generation
G. H. Ngo and M. Nguyen—Equal contribution. N. F. Chen and M. R. Sabuncu—Equal contribution.
This is a preview of subscription content, access via your institution.
Buying options
References
Carp, J.: The secret lives of experiments: methods reporting in the fMRI literature. Neuroimage 63(1), 289–300 (2012)
Button, K.S., et al.: Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14(5), 365–376 (2013)
Church, J.A., Petersen, S.E., Schlaggar, B.L.: The “Task B problem’’ and other considerations in developmental functional neuroimaging. Hum. Brain Mapp. 31(6), 852–862 (2010)
Costafreda, S.G., Brammer, M.J., David, A.S., Fu, C.H.Y.: Predictors of amygdala activation during the processing of emotional stimuli: a meta-analysis of 385 pet and FMRI studies. Brain Res. Rev. 58(1), 57–70 (2008)
Minzenberg, M.J., Laird, A.R., Thelen, S., Carter, C.S., Glahn, D.C.: Meta-analysis of 41 functional neuroimaging studies of executive function in schizophrenia. Arch. Gen. Psychiatry 66(8), 811–822 (2009)
Shackman, A.J., Salomons, T.V., Slagter, H.A., Fox, A.S., Winter, J.J., Davidson, R.J.: The integration of negative affect, pain and cognitive control in the cingulate cortex. Nat. Rev. Neurosci. 12(3), 154–167 (2011)
Smallwood, J.: Distinguishing how from why the mind wanders: a process-occurrence framework for self-generated mental activity. Psychol. Bull. 139(3), 519 (2013)
Andrews-Hanna, J.R., Smallwood, J., Spreng, R.N.: The default network and self-generated thought: component processes, dynamic control, and clinical relevance. Ann. N. Y. Acad. Sci. 1316(1), 29 (2014)
Yarkoni, T., Poldrack, R.A., Nichols, T.E., Van Essen, D.C., Wager, T.D.: Large-scale automated synthesis of human functional neuroimaging data. Nat. Methods 8(8), 665–670 (2011)
Dockès, J., et al.: NeuroQuery, comprehensive meta-analysis of human brain mapping. Elife 9, e53385 (2020)
Rubin, T.N., Koyejo, O., Gorgolewski, K.J., Jones, M.N., Poldrack, R.A., Yarkoni, T.: Decoding brain activity using a large-scale probabilistic functional-anatomical atlas of human cognition. PLoS Comput. Biol. 13(10), e1005649 (2017)
Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Tenney, I., et al.: What do you learn from context? Probing for sentence structure in contextualized word representations. arXiv preprint arXiv:1905.06316 (2019)
Petroni, F., et al.: Language models as knowledge bases? In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2463–2473 (2019)
Lancaster, J.L., et al.: Bias between MNI and Talairach coordinates analyzed using the ICBM-152 brain template. Hum. Brain Mapp. 28(11), 1194–1205 (2007)
Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Proceedings of ICLR (2018)
Barch, D.M., et al.: Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80, 169–189 (2013)
Pinho, A.L., et al.: Individual brain charting dataset extension, second release of high-resolution fMRI data for cognitive mapping. Sci. Data 7(1), 1–16 (2020)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Inf. Process. Manag. 24(5), 513–523 (1988)
Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)
Buckner, R.L., Andrews-Hanna, J.R., Schacter, D.L.: The brain’s default network: anatomy, function, and relevance to disease (2008)
Ngo, G.H., et al.: Beyond consensus: embracing heterogeneity in curated neuroimaging meta-analysis. Neuroimage 200, 142–158 (2019)
Turkeltaub, P.E., Eden, G.F., Jones, K.M., Zeffiro, T.A.: Meta-analysis of the functional neuroanatomy of single-word reading: method and validation. Neuroimage 16(3), 765–780 (2002)
Laird, A.R., et al.: ALE meta-analysis: controlling the false discovery rate and performing statistical contrasts. Hum. Brain Mapp. 25(1), 155–164 (2005)
Eickhoff, S.B., Laird, A.R., Grefkes, C., Wang, L.E., Zilles, K., Fox, P.T.: Coordinate-based activation likelihood estimation meta-analysis of neuroimaging data: a random-effects approach based on empirical estimates of spatial uncertainty. Hum. Brain Mapp. 30(9), 2907–2926 (2009)
Spreng, R.N., Mar, R.A., Kim, A.S.N.: The common neural basis of autobiographical memory, prospection, navigation, theory of mind, and the default mode: a quantitative meta-analysis. J. Cogn. Neurosci. 21(3), 489–510 (2009)
Mar, R.A.: The neural bases of social cognition and story comprehension. Annu. Rev. Psychol. 62, 103–134 (2011)
Sevinc, G., Spreng, R.N.: Contextual and perceptual brain processes underlying moral cognition: a quantitative meta-analysis of moral reasoning and moral emotions. PLoS ONE 9(2), e87427 (2014)
Acknowledgment
This work was supported by NIH grants R01LM012719, R01AG053949, the NSF NeuroNex grant 1707312, the NSF CAREER 1748377 grant and Jacobs Scholar Fellowship.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ngo, G.H., Nguyen, M., Chen, N.F., Sabuncu, M.R. (2021). Text2Brain: Synthesis of Brain Activation Maps from Free-Form Text Query. In: , et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12907. Springer, Cham. https://doi.org/10.1007/978-3-030-87234-2_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-87234-2_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87233-5
Online ISBN: 978-3-030-87234-2
eBook Packages: Computer ScienceComputer Science (R0)
-
Published in cooperation with
http://miccai.org/
