, Volume 12, Issue 1, pp 181–208 | Cite as

Template Construction Grammar: From Visual Scene Description to Language Comprehension and Agrammatism

  • Victor Barrès
  • Jinyong Lee
Original Article


How does the language system coordinate with our visual system to yield flexible integration of linguistic, perceptual, and world-knowledge information when we communicate about the world we perceive? Schema theory is a computational framework that allows the simulation of perceptuo-motor coordination programs on the basis of known brain operating principles such as cooperative computation and distributed processing. We present first its application to a model of language production, SemRep/TCG, which combines a semantic representation of visual scenes (SemRep) with Template Construction Grammar (TCG) as a means to generate verbal descriptions of a scene from its associated SemRep graph. SemRep/TCG combines the neurocomputational framework of schema theory with the representational format of construction grammar in a model linking eye-tracking data to visual scene descriptions. We then offer a conceptual extension of TCG to include language comprehension and address data on the role of both world knowledge and grammatical semantics in the comprehension performances of agrammatic aphasic patients. This extension introduces a distinction between heavy and light semantics. The TCG model of language comprehension offers a computational framework to quantitatively analyze the distributed dynamics of language processes, focusing on the interactions between grammatical, world knowledge, and visual information. In particular, it reveals interesting implications for the understanding of the various patterns of comprehension performances of agrammatic aphasics measured using sentence-picture matching tasks. This new step in the life cycle of the model serves as a basis for exploring the specific challenges that neurolinguistic computational modeling poses to the neuroinformatics community.


Neurolinguistics Computational model Construction grammar Visual scene description Schema theory Agrammatism Language comprehension Language production 



This research was supported by the National Science Foundation under Grant No. 0924674 (M.A. Arbib, Principal Investigator). We thank Brad Gasser and Michael Arbib for their fruitful comments on the model.


  1. Aine, C. J., Sanfratello, L., Ranken, D., Best, E., MacArthur, J. A., et al. (2012). MEG-SIM: a web portal for testing MEG analysis methods using realistic simulated and empirical data. Neuroinformatics, 10, 141–158.PubMedCentralPubMedCrossRefGoogle Scholar
  2. Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition, 73, 247–264.PubMedCrossRefGoogle Scholar
  3. Ansell, B. J., & Flowers, C. R. (1982). Aphasic adults’ use of heuristic and structural linguistic cues for sentence analysis. Brain and Language, 16, 61–72.PubMedCrossRefGoogle Scholar
  4. Arbib, M. A. (1981). Perceptual structures and distributed motor control. In V. B. Brooks (Ed.), Handbook of physiology — The nervous system II. Motor control (pp. 1449–1480). Bethesda: American Physiological Society.Google Scholar
  5. Arbib, M. A. (2012). How the brain got language: the mirror system hypothesis. New York & Oxford: Oxford University Press.Google Scholar
  6. Arbib, M. A., & Bota, M. (2003). Language evolution: neural homologies and neuroinformatics. Neural Networks: The Official Journal of the International Neural Network Society, 16, 1237–1260.CrossRefGoogle Scholar
  7. Arbib, M. A., & Caplan, D. (1979). Neurolinguistics must be computational. The Behavioral and Brain Sciences, 2, 449–483.CrossRefGoogle Scholar
  8. Arbib, M. A., & Lee, J. Y. (2008). Describing visual scenes: towards a neurolinguistics based on construction grammar. Brain Research, 1225, 146–162.PubMedCrossRefGoogle Scholar
  9. Arbib, M. A., Conklin, E. J., & Hill, J. C. (1987). From schema theory to language. New York: Oxford University Press. x + 253 pp.Google Scholar
  10. Arbib, M. A., Billard, A., Iacoboni, M., & Oztop, E. (2000). Synthetic brain imaging: grasping, mirror neurons and imitation. Neural Networks: The Official Journal of the International Neural Network Society, 13, 975–997.CrossRefGoogle Scholar
  11. Arbib, M. A., Plangprasopchok, A., Bonaiuto, J. J., Schuler, R. E. (2013). A neuroinformatics of brain modeling and its implementation in the Brain Operation Database BODB. Neuroinformatics, in press.Google Scholar
  12. Baker, C., Fillmore, C. J., Lowe, J. B. (1998). The {B}erkeley {F}rame{N}et project.Google Scholar
  13. Bakker, R., Wachtler, T., Diesmann, M. (2012). CoCoMac 2.0 and the future of tract-tracing databases. Frontiers in Neuroinformatics, 6.Google Scholar
  14. Barrès, V., Simons, A., & Arbib, M. A. (2013). Synthetic event-related potentials: a computational bridge between neurolinguistic models and experiments. Neural Networks, 37, 66–92.PubMedCrossRefGoogle Scholar
  15. Bergen B. K, Chang N. (2005a). Embodied construction grammar in simulation-based language understanding. In Construction Grammars: cognitive grounding and theoretical extensions. Google Scholar
  16. Bergen, B. K., & Chang, N. (2005b). Embodied construction grammar in simulation-based language understanding. In J.-O. OÖstman & M. Fried (Eds.), Construction grammar(s): cognitive and cross-language dimensions (pp. 147–190). Amsterdam: John Benjamins.Google Scholar
  17. Berndt, R. S., & Caramazza, A. (1999). How “regular” is sentence comprehension in Broca's aphasia? It depends on how you select the patients. Brain and Language, 67, 242–247.PubMedCrossRefGoogle Scholar
  18. Berndt, R. S., Mitchum, C. C., & Haendiges, A. N. (1996). Comprehension of reversible sentences in “agrammatism”: a meta-analysis. Cognition, 58, 289–308.PubMedCrossRefGoogle Scholar
  19. Bornkessel, I., & Schlesewsky, M. (2006). The extended argument dependency model: a neurocognitive approach to sentence comprehension across languages. Psychological Review, 113, 787–821.PubMedCrossRefGoogle Scholar
  20. Caramazza, A., & Zurif, E. B. (1976). Dissociation of algorithmic and heuristic processes in language comprehension: evidence from aphasia. Brain and Language, 3, 572–582.PubMedCrossRefGoogle Scholar
  21. Caramazza, A., Capasso, R., Capitani, E., & Miceli, G. (2005). Patterns of comprehension performance in agrammatic Broca’s aphasia: a test of the trace deletion hypothesis. Brain and Language, 94, 43–53.PubMedCrossRefGoogle Scholar
  22. Chambers, C. G., Tanenhaus, M. K., & Magnuson, J. S. (2004). Actions and affordances in syntactic ambiguity resolution. Journal of Experimental Psychology Learning, Memory, and Cognition, 30, 687–696.PubMedCrossRefGoogle Scholar
  23. Chen, R., Hillis, A. E., Pawlak, M., & Herskovits, E. H. (2008). Voxelwise Bayesian lesion-deficit analysis. NeuroImage, 40, 1633–1642.PubMedCentralPubMedCrossRefGoogle Scholar
  24. Christianson, K., & Luke, S. G. (2011). Context strengthens initial misinterpretations of text. Scientific Studies of Reading, 15, 136–166.CrossRefGoogle Scholar
  25. Christianson, K., Hollingworth, A., Halliwell, J. F., & Ferreira, F. (2001). Thematic roles assigned along the garden path linger. Cognitive Psychology, 42, 368–407.PubMedCrossRefGoogle Scholar
  26. Croft, W. (2001). Radical construction grammar: syntactic theory in typological perspective. Oxford: Oxford University Press.CrossRefGoogle Scholar
  27. Croft, W., & Cruse, D. A. (2005). Cognitive linguistics. Cambridge: Cambridge University Press.Google Scholar
  28. David, O., Kiebel, S. J., Harrison, L. M., Mattout, J., Kilner, J. M., & Friston, K. J. (2006). Dynamic causal modeling of evoked responses in EEG and MEG. NeuroImage, 30, 1255–1272.PubMedCrossRefGoogle Scholar
  29. De Beule, J., & Steels, L. (2005). Hierarchy in fluid construction grammar. In U. Furbach (Ed.), Proceedings of the 28th annual German conference on AI, KI 2005, lecture notes in artificial intelligence (vol. 3698) (pp. 1–15). Berlin, Heidelberg: Springer.Google Scholar
  30. Dominey, P. F., & Boucher, J.-D. (2005). Learning to talk about events from narrated video in a construction grammar framework. Artificial Intelligence, 167, 31–61.CrossRefGoogle Scholar
  31. Dominey, P. F., Hoen, M., & Inui, T. (2006a). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.PubMedCrossRefGoogle Scholar
  32. Dominey, P. F., Hoen, M., & Inui, T. (2006b). A neurolinguistic model of grammatical construction processing. Journal of Cognitive Neuroscience, 18, 2088–2107.PubMedCrossRefGoogle Scholar
  33. Dou, D., Frishkoff, G., Rong, J., Frank, R., Malony, A., & Tucker, D. (2007). Development of NeuroElectroMagnetic Ontologies(NEMO): a framework for mining brainwave ontologies. New York: Assoc Computing Machinery.Google Scholar
  34. Draper, B. A., Collins, R. T., Brolio, J., Hanson, A. R., & Riseman, E. M. (1989). The schema system. International Journal of Computer Vision, 2, 209–250.CrossRefGoogle Scholar
  35. Evans. A. C., Collins, D. L., Mills, S. R., Brown, E. D., Kelly, R. L., Peters, T. M. (1993). 1813–17 vol.3-13 -17 vol.3.Google Scholar
  36. Fellbaum, C. (2010). WordNet. In R. Poli, M. Healy, & A. Kameas (Eds.), Theory and applications of ontology: computer applications (pp. 231–43). Springer: Netherlands.CrossRefGoogle Scholar
  37. Ferreira, F. (2003). The misinterpretation of noncanonical sentences. Cognitive Psychology, 47, 164–203.PubMedCrossRefGoogle Scholar
  38. Ferreira, F., & Patson, N. D. (2007). The ‘good enough’ approach to language comprehension. Language and Linguistics Compass, 1, 71–83.CrossRefGoogle Scholar
  39. Fox, P. T., & Lancaster, J. L. (2002). Mapping context and content: the BrainMap model. Nature Reviews Neuroscience, 3, 319–321.PubMedCrossRefGoogle Scholar
  40. Fox, P. T., Laird, A. R., Fox, S. P., Fox, P. M., Uecker, A. M., et al. (2005). Brainmap taxonomy of experimental design: description and evaluation. Human Brain Mapping, 25, 185–198.PubMedCrossRefGoogle Scholar
  41. Frazier, L., & Fodor, J. D. (1978). The sausage machine: a new two-stage parsing model. Cognition, 6, 291–325.CrossRefGoogle Scholar
  42. Friederici, A. D. (2002). Towards a neural basis of auditory sentence processing. Trends in Cognitive Sciences, 6, 78–84.PubMedCrossRefGoogle Scholar
  43. Friederici, A. D. (2009). Pathways to language: fiber tracts in the human brain. Trends in Cognitive Sciences, 13, 175–181.PubMedCrossRefGoogle Scholar
  44. Gleitman, L. R., January, D., Nappa, R., & Trueswell, J. C. (2007). On the give and take between even apprehension and utterance formulation. Journal of Memory and Language, 57, 544–569.PubMedCentralPubMedCrossRefGoogle Scholar
  45. Goldberg, A. E. (1995). Constructions: a construction grammar approach to argument structure. Chicago: The University of Chicago Press.Google Scholar
  46. Goodglass, H. (1976). Agrammatism. Studies in Neurolinguistics, 1, 237–260.Google Scholar
  47. Grodzinsky, Y. (2000). The neurology of syntax: language use without Broca’s area. The Behavioral and Brain Sciences, 23, 1–21.PubMedCrossRefGoogle Scholar
  48. Grodzinsky, Y., Piñango, M. M., Zurif, E., & Drai, D. (1999). The critical role of group studies in neuropsychology: comprehension regularities in Broca’s Aphasia. Brain and Language, 67, 134–147.PubMedCrossRefGoogle Scholar
  49. Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cognitive Sciences, 9, 416–423.PubMedCrossRefGoogle Scholar
  50. Hanson, A. R., & Riseman, E. M. (1978). VISIONS: a computer system for interpreting scenes. In A. R. Hanson & E. M. Riseman (Eds.), Computer vision systems (pp. 129–163). New York: Academic.Google Scholar
  51. Hawkins, J. A. (1999). Processing complexity and Filler-Gap dependencies across grammars. Language & Cognitive Processes, 75, 244–285.Google Scholar
  52. Hickok, G., & Poeppel, D. (2004). Dorsal and ventral streams: a framework for understanding aspects of the functional anatomy of language. Cognition, 92, 67–99.PubMedCrossRefGoogle Scholar
  53. Hurford, J. R. (2011). The origins of grammar II: language in the light of evolution. Oxford, New York: Oxford University Press.Google Scholar
  54. Kay, P. (2002). An informal sketch of the formal architecture of construction grammar. Grammars, 5, 1–19.CrossRefGoogle Scholar
  55. Kay, P., Fillmore, C. J. (1999). Grammatical constructions and linguistic generalizations: the what’s X doing Y? Construction.Google Scholar
  56. Kemmerer, D. (2000a). Grammatically relevant and grammatically irrelevant features of verb meaning can be independently impaired. Aphasiology, 14, 997–1020.CrossRefGoogle Scholar
  57. Kemmerer, D. (2000b). Selective impairment of knowledge underlying prenominal adjective order: evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 13, 57–82.CrossRefGoogle Scholar
  58. Kemmerer, D. (2003). Why can you hit someone on the arm but not break someone on the arm?—a neuropsychological investigation of the English body-part possessor ascension construction. Journal of Neurolinguistics, 16, 13–36.CrossRefGoogle Scholar
  59. Kemmerer, D., & Wright, S. K. (2002). Selective impairment of knowledge underlying un- prefixation: further evidence for the autonomy of grammatical semantics. Journal of Neurolinguistics, 15, 403–432.CrossRefGoogle Scholar
  60. Kemmerer, D., Tranel, D., & Zdanczyk, C. (2009). Knowledge of the semantic constraints on adjective order can be selectively impaired. Journal of Neurolinguistics, 22, 91–108.PubMedCentralPubMedCrossRefGoogle Scholar
  61. Kempen, G., Olsthoorn, N., & Sprenger, S. (2012). Grammatical workspace sharing during language production and language comprehension: evidence from grammatical multitasking. Language & Cognitive Processes, 27, 345–380.CrossRefGoogle Scholar
  62. Kim, & Osterhout. (2005). The independence of combinatory semantic processing: evidence from event-related potentials. Journal of Memory and Language, 52, 205–225.CrossRefGoogle Scholar
  63. Kos, M., Vosse, T., Dvd, B., & Hagoort, P. (2010). About edible restaurants: conflicts between syntax and semantics as revealed by ERPs. Frontiers in Language Sciences, 1, 222–22.Google Scholar
  64. Kudo, T. (1984). The effect of semantic plausibility on sentence comprehension in aphasia. Brain and Language, 21, 208–218.PubMedCrossRefGoogle Scholar
  65. Kuperberg, G. R. (2007). Neural mechanisms of language comprehension: challenges to syntax. Brain Research, 1146, 23–49.PubMedCrossRefGoogle Scholar
  66. Laird, A. R., Eickhoff, S. B., Kurth, F., Fox, P. M., Uecker, A. M., et al. (2009). ALE meta-analysis workflows via the brainmap database: progress towards a probabilistic functional brain atlas. Frontiers in Neuroinformatics, 3, 23–23.PubMedCentralPubMedCrossRefGoogle Scholar
  67. Lee J. (2012). Linking eyes to mouth: a schema-based computational model for describing visual scenes. Ph.D. Thesis, Computer Science, University of Southern California, Los Angeles, CA.Google Scholar
  68. Lee J. (In preparation-a). Implementing Template Construction Grammar (TCG) for visual scene description.Google Scholar
  69. Lee J. (In preparation-b). The temporal unfolding of eye movements and utterance formulation.Google Scholar
  70. Lesser, V. R., Fennel, R. D., Erman, L. D., & Reddy, D. R. (1975). Organization of the HEARSAY-II speech understanding system. IEEE Transactions on Acoustics, Speech, and Signal Processing, 23, 11–23.CrossRefGoogle Scholar
  71. Letovsky, S. I., Whitehead, S. H., Paik, C. H., Miller, G. A., Gerber, J., et al. (1998). A brain image database for structure/function analysis. AJNR. American Journal of Neuroradiology, 19, 1869–1877.PubMedGoogle Scholar
  72. Levin B. (1993). English verb classes and alternations: a preliminary investigation. University of Chicago Press.Google Scholar
  73. Luria, A. R. (1973). The working brain. Harmondsworth: Penguin.Google Scholar
  74. MacWhinney, B. (2007). The TalkBank Project.Google Scholar
  75. MacWhinney, B., Fromm, D., Forbes, M., & Holland, A. (2011). AphasiaBank: methods for studying discourse. Aphasiology, 25, 1286–1307.PubMedCentralPubMedCrossRefGoogle Scholar
  76. Makkai, A. (1972). Idiom structure in EnglishGoogle Scholar
  77. Marcus, D. S., Harwell, J., Olsen, T., Hodge, M., Glasser, M. F., et al. (2011). Informatics and data mining tools and strategies for the human connectome project. Frontiers in Neuroinformatics, 5.Google Scholar
  78. Mayberry, M., Crocker, M. W., Knoeferle, P. (2006). A connectionist model of the coordinated interplay of scene, utterance, and world knowledge.Google Scholar
  79. Menenti, L., Gierhan, S. M. E., Segaert, K., & Hagoort, P. (2011). Shared language overlap and segregation of the neuronal infrastructure for speaking and listening revealed by functional MRI. Psychological Science, 22, 1173–1182.PubMedCrossRefGoogle Scholar
  80. Miyake, A., Carpenter, P. A., & Just, M. A. (1994). A capacity approach to syntactic comprehension disorders: making normal adults perform like aphasic patients. Cognitive Neuropsychology, 11, 671–717.CrossRefGoogle Scholar
  81. Miyake, A., Carpenter, P. A., & Just, M. A. (1995). Reduced resources and specific impairments in normal and aphasic sentence comprehension. Cognitive Neuropsychology, 12, 651–679.CrossRefGoogle Scholar
  82. Mohanan, T., Wee, L. (1999). Grammatical semantics: evidence for structure in meaning. CSLI.Google Scholar
  83. Osterhout, L., Albert, K., Kuperberg, G. (2007). The neurobiology of sentence comprehension, CiteSeerX.Google Scholar
  84. Pinker, S. (1989). Learnability and cognition: the acquisition of argument structure. Cambridge: The MIT Press. 411 pp.Google Scholar
  85. Poeppel, D., Emmorey, K., Hickok, G., & Pylkkänen, L. (2012). Towards a new neurobiology of language. The Journal of Neuroscience, 32, 14125–14131.PubMedCentralPubMedCrossRefGoogle Scholar
  86. Saffran, E. M., Schwartz, M. F., & Linebarger, M. C. (1998). Semantic influences on thematic role assignment: evidence from normals and aphasics. Brain and Language, 62, 255–297.PubMedCrossRefGoogle Scholar
  87. Schuler, K. K. (2005). VerbNet: a broad-coverage, comprehensive verb lexicon. University of PennsylvaniaGoogle Scholar
  88. Schwartz, M. F., Linebarger, M. C., Saffran, E. M., & Pate, D. S. (1987). Syntactic transparency and sentence interpretation in aphasia. Language & Cognitive Processes, 2, 85–113.CrossRefGoogle Scholar
  89. Segaert, K., Menenti, L., Weber, K., Petersson, K. M., & Hagoort, P. (2012). Shared syntax in language production and language comprehension—an fMRI study. Cerebral Cortex, 22, 1662–1670.PubMedCrossRefGoogle Scholar
  90. Sherman, J. C., & Schweickert, J. (1989). Syntactic and semantic contributions to sentence comprehension in agrammatism. Brain and Language, 37, 419–439.PubMedCrossRefGoogle Scholar
  91. Spivey, M. J., Richardson, D. C., & Fitneva, S. A. (2005). Thinking outside the brain: spatial indices to visual and linguistic information. In J. M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: eye movements and the visual world (pp. 161–190). New York, Hove: Psychology Press.Google Scholar
  92. Sporns, O., Tononi, G., & Kötter, R. (2005). The human connectome: a structural description of the human brain. PLoS Computational Biology, 1, e42–e42.PubMedCentralPubMedCrossRefGoogle Scholar
  93. Steels, L. (1999). The talking heads experiment.Google Scholar
  94. Steels, L., & De Beule, J. (2006). Unify and merge in fluid construction grammar. In P. Vogt, Y. Sugita, E. Tuci, & C. Nehaniv (Eds.), Symbol grounding and beyond, proceedings (pp. 197–223). Berlin: Springer-Verlag Berlin.CrossRefGoogle Scholar
  95. Stefanowitsch, A., & Gries, S. T. (2003). Collostructions: investigating the interaction of words and constructions. International Journal of Corpus Linguistics, 8, 209–243.CrossRefGoogle Scholar
  96. Stephan, K. E., Kamper, L., Bozkurt, A., Burns, G. A. P. C., Young, M. P., & Kotter, R. (2001). Advanced database methodology for the collation of connectivity data on the Macaque brain (CoCoMac). Philosophical Transactions of the Royal Society B: Biological Sciences, 356, 1159–1186.CrossRefGoogle Scholar
  97. Talairach, Tournoux P. (1988). Co-planar stereotaxic atlas of the human brain: 3-dimensional proportional system: an approach to cerebral imaging. Thieme.Google Scholar
  98. Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 1632–1634.PubMedCrossRefGoogle Scholar
  99. Van Essen, D. C. (2009). Lost in localization–but found with foci?! NeuroImage, 48, 14–17.PubMedCentralPubMedCrossRefGoogle Scholar
  100. Vigneau, M., Beaucousin, V., Hervé, P. Y., Duffau, H., Crivello, F., et al. (2006). Meta-analyzing left hemisphere language areas: phonology, semantics, and sentence processing. NeuroImage, 30, 1414–1432.PubMedCrossRefGoogle Scholar
  101. Vosse, T., & Kempen, G. (2009). In defense of competition during syntactic ambiguity resolution. Journal of Psycholinguistic Research, 38, 1–9.PubMedCentralPubMedCrossRefGoogle Scholar
  102. Wendel, K., Väisänen, O., Malmivuo. J., Gencer, N. G., Vanrumste. B., et al. (2009). EEG/MEG source imaging: methods, challenges, and open issues. Intelligence and Neuroscience, 2009: 13:1–13:12–13:1–13:12.Google Scholar
  103. Wilbur, R., Kak, A. (2006). Purdue RVL-SLLL American Sign Language Database. ECE Technical Reports. Google Scholar
  104. Zurif, E. B., & Piñango, M. M. (1999). The existence of comprehension patterns in Broca’s Aphasia. Brain and Language, 70, 133–138.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Neuroscience Graduate Program and USC Brain ProjectUniversity of Southern CaliforniaLos AngelesUSA
  2. 2.Computer Science Department and USC Brain ProjectUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations