Skip to main content

Bridging the theoretical gap between semantic representation models without the pressure of a ranking: some lessons learnt from LSA

Cognitive Processing volume 21, pages 1–21 (2020)Cite this article

Abstract

In recent years, latent semantic analysis (LSA) has reached a level of maturity at which its presence is ubiquitous in technology as well as in simulation of cognitive processes. In spite of this, in recent years there has been a trend of subjecting LSA to some criticisms, usually because it is compared to other models in very specific tasks and conditions and sometimes without having good knowledge of what the semantic representation of LSA means, and without exploiting all the possibilities of which LSA is capable other than the cosine. This paper provides a critical review to clarify some of the misunderstandings regarding LSA and other space models. The historical stability of the predecessors of LSA, the representational structure of word meaning and the multiple topologies that could arise from a semantic space, the computation of similarity, the myth that LSA dimensions have no meaning, the computational and algorithm plausibility to account for meaning acquisition in LSA (in contrast to others models based on online mechanisms), the possibilities of spatial models to substantiate recent proposals, and, in general, the characteristics of classic vector models and their ease and flexibility to simulate some cognitive phenomena will be reviewed. The review highlights the similarity between LSA and other techniques and proposes using long LSA experiences in other models, especially in predicting models such as word2vec. In sum, it emphasizes the lessons that can be learned from comparing LSA-based models to other models, rather than making statements about “the best.”

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. The construction–integration model proposed by Kintsch is a spreading activation algorithm. First, a net is constructed with the neighbors of the words in a text. That net is expressed in a matrix which contains the similarities between each neighbor with the others. This matrix is iteratively multiplied by a vector that represents a meaning hypothesis. This process is carried out until the net is stable, in other words, until the change in the mean of the vector activation is lower than a parameterized value. The number of cycles is then the cycle when that value is reached (see Kintsch and Welsch 1991 for details of the original conception).

  2. LSA usually requires more RAM than other models, but math libraries use to implement sparse matrices classes (with no zero entries) to perform the SVD calculations. This makes it possible to analyze big corpora in ordinary computers. Modern engines for large-scale data processing such as Spark allow for larger corpora as well. We assume that Gamallo and Bordag did not use high-performance techniques in that study.

  3. The context unit in LSA is usually misunderstood. Influenced by the information retrieval field and the constrictions of the formats of some software packages, many researchers consider that LSA uses only whole documents (reports, chapters, sections, books, etc.) as a unit of analysis. For this reason, many studies use only that kind of document as context in the initial word-context matrix. This is not accurate. Paragraphs could be a better alternative for cognitive studies. The classical work of Landauer and Dumais (1997) already used paragraphs.

  4. In David Marr’s levels of analysis, the computational level specifies what the system does, that is, the calculations that it performs. The algorithmic level describes how the system performs the calculations at the computational level, that is, it describes the formal representation for the input and output, and the algorithm (functions) for the transformation. The implementational/physical level describes how the representation and algorithm can be realized physically. However, the difference between computational and algorithmic levels is not too clear. Sometimes, computational descriptions are close to algorithmic ones and vice versa.

  5. To calculate Sim(Jamaica, Russia) and Sim(Russia, Jamaica) in Tversky’s (1977) assumption, we must to adjust a set of parameters in its mathematical formula. To do that, we must to identify a priori which word has more features, Russia or Jamaica (if we know more about Russia or about Jamaica) and in what proportion. We think that identifying such parameters values in that formula using an LSA space representation of words would be automatic, because the vector length provides information about word familiarity (an indirect measure of features in a word).

  6. However, note that arbitrary parameterization could result in an over-fitting situation, where modelers pick some parameters that result in a big adjustment to the cognitive process they are interested in, but the same parameters perform badly when accounting for more general cognitive phenomena. This is an open debate about modularization and generalization.

  7. The fact that distributional models as LSA are currently textl-based does not mean that distributional models must be by definition exclusively linguistic. Lenci (2008) remarks that the contextual hypothesis that defines distributional models is not restricted to features extracted from linguistic context alone and could contain extralinguistic features as context (visual, emotional, communicative situations, etc). Probably, something similar to LSA can be applied to perceptual realities, as has been done with SVD or principal components for face and gesture recognition (Chin et al. 2006; Turk and Pentland 1991). Nonetheless, in this paper, to follow the argumentation of the presented studies, we consider that distributional models are exclusively textual-based only.

References

  • Abad FJ, Olea J, Ponsoda V, García C (2011) Medición en ciencias del comportamiento y de la salud. Editorial Síntesis, Madrid

    Google Scholar 

  • Altszyler E, Mariano S, Fernández-Slezak F (2016) Comparative study of LSA versus Word2vec embeddings in small corpora: a case study in dreams database. CoRR abs/1610.01520

  • Andrews M, Vigliocco G, Vinson D (2009) Integrating experiential and distributional data to learn semantic representations. Psychol Rev 116:463–498. https://doi.org/10.1037/a0016261

    Article  PubMed  Google Scholar 

  • Anguera MT (1977) Construcción de modelos en Psicología. Anuario de Psicología 16:35–60

    Google Scholar 

  • Balbi S, Esposito V (1998) Comparing advertising campaigns by means of textual data analysis with external information. In: Mellet S (ed) Actes des 4es Journées internationales d’Analyse statistique des Données Textuelles. UPRESA, Nice, pp 39–47

    Google Scholar 

  • Balbi S, Misuraca M (2006) Rotated canonical correlation analysis for multilingual corpora JADT’06: Actes Des 8es Journées Internationales D’analyse Statistique Des Données Textuelles, pp 99–106

  • Balkenius C, Gärdenfors P (2016) Spaces in the brain: from neurons to meanings. Front Psychol 7:1820

    PubMed  PubMed Central  Google Scholar 

  • Ballesteros S (1993) Representaciones analógicas en percepción y memoria: imágenes, transformaciones mentales y representaciones estructurales. Psicothema 5(1):5–17

    Google Scholar 

  • Banjade R, Maharjan N, Gautam D, Rus V (2017) Pooling word vector representations across models. In: Proceedings of the international conference on computational linguistics and intelligent text processing, Budapest, Hungary

  • Baroni M, Lenci A (2009) One distributional memory, many semantic spaces. In: Proceedings of the workshop on geometrical models of natural language semantics. Association for Computational Linguistics, pp 1–8

  • Baroni M, Dinu G, Kruszewski G (2014) Don’t count, predict! A systematic comparison of context-counting versus context-predicting semantic vectors. In: Proceedings of the 52nd annual meeting of the association for computational linguistics, ACL 2014, pp 238–247

  • Barque-Duran A, Pothos EM, Yearsley JM, Hampton JA, Busemeyer JR, Trueblood JS (2016) Similarity judgments: from classical to complex vector psychological spaces. In: Contextuality from Quantum Physics to Psychology, pp. 415–448

    Google Scholar 

  • Barsalou LW, Santos A, Simmons WK, Wilson CD (2008) Language and simulation in conceptual processing. In: De Vega M, Glenberg AM, Graesser AC (eds) Symbols, embodiment, and meaning. Oxford University Press, Oxford

    Google Scholar 

  • Beckage N, Smith L, Hills T (2011) Small worlds and semantic network growth in typical and late talkers. PLoS ONE 6(5):e19348

    CAS  PubMed  PubMed Central  Google Scholar 

  • Bergamaschi S, Po L (2014) Comparing LDA and LSA topic models for content-based movie recommendation systems. In: International conference on web information systems and technologies. Springer, pp 247–263

  • Bestgen Y, Vincze N (2012) Checking and bootstrapping lexical norms by means of word similarity indices. Behav Res Methods 44:998–1006

    PubMed  Google Scholar 

  • Bhatia S (2017) Associative judgment and vector space semantics. Psychol Rev 124(1):1

    PubMed  Google Scholar 

  • Biemiller A, Rosenstein M, Sparks R, Landauer TK, Foltz PW (2014) Models of vocabulary acquisition: direct tests and text-derived simulations of vocabulary growth. Sci Stud Read 18(2):130–154

    Google Scholar 

  • Binder JR, Westbury CF, McKiernan KA, Possing ET, Medler DA (2005) Distinct brain systems for processing concrete and abstract concepts. J Cogn Neurosci 17:905–917. https://doi.org/10.1162/0898929054021102

    CAS  Article  PubMed  Google Scholar 

  • Biro I, Benczur A, Szabo J, Maguitman A (2008) A comparative analysis of latent variable models for web page classification. In: Proceedings of the 2008 Latin American web conference. IEEE Computer Society, Washington, pp 23–28

  • Bolukbasi T, Chang KW, Zou, JY, Saligrama VK, Adam T (2016) Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In: Advances in neural information processing systems, pp 4349–4357

  • Burgess C, Nobel K (2017) Avoiding the comparing apples to oranges problem in model comparison. In: Proceedings of the 47th annual meeting of the society for computers in psychology (SCiP). Vancouver

  • Cederberg S, Widdows D (2003) Using LSA and noun coordination information to improve the precision and recall of automatic hyponymy extraction. In: Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4. Association for Computational Linguistics, pp 111–118

  • Chen RC, Lee YC, Pan RH (2006) Adding new concepts on the domain ontology based on semantic similarity. In: International conference on business and information, pp 12–14

  • Chen PN, Chen KY, Chen B (2011) Leveraging relevance cues for improved spoken document retrieval. In Twelfth Annual Conference of the International Speech Communication Association

  • Chin TJ, Schindler K, Suter D (2006) Incremental kernel SVD for face recognition with image sets. In: 7th International conference on automatic face and gesture recognition. FGR 2006. IEEE, pp 461–466

  • Chiru CG, Rebedea T, Ciotec S (2014) Comparison between LSA-LDA-Lexical chains. In: WEBIST (2), pp 255–262

  • Chwilla DJ, Kolk HH (2002) Three-step priming in lexical decision. Mem Cogn 30(2):217–225

    Google Scholar 

  • Cooper RP, Peebles D (2018) On the relation between marr’s levels: a response to blokpoel (2017). Top Cogn Sci 10(3):649–653

    PubMed  Google Scholar 

  • Dascalu M, McNamara DS, Crossley S, Trausan-Matu S (2016) Age of exposure: a model of word learning. In: Thirtieth AAAI conference on artificial intelligence

  • Denhière GY, Lemaire B (2004) A computational model of children’s semantic memory. In: Forbus K, Gentner D, Regier YT (eds) Proceedings of the 26th annual meeting of the cognitive science society. Chicago, pp 297–302

  • Edelman S (1995) Representation, similarity, and the chorus of prototypes. Mind Mach 5(1):45–68

    Google Scholar 

  • Elman JL (1993) Learning and development in neural networks: the importance of starting small. Cognition 48(1):71–99

    CAS  PubMed  Google Scholar 

  • Elvevåg B, Foltz PW, Rosenstein M, DeLisi LE (2010) An automated method to analyze language use in patients with schizophrenia and their first-degree relatives. J Neurolinguist 23(3):270–284. https://doi.org/10.1016/j.jneuroling.2009.05.002

    Article  Google Scholar 

  • Evangelopoulos NE (2013) Latent semantic analysis. Cogn Sci 4:683–692

    Google Scholar 

  • Evangelopoulos N, Visinescu L (2012) Text-mining the voice of the people. Commun ACM 55:62–69

    Google Scholar 

  • Evangelopoulos N, Zhang X, Prybutok VR (2012) Latent semantic analysis: five methodological recommendations. Eur J Inf Syst 21(1):70–86

    Google Scholar 

  • Fabrigar LR, Wegener DT, MacCallum RC, Strahan EJ (1999) Evaluating the use of exploratory factor analysis in psychological research. Psychol Methods 4(3):272

    Google Scholar 

  • Feyerabend P (1974) Contra el método. Ensayo de una teoría anarquista del conocimiento. Ariel Quincenal

  • Field AP, Schorah H (2007) The verbal information pathway to fear and heart rate changes in children. J Child Psychol Psychiatry 48(11):1088–1093

    PubMed  Google Scholar 

  • Furnas GW, Gomez LM, Landauer TK, Dumais ST (1982) Statistical semantics: How can a computer use what people name things to guess what things people mean when they name things? In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 251–253

  • Gamallo P, Bordag S (2011) Is singular value decomposition useful for word similarity extraction? Lang Resour Eval 45(2):95–119

    Google Scholar 

  • García-Palacios A, Costa A, Castilla D, del Río E, Casaponsa A, Duñabeitia JA (2018) The effect of foreign language in fear acquisition. Sci Rep 8:1157

    PubMed  PubMed Central  Google Scholar 

  • Gärdenfors P (1996) Conceptual spaces as a basis for cognitive semantics. In: Philosophy and cognitive science: categories, consciousness, and reasoning. Springer, Dordrecht, pp 159–180

    Google Scholar 

  • Gärdenfors P (2000) Conceptual spaces: on the geometry of thought. MIT Press, Cambridge

    Google Scholar 

  • Glenberg AM, Mehta S (2008) Constraint on covariation: it’s not meaning. Italian J Linguist 20:33–53

    Google Scholar 

  • Goldberg Y, Levy O (2014) word2vec Explained: deriving Mikolov et al.’s negative-sampling word-embedding method. arXiv preprint arXiv:1402.3722

  • Griffiths TL, Steyvers M, Tenenbaum JB (2007) Topics in semantic representation. Psychol Rev 114(2):211

    PubMed  Google Scholar 

  • Günther F, Dudschig C, Kaup B (2016) Latent semantic analysis cosines as a cognitive similarity measure: evidence from priming studies. Quart J Exp Psychol 69(4):626–653

    Google Scholar 

  • Günther F, Dudschig C, Kaup B (2018) Symbol grounding without direct experience: Do words inherit sensorimotor activation from purely linguistic context? Cogn Sci. 42:336–374. https://doi.org/10.1111/cogs.12549

    Article  PubMed  Google Scholar 

  • Günther F, Rinaldi L, Marelli M (2019) Vector-Space Models of Semantic Representation From a Cognitive Perspective: A Discussion of Common Misconceptions. Perspect Psychol Sci. https://doi.org/10.1177/1745691619861372

    Article  PubMed  Google Scholar 

  • Hadley RF (2004) On the proper treatment of semantic systematicity. Mind Mach 14:145–172

    Google Scholar 

  • Haig BD (2013) Analogical modeling: a strategy for developing theories in psychology. Front Psychol 4:348

    PubMed  PubMed Central  Google Scholar 

  • Harnad S (1990) The symbol grounding problem. Physica D 42:335–346

    Google Scholar 

  • Hauk O, Johnsrude I, Pulvermüller F (2004) Somatotopic representation of action words in human motor and premotor cortex. Neuron 41(2):301–307

    CAS  PubMed  Google Scholar 

  • Hintzman DL (1984) MINERVA 2: a simulation model of human memory. Behav Res Methods Instrum Comput 16(2):96–101

    Google Scholar 

  • Hoffman P, Rogers TT, Lambon Ralph MA (2011) Semantic diversity accounts for the “missing” word frequency effect in stroke aphasia: insights using a novel method to quantify contextual variability in meaning. J Cogn Neurosci 23(9):2432–2446

    PubMed  Google Scholar 

  • Hoffman P, Ralph MAL, Rogers TT (2013) Semantic diversity: a measure of semantic ambiguity based on variability in the contextual usage of words. Behav Res Methods 45(3):718–730

    PubMed  Google Scholar 

  • Hoffman P, McClelland JL, Lambon Ralph MA (2018) Concepts, control and context: a connectionist account of normal and disordered semantic cognition. Psychol Rev 125(3):293–328

    PubMed  PubMed Central  Google Scholar 

  • Hofmann MJ, Biemann C, Westbury C, Murusidze M, Conrad M, Jacobs AM (2018) Simple co-occurrence statistics reproducibly predict association ratings. Cogn Sci 42(7):2287–2312. https://doi.org/10.1111/cogs.12662

    Article  PubMed  Google Scholar 

  • Hollis G, Westbury C (2016) The principals of meaning: extracting semantic dimensions from co-occurrence models of semantics. Psychon Bull Rev 23(6):1744–1756. https://doi.org/10.3758/s13423-016-1053-2

    Article  PubMed  Google Scholar 

  • Hollis G, Westbury C, Lefsrud L (2016) Extrapolating human judgments from skip-gram vector representations of word meaning. Quart J Exp Psychol. https://doi.org/10.1080/17470218.2016.1195417

    Article  Google Scholar 

  • Hu X, Cai Z, Wiemer-Hastings P, Graesser AC, McNamara DS (2007) Strengths, limitations, and extensions of LSA. The handbook of latent semantic analysis, pp 401–426

  • Huang P, He X, Gao J, Deng L, Acero A, Heck L (2013) Learning deep structured semantic models for web search using clickthrough data. In: Proceedings of the 22nd ACM international conference on information and knowledge management

  • Huettig F, Quinlan PT, McDonald SA, Altmann GT (2006) Models of high-dimensional semantic space predict language-mediated eye movements in the visual world. Acta Physiol 121(1):65–80

    Google Scholar 

  • Jacobs Am, Kinder A (2018) Features of word similarity. arXiv:1808.07999

  • Jones MN, Mewhort DJK (2007) Representing word meaning and order information in a composite holographic lexicon. Psychol Rev 114:1–37

    PubMed  Google Scholar 

  • Jones M, Gruenfelder T, Recchia G (2011) In defense of spatial models of lexical semantics. In: Proceedings of the annual meeting of the cognitive science society, vol 33, p 33

  • Jones MN, Willits J, Dennis S (2015) Models of semantic memory. In: Busemeyer JR, Wang Z, Townsend JT, Eidels A (eds) Oxford handbook of computational and mathematical psychology. Oxford University Press, Oxford

    Google Scholar 

  • Jones MN, Gruenenfelder TM, Recchia G (2018) In defense of spatial models of semantic representation. New Ideas Psychol 50:54–60

    Google Scholar 

  • Jorge-Botana G, Olmos R (2014) How lexical ambiguity distributes activation to semantic neighbors: some possible consequences within a computational framework. Mental Lexicon 9(1):67–106

    Google Scholar 

  • Jorge-Botana G, León JA, Olmos R, Hassan-Montero Y (2010a) Visualizing polysemy using LSA and the predication algorithm. J Assoc Inf Sci Technol 61(8):1706–1724

    Google Scholar 

  • Jorge-Botana G, Leon JA, Olmos R, Escudero I (2010b) Latent semantic analysis parameters for essay evaluation using small-scale corpora. J Quant Linguist 17(1):1–29

    Google Scholar 

  • Jorge-Botana G, León JA, Olmos R, Escudero I (2011) The representation of polysemy through vectors: some building blocks for constructing models and applications with LSA. Int J Contin Eng Educ Life Long Learn 21(4):328–342

    Google Scholar 

  • Jorge-Botana G, Olmos R, Sanjosé V (2017a) Predicting word maturity from frequency and semantic diversity: a computational study. Discourse Process 54(8):682–694

    Google Scholar 

  • Jorge-Botana G, Olmos R, Luzón JM (2017b) Word maturity indices with LSA: Why, when, and where is procrustes rotation applied? Cognit Sci. https://doi.org/10.1002/wcs.1457

    Article  Google Scholar 

  • Jorge-Botana G, Olmos R, Luzón JM (2019) Could LSA become a “Bifactor” model? Towards a model with general and group factors. Experts Syst Appl 131(1):71–80

    Google Scholar 

  • Kaiser HF (1958) The varimax criterion for analytic rotation in factor analysis. Psychometrika 23(3):187–200

    Google Scholar 

  • Kakkonen T, Myller N, Sutinen E (2006) Applying latent dirichlet allocation to automatic essay grading. In: Salakoski T, Ginter F, Pyysalo S, Pahikkala T (eds) Advances in natural language processing, vol 4139. FinTAL 2006. Lecture notes in computer science. Springer, Berlin

    Google Scholar 

  • Karanam S, Jorge-Botana G, Olmos R, van Oostendorp H (2017) The role of domain knowledge in cognitive modeling of information search. Inf Ret J 20(5):456–479

    Google Scholar 

  • Kenett YN, Wechsler-Kashi D, Kenett DY, Schwartz RG, Ben-Jacob E, Faust M (2013) Semantic organization in children with cochlear implants: computational analysis of verbal fluency. Front Psychol 4:543

    Google Scholar 

  • Kintsch W (1998) Comprehension: a paradigm for cognition. Cambridge University Press, Cambridge

    Google Scholar 

  • Kintsch W (2000) Metaphor comprehension: a computational theory. Psychon Bull Rev 7(2):257–266

    CAS  PubMed  Google Scholar 

  • Kintsch W (2001) Predication. Cognit Sci 25(2):173–202

    Google Scholar 

  • Kintsch W (2007) Meaning in context. In: Landauer TK, McNamara D, Dennis S, Kintsch W (eds) Handbook of latent semantic analysis. Erlbaum, Mahwah, pp 89–105

    Google Scholar 

  • Kintsch W (2008) Symbol systems and perceptual representations. In: De Vega M, Glenberg A, Graesser A (eds) Symbols and embodiment. Oxford Univ. Press, Oxford, pp 145–164

    Google Scholar 

  • Kintsch W (2014) Similarity as a function of semantic distance and amount of knowledge. Psychol Rev 121(3):559

    PubMed  Google Scholar 

  • Kintsch W, Bowles AR (2002) Metaphor comprehension: What makes a metaphor difficult to understand? Metaphor Symbol 17(4):249–262

    Google Scholar 

  • Kintsch W, Mangalath P (2011) The construction of meaning. Top Cognit Sci 3(2):346–370

    Google Scholar 

  • Kintsch W, Welsch D (1991) The construction-integration model: a framework for studying memory for text. In: Hockley WE, Lewandowsky S (eds) Relating theory and data: essays on human memory in honor of Bennet B. Murdock. Erlbaum, Hillsdale, pp 367–385

    Google Scholar 

  • Kireyev K, Landauer TK (2011) Word maturity: computational modeling of word knowledge. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, vol 1. Association for Computational Linguistics, pp 299–308

  • Kivisaari SL, van Vliet M, Hulten A, Lindh-Knuutila T, Faisal A, Salmelin R (in press) Reconstructing meaning from bits of information bioRxiv 401380

  • Krumhansl CL (1978) Concerning the applicability of geometric models to similarity data: the interrelationship between similarity and spatial density. Psychol Rev 85:445–463

    Google Scholar 

  • Kuhlmann M, Hofmann MJ, Jacobs AM (2017) If you don’t have valence, ask your neighbor: evaluation of neutral words as a function of affective semantic associates. Front Psychol 8(343):1–7

    Google Scholar 

  • Kundu A, Jain V, Kumar S, Chandra C (2015) A journey from normative to behavioral operations in supply chain management: a review using latent semantic analysis. Expert Syst Appl 42(2):796–809

    Google Scholar 

  • Kwantes PJ (2005) Using context to build semantics. Psychon Bull Rev 12(4):703–710

    PubMed  Google Scholar 

  • Landauer T (1999) Latent semantic analysis (LSA), a disembodied learning machine, acquires human word meaning vicariously from language alone. Behav Brain Sci 22(4):624–625

    Google Scholar 

  • Landauer TK, Dumais ST (1997) A solution to Plato’s problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol Rev 104(2):211

    Google Scholar 

  • Landauer TK, Foltz PW, Laham D (1998) An introduction to latent semantic analysis. Discourse Process 25(2–3):259–284

    Google Scholar 

  • Landauer TK, Kireyev K, Panaccione C (2011) Word maturity: a new metric for word knowledge. Sci Stud Read 15(1):92–108

    Google Scholar 

  • Lebret R, Collobert R (2015) Rehabilitation of count-based models for word vector representations. In: Gelbukh AF (ed) CICLing (1), vol 9041. Lecture notes in computer science. Springer, New York, pp 417–429

    Google Scholar 

  • Lemaire B, Denhière G, Bellissens C, Jhean-Larose S (2006) A computational model for simulating text comprehension. Behav Res Methods 38(4):628–637

    PubMed  Google Scholar 

  • Lenci A (2008) Distributional semantics in linguistic and cognitive research. Ital J Linguist 20(1):1–31

    Google Scholar 

  • Levy O, Goldberg Y (2014) Neural word embedding as implicit matrix factorization. In: Ghahramani Z, Welling M, Cortes C, Lawrence ND, Weinberger KQ (eds) Advances in neural information processing systems, vol 27. Curran Associates Inc, New York, pp 2177–2185

    Google Scholar 

  • Levy O, Goldberg Y, Dagan I (2015) Improving distributional similarity with lessons learned from word embeddings. Trans Assoc Comput Linguist 3:211–225

    Google Scholar 

  • Littman ML, Jiang F, Keim GA (1998) Learning a language-independent representation for terms from a partially aligned corpus. In: ICML, pp 314–322

  • Lofi C (2016) Measuring semantic similarity and relatedness with distributional and knowledge-based approaches. Database Soc Japan J 14:1–9

    Google Scholar 

  • Louwerse MM (2011) Symbol interdependency in symbolic and embodied cognition. Top Cognit Sci 3:273–302. https://doi.org/10.1111/j.1756-8765.2010.01106.x

    Article  Google Scholar 

  • Louwerse M (2018) Knowing the meaning of a word by the linguistic and perceptual company it keeps. Top Cognit Sci 10(3):573–589

    Google Scholar 

  • Louwerse M, Hutchinson S (2012) Neurological evidence linguistic processes precede perceptual simulation in conceptual processing. Front Psychol 3:385

    PubMed  PubMed Central  Google Scholar 

  • Louwerse MM, Zwaan RA (2009) Language encodes geographical information. Cognit Sci 33:51–73. https://doi.org/10.1111/j.1551-6709.2008.01003.x

    Article  Google Scholar 

  • Lund K, Burgess C (1996) Producing high-dimensional semantic spaces from lexical co-occurrence. Behav Res Meth Instrum Comput 28(2):203–208

    Google Scholar 

  • Lund K, Burgess C, Atchley RA (1995) Semantic and associative priming in high-dimensional semantic space. In: Proceedings of the 17th annual conference of the cognitive science society, vol 17, pp 660–665

  • Mandera P, Keuleers E, Brysbaert M (2017) Explaining human performance in psycholinguistic tasks with models of semantic similarity based on prediction and counting: a review and empirical validation. J Mem Lang 97:57–78

    Google Scholar 

  • Mandl T (1998) Tolerant and adaptative information retrieval with neural networks. Technical report. Information Science-University of Hildesheim

  • Mandl T (1999) Efficient preprocessing for information retrieval with neural networks. In: Zimmermann HJ (ed) EUFIT ‘99. 7th European congress on intelligent techniques and soft computing. Aachen, Germany

  • Marr (1982) Vision, San Francisco: W. H. Freeman, pp 18–38, 54–61

  • Martínez-Huertas JÁ, Jastrzebska O, Mencu A, Moraleda J, Olmos R, León JA (2018) Analyzing two automatic assessment LSA methods (Inbuilt Rubric versus Golden Summary) in summaries extracted from expository texts. Psicología Educativa 24(2):85–92

    Google Scholar 

  • McGregor S, Agres K, Rataj K, Purver M, Wiggins G (2019) Re-representing metaphor: modeling metaphor perception using dynamically contextual distributional semantics. Front Psychol 10:765. https://doi.org/10.3389/fpsyg.2019.00765

    Article  PubMed  PubMed Central  Google Scholar 

  • McNamara DS (2011) Computational methods to extract meaning from text and advance theories of human cognition. Top Cognit Sci 3(1):3–17

    Google Scholar 

  • Mehler A, Sichelschmidt L (2006) Reconceptualizing latent semantic analysis in terms of complex network theory. Presented at the second international conference of the german cognitive linguistics association. Munich, Germany

  • Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

  • Millis K, Larson M (2008) Applying the construction-integration framework to aesthetic responses to representational artworks. Discourse Process 45(3):263–287

    Google Scholar 

  • Mitchell TM, Shinkareva SV, Carlson A, Chang KM, Malave VL, Mason RA, Just MA (2008) Predicting human brain activity associated with the meanings of nouns. Science 320(5880):1191–1195

    CAS  PubMed  Google Scholar 

  • Nastase SA, Haxby JV (2017) Structural basis of semantic memory. In: Byrne JH (ed) Learning and memory: a comprehensive reference, 2nd edn. Academic Press, New York, pp 133–151

    Google Scholar 

  • Nicodemus KK, Elvevåg B, Foltz PW, Rosenstein M, Diaz-Asper C, Weinberger DR (2014) Category fluency, latent semantic analysis and schizophrenia: a candidate gene approach. Cortex 55:182–191

    PubMed  Google Scholar 

  • Olmos R, Jorge-Botana G, León JA, Escudero I (2014) Transforming selected concepts into dimensions in latent semantic analysis. Discourse Process 51(5–6):494–510

    Google Scholar 

  • Olmos R, Jorge-Botana G, Luzón JM, Martín-Cordero JI, León JA (2016) Transforming LSA space dimensions into a rubric for an automatic assessment and feedback system. Inf Process Manag 52(3):359–373

    Google Scholar 

  • Ozcan R, Aslandogan YA (2005) Concept-based information access. In: International conference on information technology: coding and computing. ITCC 2005, vol 1. IEEE, pp 794–799

  • Paivio A (1971) Imagery and language. In: Segal SJ (ed) Imagery: current cognitive approaches. Academic, New York, pp 7–32

    Google Scholar 

  • Palmiero M, Piccardi L, Giancola M, Nori R, D’Amico S, Belardinelli MO (2019) The format of mental imagery: from a critical review to an integrated embodied representation approach. Cognit Process. https://doi.org/10.1007/s10339-019-00908-z

    Article  Google Scholar 

  • Pexman PM (2017) The role of embodiment in conceptual development. Lang Cognit Neurosci. https://doi.org/10.1080/23273798.2017.1303522

    Article  Google Scholar 

  • Pothos EM, Busemeyer JR (2013) Can quantum probability provide a new direction for cognitive modeling? Behav Brain Sci 36:255–327

    PubMed  Google Scholar 

  • Quesada JF, Kintsch W, Gomez E (2001) A computational theory of complex problem solving using the vector space model: latent semantic analysis, through the path of thousands of ants. Cognit Res Microworlds 2001:117–131

    Google Scholar 

  • Recchia RG, Louwerse MM (2015) Reproducing affective norms with lexical co-occurrence statistics: predicting valence, arousal, and dominance. Quart J Exp Psychol 68(8):1584–1598

    Google Scholar 

  • Riordan B, Jones MN (2011) Redundancy in perceptual and linguistic experience: comparing feature-based and distributional models of semantic representation. Top Cognit Sci 3(2):303–345. https://doi.org/10.1111/j.1756-8765.2010.01111.x

    Article  Google Scholar 

  • Rogers TT, McClelland JL (2004) Semantic cognition: a parallel distributed processing approach. MIT Press, Cambridge

    Google Scholar 

  • Sainz J (1991) Conceptos naturales y conceptos artificiales. In: En Mayor J, Pinillos J (eds) Pensamiento e inteligencia Tratado de Psicología General. Alhambra, España, pp 181–302

    Google Scholar 

  • Salton G, Wong A, Yang C-S (1975) A vector space model for automatic indexing. Commun ACM 18(11):613–620

    Google Scholar 

  • Schlaggar BL, McCandliss BD (2007) Development of neural systems for reading. Annu Rev Neurosci 30:475–503

    CAS  PubMed  Google Scholar 

  • Schunn CD (1999) The presence and absence of category knowledge in LSA. In: Proceedings of the 21st annual conference of the cognitive science society. Erlbaum, Mahwah

  • Shepard RN, Chipman S (1970) Second-order isomorphism of internal representations: shapes of states. Cognit Psychol 1(1):1–17

    Google Scholar 

  • Shultz TR (2003) Computational developmental psychology. MIT Press, Cambridge

    Google Scholar 

  • Sierra-Vázquez V (1986) Procesamiento visual temprano: Aspectos psicofísicos del análisis espacial de imágenes. In: Peraita H (ed) Psicología cognitiva y ciencia cognitiva. UNED, Madrid, pp 42–126

    Google Scholar 

  • Simmons W Kyle, Hamann SB, Harenski CL, Hu XP, Barsalou LW (2008) fMRI evidence for word association and situated simulation in conceptual processing. J Physiol 102(1–3):106–119. https://doi.org/10.1016/j.jphysparis.2008.03.014

    Article  Google Scholar 

  • Skoyles JR (2011) Autism, context/noncontext information processing, and atypical development. Autism Research Treat 2011:681627. https://doi.org/10.1155/2011/681627

    Article  Google Scholar 

  • Squire LR (1987) Memory and brain. Oxford University Press, New York

    Google Scholar 

  • Steyvers M, Griffiths T (2006) Probabilistic topic models. In: Landauer T, Mcnamara D, Dennis S, Kintsch W (eds) Latent semantic analysis: a road to meaning. Lawrence Erlbaum, Milton Park

    Google Scholar 

  • Steyvers M, Tenenbaum JB (2005) The large-scale structure of semantic networks: statistical analyses and a model of semantic growth. Cognit Sci 29(1):41–78

    Google Scholar 

  • Thurstone LL (1931) The measurement of social attitudes. J Abnormal Soc Psychol 26(3):249

    Google Scholar 

  • Tonta Y, Darvish HR (2010) Diffusion of latent semantic analysis as a research tool: a social network analysis approach. J Informetr 4(2):166–174

    Google Scholar 

  • Turk MA, Pentland AP (1991) Face recognition using eigenfaces. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, CVPR’91. IEEE, pp 586–591

  • Turney PD, Littman M (2003) Measuring praise and criticism: inference of semantic orientation from association. ACM Trans Inf Syst 21:315–346

    Google Scholar 

  • Tversky A (1977) Features of similarity. Psychol Rev 84(4):327–352

    Google Scholar 

  • Van Dijk TA, Kintsch W, Van Dijk TA (1983) Strategies of discourse comprehension. Academic Press, New York, pp 11–12

    Google Scholar 

  • Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small-world’networks. Nature 393(6684):440

    CAS  Google Scholar 

  • Wierzbicka A (1996) Semantics: primes and universals: primes and universals. Oxford University Press, Oxford

    Google Scholar 

  • Wittgenstein L (1953) Philosophical investigations. Philosophische Untersuchungen. Macmillan, Oxford

    Google Scholar 

  • Yan X, Li X, Song D (2004) A correlation analysis on LSA and HAL semantic space models. In: International conference on computational and information science. Springer, Berlin, pp 711–717

    Google Scholar 

  • Yeari M, van den Broek P (2015) The role of textual semantic constraints in knowledge-based inference generation during reading comprehension: a computational approach. Memory 23:1193–1214. https://doi.org/10.1080/09658211.2014.968169

    Article  PubMed  Google Scholar 

  • Yeari M, van den Broek P (2016) A computational modeling of semantic knowledge in reading comprehension: integrating the landscape model with latent semantic analysis. Behav Res Methods 48(3):880–896

    PubMed  Google Scholar 

  • Yearsley JM, Pothos EM, Hampton JA, Duran AB (2015) Towards a quantum probability theory of similarity judgments. Lect Notes Comput Sci 8951:132–145

    Google Scholar 

  • Zwaan RA, Yaxley RH (2003) Spatial iconicity affects semantic relatedness judgments. Psychon Bull Rev 10(4):954–958

    PubMed  Google Scholar 

Download references

Acknowledgements

Some ideas of this paper were presented orally in the Psychology of Thinking and Comprehension: International Meeting in Honor of Prof. Juan Antonio García Madruga. We want to thank to the organization and to Prof. García Madruga for inviting us to debate these ideas. The presentation (Spanish) is in: https://canal.uned.es/video/5a6fa16eb1111f636a8b45bc.

Author information

Affiliations

  1. Universidad Nacional de Educación a Distancia, Juan del Rosal, nº 10, 28023, Madrid, Spain

    Guillermo Jorge-Botana & José María Luzón

  2. Universidad Autónoma de Madrid, Ciudad Universitaria de Cantoblanco, C/Iván Pavlov, s/n., 28049, Madrid, Spain

    Ricardo Olmos

Authors
  1. Guillermo Jorge-Botana
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ricardo Olmos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. José María Luzón
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guillermo Jorge-Botana.

Ethics declarations

Conflict of interest

The author declares no conflict of interest.

Human and animal rights

The research involved no human participants or animals.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Handling editor: Markus Kiefer (University of Ulm).

Reviewers: Markus Hofmann (University of Wuppertal) and a second researcher who prefers to remain anonymous.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jorge-Botana, G., Olmos, R. & Luzón, J.M. Bridging the theoretical gap between semantic representation models without the pressure of a ranking: some lessons learnt from LSA. Cogn Process 21, 1–21 (2020). https://doi.org/10.1007/s10339-019-00934-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10339-019-00934-x

Keywords

  • Latent semantic analysis (LSA)
  • Spatial models
  • Distributional models
  • Topic model
  • word2vec
  • Counting models
  • Predicting models
  • Lexical dynamicity