Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL

  • Peter D. Turney
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2167)


This paper presents a simple unsupervised learning algorithm for recognizing synonyms, based on statistical data acquired by querying a Web search engine. The algorithm, called PMI-IR, uses Pointwise Mutual Information (PMI) and Information Retrieval (IR) to measure the similarity of pairs of words. PMI-IR is empirically evaluated using 80 synonym test questions from the Test of English as a Foreign Language (TOEFL) and 50 synonym test questions from a collection of tests for students of English as a Second Language (ESL). On both tests, the algorithm obtains a score of 74%. PMI-IR is contrasted with Latent Semantic Analysis (LSA), which achieves a score of 64% on the same 80 TOEFL questions. The paper discusses potential applications of the new unsupervised learning algorithm and some implications of the results for LSA and LSI (Latent Semantic Indexing).


Singular Value Decomposition Semantic Similarity Problem Word Latent Semantic Analysis Query Expansion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Church, K.W., Hanks, P.: Word Association Norms, Mutual Information and Lexicography. In: Proceedings of the 27th Annual Conference of the Association of Computational Linguistics, (1989) 76–83.Google Scholar
  2. 2.
    Church, K.W., Gale, W., Hanks, P., Hindle, D.: Using Statistics in Lexical Analysis. In: Uri Zernik (ed.), Lexical Acquisition: Exploiting On-Line Resources to Build a Lexicon. New Jersey: Lawrence Erlbaum (1991) 115–164.Google Scholar
  3. 3.
    Alta Vista, Alta Vista Company, Palo Alto, California,
  4. 4.
    Test of English as a Foreign Language (TOEFL), Educational Testing Service, Princeton, New Jersey,
  5. 5.
    Tatsuki, D.: Basic 2000 Words-Synonym Match 1. In: Interactive JavaScript Quizzes for ESL Students, (1998).
  6. 6.
    Landauer, T.K., Dumais, S.T.: A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of the Acquisition, Induction, and Representation of Knowledge. Psychological Review, 104 (1997) 211–240.CrossRefGoogle Scholar
  7. 7.
    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by Latent Semantic Analysis. Journal of the American Society for Information Science, 41 (1990) 391–407.CrossRefGoogle Scholar
  8. 8.
    Berry, M.W., Dumais, S.T., Letsche, T.A.: Computational Methods for Intelligent Information Access. Proceedings of Supercomputing’ 95, San Diego, California, (1995).Google Scholar
  9. 9.
    Manning, C.D., Schütze, H.: Foundations of Statistical Natural Language Processing. Cambridge, Massachusetts: MIT Press (1999).zbMATHGoogle Scholar
  10. 10.
    Firth, J.R.: A Synopsis of Linguistic Theory 1930–1955. In Studies in Linguistic Analysis, pp. 1–32. Oxford: Philological Society (1957). Reprinted in F.R. Palmer (ed.), Selected Papers of J.R. Firth 1952–1959, London: Longman (1968).Google Scholar
  11. 11.
    Alta Vista: Alta Vista Advanced Search Cheat Sheet, Alta Vista Company, Palo Alto, California, (2001).
  12. 12.
    Fellbaum, C. (ed.): WordNet: An Electronic Lexical Database. Cambridge, Massachusetts: MIT Press (1998). For more information: Scholar
  13. 13.
    Haase, K.: Interlingual BRICO. IBM Systems Journal, 39 (2000) 589–596. For more information: Scholar
  14. 14.
    Vossen, P. (ed.): EuroWordNet: A Multilingual Database with Lexical Semantic Networks. Dordrecht, Netherlands: Kluwer (1998). See: Scholar
  15. 15.
    Turney, P.D.: Learning Algorithms for Keyphrase Extraction. Information Retrieval, 2 (2000) 303–336.CrossRefGoogle Scholar
  16. 16.
    Grefenstette, G.: Finding Semantic Similarity in Raw Text: The Deese Antonyms. In: R. Goldman, P. Norvig, E. Charniak and B. Gale (eds.), Working Notes of the AAAI Fall Symposium on Probabilistic Approaches to Natural Language. AAAI Press (1992) 61–65.Google Scholar
  17. 17.
    Schütze, H.: Word Space. In: S.J. Hanson, J.D. Cowan, and C.L. Giles (eds.), Advances in Neural Information Processing Systems 5, San Mateo California: Morgan Kaufmann (1993) 895–902.Google Scholar
  18. 18.
    Lin, D.: Automatic Retrieval and Clustering of Similar Words. In: Proceedings of the 17th International Conference on Computational Linguistics and 36th Annual Meeting of the Association for Computational Linguistics, Montreal (1998) 768–773.Google Scholar
  19. 19.
    Richardson, R., Smeaton, A., Murphy, J.: Using WordNet as a Knowledge Base for Measuring Semantic Similarity between Words. In Proceedings of AICS Conference. Trinity College, Dublin (1994).Google Scholar
  20. 20.
    Lee, J.H., Kim, M.H., Lee, Y.J.: Information Retrieval Based on Conceptual Distance in IS-A Hierarchies. Journal of Documentation, 49 (1993) 188–207.CrossRefGoogle Scholar
  21. 21.
    Resnik, P.: Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language. Journal of Artificial Intelligence Research, 11 (1998) 95–130.Google Scholar
  22. 22.
    Jiang, J., Conrath, D.: Semantic Similarity Based on Corpus Statistics and Lexical Taxonomy. In: Proceedings of the 10th International Conference on Research on Computational Linguistics, Taiwan, (1997).Google Scholar
  23. 23.
    Brin, S., Motwani, R., Ullman, J., Tsur, S.: Dynamic Itemset Counting and Implication Rules for Market Basket Data. In: Proceedings of the 1997 ACM-SIGMOD International Conference on the Management of Data (1997) 255–264.Google Scholar
  24. 24.
    Sullivan, D.: Search Engine Sizes., Corporation, Darien, Connecticut, (2000).Google Scholar
  25. 25.
    Papadimitriou, C.H., Raghavan, P., Tamaki, H., Vempala, S.: Latent Semantic Indexing: A Probabilistic Analysis. In: Proceedings of the Seventeenth ACM-SIGACT-SIGMOD-SIGART Symposium on Principles of Database Systems, Seattle, Washington (1998) 159–168.Google Scholar
  26. 26.
    Sparck Jones, K.: Comparison Between TREC2 and TREC3. In: D. Harman (ed.), The Third Text REtrieval Conference (TREC3), National Institute of Standards and Technology Special Publication 500–226, Gaithersburg, Maryland (1994) C1–C4.Google Scholar
  27. 27.
    Buckley, C., Salton, G., Allan, J., Singhal, A.: Automatic Query Expansion Using SMART: TREC 3. In: The Third Text REtrieval Conference (TREC3), D. Harman (ed.), National Institute of Standards and Technology Special Publication 500–226, Gaithersburg, Maryland (1994) 69–80.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2001

Authors and Affiliations

  • Peter D. Turney
    • 1
  1. 1.Institute for Information TechnologyNational Research Council of CanadaOttawaCanada

Personalised recommendations