Advances in Information Retrieval

Volume 4425 of the series Lecture Notes in Computer Science pp 16-27

Similarity Measures for Short Segments of Text

  • Donald MetzlerAffiliated withUniversity of Massachusetts, Amherst, MA
  • , Susan DumaisAffiliated withMicrosoft Research, Redmond, WA
  • , Christopher MeekAffiliated withMicrosoft Research, Redmond, WA

* Final gross prices may vary according to local VAT.

Get Access


Measuring the similarity between documents and queries has been extensively studied in information retrieval. However, there are a growing number of tasks that require computing the similarity between two very short segments of text. These tasks include query reformulation, sponsored search, and image retrieval. Standard text similarity measures perform poorly on such tasks because of data sparseness and the lack of context. In this work, we study this problem from an information retrieval perspective, focusing on text representations and similarity measures. We examine a range of similarity measures, including purely lexical measures, stemming, and language modeling-based measures. We formally evaluate and analyze the methods on a query-query similarity task using 363,822 queries from a web search log. Our analysis provides insights into the strengths and weaknesses of each method, including important tradeoffs between effectiveness and efficiency.