Analyzing Document Retrievability in Patent Retrieval Settings
Most information retrieval settings, such as web search, are typically precision-oriented, i.e. they focus on retrieving a small number of highly relevant documents. However, in specific domains, such as patent retrieval or law, recall becomes more relevant than precision: in these cases the goal is to find all relevant documents, requiring algorithms to be tuned more towards recall at the cost of precision. This raises important questions with respect to retrievability and search engine bias: depending on how the similarity between a query and documents is measured, certain documents may be more or less retrievable in certain systems, up to some documents not being retrievable at all within common threshold settings. Biases may be oriented towards popularity of documents (increasing weight of references), towards length of documents, favour the use of rare or common words; rely on structural information such as metadata or headings, etc. Existing accessibility measurement techniques are limited as they measure retrievability with respect to all possible queries. In this paper, we improve accessibility measurement by considering sets of relevant and irrelevant queries for each document. This simulates how recall oriented users create their queries when searching for relevant information. We evaluate retrievability scores using a corpus of patents from US Patent and Trademark Office.
Unable to display preview. Download preview PDF.
- 1.Azzopardi, L., Vinay, V.: Retrievability: An evaluation measure for higher order information access tasks. In: Proc. of CIKM 2008, Napa Valley, CA, USA, pp. 561–570 (2008)Google Scholar
- 6.Itoh, H., Mano, H., Ogawa, Y.: Term distillation in patent retrieval. In: Proc. of ACL 2003, Sapporo, Japan, pp. 41–45 (2003)Google Scholar
- 7.Konishi, K., Kitauchi, A., Takaki, T.: Invalidity patent search system at NTT data. In: NTCIR 2004: Proceedings of NTCIR-4 Workshop Meeting, Tokyo, Japan (2004)Google Scholar
- 8.Konishi, K.: Query terms extraction from patent document for invalidity search. In: NTCIR 2005: Proceedings of NTCIR-5 Workshop Meeting, Tokyo, Japan (2005)Google Scholar
- 9.Kontostathis, A., Kulp, S.: The Effect of normalization when recall really matters. In: Proc. of IKE 2008, Las Vegas, Nevada, USA, pp. 96–101 (2008)Google Scholar
- 11.Robertson, S., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proc. of SIGIR 1994, Dublin, Ireland, pp. 345–354 (1994)Google Scholar
- 12.Robertson, S., Zaragoza, H., Taylor, M.: Simple BM25 extension to multiple weighted fields. In: Proc. of CIKM 2004, Washington, D. C., USA, pp. 42–49 (2004)Google Scholar