Hidden Markov Model for Term Weighting in Verbose Queries
It has been observed that short queries generally have better performance than their corresponding long versions when retrieved by the same IR model. This is mainly because most of the current models do not distinguish the importance of different terms in the query. Observed that sentence-like queries encode information related to the term importance in the grammatical structure, we propose a Hidden Markov Model (HMM) based method to extract such information to do term weighting. The basic idea of choosing HMM is motivated by its successful application in capturing the relationship between adjacent terms in NLP field. Since we are dealing with queries of natural language form, we think that HMM can also be used to capture the dependence between the weights and the grammatical structures. Our experiments show that our assumption is quite reasonable and that such information, when utilized properly, can greatly improve retrieval performance.
KeywordsHidden Markov Model Verbose Query Term Weighting
Unable to display preview. Download preview PDF.
- 2.Kumaran, G., Allan, J.: A Case for Shorter Queries and Helping Users Create Them. In: Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Rochester, pp. 220–227 (2007)Google Scholar
- 3.Kumaran, G., Allan, J.: Effective and Efficient User Interaction for Long Queries. In: 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 11–18. ACM Press, Singapore (2008)Google Scholar
- 4.Bendersky, M., Croft, W.B.: Discovering Key Concepts in Verbose Queries. In: 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 491–498. ACM Press, Singapore (2008)Google Scholar
- 5.Cao, G., Nie, J., Gao, J., Robertson, S.: Selecting Good Expansion Terms for Pseudo-Relevance Feedback. In: 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 243–250. ACM Press, Singapore (2008)Google Scholar
- 8.Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In: Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, Edmonton, pp. 252–259 (2003)Google Scholar
- 9.Metzler, D., Strohman, T., Zhou, Y., Croft, W.B.: Indri at TREC 2005: Terabyte Track. In: 14th Text Retrieval Conference, Gaithersburg, pp. 175–180 (2005)Google Scholar
- 11.Croft, W.B.: Combining Approaches to Information Retrieval. In: Croft, W.B. (ed.) Advances in Information Retrieval: Recent Research from the Center for Intelligent Information Retrieval, pp. 1–36. Kluwer Academic Publishers (2000)Google Scholar