Pitt at CLEF05: Data Fusion for Spoken Document Retrieval
This paper describes an investigation of data fusion techniques for spoken document retrieval. The effectiveness of retrievals solely based on the outputs from automatic speech recognition (ASR) is subject to the recognition errors introduced by the ASR process. This is especially true for retrievals on Malach test collection, whose ASR outputs have average word error rate (WER) of 35%. To overcome the problem, in this year CLEF experiments, we explored data fusion techniques for integrating the manually generated metadata information, which is provided for every Malach document, with the ASR outputs. We concentrated our effort on the post-search data fusion techniques, where multiple retrieval results using automatic generated outputs or human metadata were combined. Our initial studies indicated that a simple unweighted combination method (i.e., CombMNZ) that had demonstrated to be useful in written text retrieval environment only generated significant 38% relative decrease in retrieval effectiveness (measured by Mean Average Precision) for our task by comparing to a simple retrieval baseline where all manual metadata and ASR outputs are put together. This motivated us to explore a more elaborated weighted data fusion model, where the weights are associated with each retrieval result, and can be specified by the user in advance. We also explored multiple iterations of data fusion in our weighted fusion model, and obtained further improvement at 2nd iteration. In total, our best run on data fusion obtained 31% significant relative improvement over the simple fusion baseline, and 4% relative improvement over the manual-only baseline, which is a significant difference.
KeywordsData Fusion Automatic Speech Recognition Mean Average Precision Retrieval Result Multiple Iteration
Unable to display preview. Download preview PDF.
- 1.Belkin, N.J., Cool, C., Croft, W.B., Callan, J.P.: The effect of multiple query representations on information retrieval system performance. In: Proceedings of SIGIR 1993, pp. 339–346 (1993)Google Scholar
- 2.Chen, A.: Cross-language retrieval experiments at CLEF 2002. In: Proceedings of CLEF 2002, pp. 28–48 (2002)Google Scholar
- 3.Darwish, K., Oard, D.W.: CLIR Experiments at Maryland for TREC 2002: Evidence Combination for Arabic-English Retrieval. In: Proceedings of TREC 2002, pp. 703–710 (2002)Google Scholar
- 4.Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Proeedings of the 2nd Text REtrieval Conference (TREC-2), pp. 243–252 (1994)Google Scholar
- 5.Lee, J.H.: Analyses of multiple evidence combination. In: Proceeding of SIGIR 1997, pp. 267–276 (1997)Google Scholar
- 6.Oard, D.W., Soergel, D., Doermann, D., Huang, X., Murray, G. C., Wang, J., Ramabhadran, B., Franz, M., Gustman, S.: Building an information retrieval test collection for spontaneous conversational speech. In: Proceedings of SIGIR 2004, pp. 41–48 (2004)Google Scholar
- 7.Rocha, L.M.: Combination of evidence in recommendation systems characterized by distance functions. In: Proceedings of the 2002 World Congress on Computational Intelligence, FUZZ-IEEE 2002, pp. 203–208. IEEE Press, Los Alamitos (2002)Google Scholar