Handling OOV Words in Indian-language – English CLIR

  • Parin Chheda
  • Manaal Faruqui
  • Pabitra Mitra
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7224)

Abstract

Because of the lack of resources Cross-lingual information retrieval is a difficult task for many Indian languages. Google Translate provides an easy way of translation from Indian languages to English but due to lexicon limitations most of the out-of-vocabulory words get transliterated letter by letter along with their suffix resulting in an unusually long string. The resulting string often does not match its intended translation which hurts retrieval. We propose an approach to extract the correct word from such strings using word segmentation along with approximate string matching using Soundex algorithm & Levenshtein distance. We evaluate our approach across three Indian languages and find an average improvement of 5.8% MAP on the FIRE-2010 dataset.

Keywords

Mean Average Precision Word Segmentation Indian Language Levenshtein Distance Approximate String Match 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Efron, B., Tibshirani, R.: An Introduction to the Bootstrap. Monographs on Statistics and Applied Probability, vol. 57. Chapman & Hall (1994)Google Scholar
  2. 2.
    Faruqui, M., Majumder, P., Padó, S.: Soundex-based translation correction in urdu–english cross-language information retrieval. In: Proceedings of the Fifth International Workshop On Cross Lingual Information Access, pp. 25–29. Asian Federation of Natural Language Processing, Chiang Mai (2011)Google Scholar
  3. 3.
    Lewis, M.P. (ed.): Ethnologue – Languages of the World, 16th edn. SIL International (2009)Google Scholar
  4. 4.
    Makin, R., Pandey, N., Pingali, P., Varma, V.: Approximate String Matching Techniques for Effective CLIR Among Indian Languages. In: Masulli, F., Mitra, S., Pasi, G. (eds.) WILF 2007. LNCS (LNAI), vol. 4578, pp. 430–437. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  5. 5.
    Mandal, D., Gupta, M., Dandapat, S., Banerjee, P.: Approximate string matching techniques for effective clir among indian languages. In: Proceedings of CLEF, pp. 95–102 (2007)Google Scholar
  6. 6.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval, 1st edn. Cambridge University Press, Cambridge (2008)MATHGoogle Scholar
  7. 7.
    Ounis, I., Amati, G., Plachouras, V., He, B., Macdonald, C., Lioma, C.: Terrier: A High Performance and Scalable Information Retrieval Platform. In: Proceedings of ACM SIGIR 2006 Workshop on Open Source Information Retrieval, Seattle, WA, USA (2006)Google Scholar
  8. 8.
    Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)CrossRefGoogle Scholar
  9. 9.
    Suwanvisat, P., Prasitjutrakul, S.: Thai-English cross-language transliterated word retrieval using soundex technique. In: Proceesings of the National Computer Science and Engineering Conference, Bangkok, Thailand (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Parin Chheda
    • 1
  • Manaal Faruqui
    • 1
  • Pabitra Mitra
    • 1
  1. 1.Computer Science and EngineeringIndian Institute of Technology KharagpurIndia

Personalised recommendations