Advertisement

Character-Based N-gram Model for Uyghur Text Retrieval

  • Turdi Tohti
  • Lirui Xu
  • Jimmy Huang
  • Winira Musajan
  • Askar Hamdulla
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10996)

Abstract

Uyghur is a low resourced language, but Uyghur Information Retrieval (IR) is getting more and more important recently. Although there are related research results and stem-based Uyghur IR systems, it is always difficult to obtain high-performance retrieval results due to the limitations of the existing stemming method. In this paper, we propose a character-based N-gram model and the corresponding smoothing algorithm for Uyghur IR. A full-text IR system based on character N-gram model is developed using the open-source tool Lucene. A series of experiments and comparative analysis are conducted. Experimental results show that our proposed method has the better performance compared with conventional Uyghur IR systems.

Keywords

Uyghur Information retrieval Stemming N-gram Lucene 

Notes

Acknowledgments

This work has been supported by the National Natural Science Foundation of China (61562083, 61262062), Western Region Talent Cultivation Special Projects of China Scholarship Council (201608655002).

References

  1. 1.
    Tohti, T., Musajan, W., Hamdull, A.: Design and implementation of Uyghur, Kazak, Kyrgyz web-based full-text search engine. Comput. Appl. Softw. 26(6), 96–98 (2009)Google Scholar
  2. 2.
    Tohti, T., Musajan, W., Hamdull, A.: Key techniques of Uyghur, Kazak, Kyrgyz full-text search engine retrieval server. Comput. Eng. 34(21), 45–47 (2008)Google Scholar
  3. 3.
    Tohti, T., Hamdull, A., Musajan, W.: Research on web text representation and the similarity based on improved VSM in Uyghur web information retrieval. In: Chinese Conference on Pattern Recognition (CCPR 2010), pp. 984–988 (2010)Google Scholar
  4. 4.
    Huang, X., Peng, F., Schuurmans, D., Cercone, N., Robertson, S.: Applying machine learning to text segmentation for information retrieval. Inf. Retr. 6(3), 333–362 (2003)CrossRefGoogle Scholar
  5. 5.
    Beaulieu, M., Gatford, M., Huang, X., Robertson, S., Walker, S., Williams, P.: Okapi at TREC-5. In: Proceedings of the 5th Text Retrieval Conference, National Institute of Standards and Technology (NIST), pp. 238–500, 143–166. NIST Special Publication (1997)Google Scholar
  6. 6.
    Ponte, J.M., Croft, W.B.: A language modeling approach to information retrieval. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 275–281 (1998)Google Scholar
  7. 7.
    Miller, D.R.H., Leek, T., Schwartz, R.M.: A hidden Markov model information retrieval system. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 214–221 (1999)Google Scholar
  8. 8.
    Berger, A., Lafferty, J.: Information retrieval as statistical translation. In: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 222–229 (1999)Google Scholar
  9. 9.
    Jin, R., Hauptmann, A.G., Zhai, C.X.: Language model for information retrieval. In: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 42–48 (2002)Google Scholar
  10. 10.
    Lavrenko, V., Croft, W.B.: Relevance based language models. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 120–127 (2001)Google Scholar
  11. 11.
    Ren, Z.F., Cang, Y.Q., Fan, A.W.: N-Gram statistical information retrieval model based on bayesian theory. J. Zhengzhou Univ. 42(1), 21–23 (2010)Google Scholar
  12. 12.
    Li, X.G., Wang, D.L., Yu, G.: Information retrieval based on statistical language model. Comput. Sci. 32(8), 124–127 (2005)Google Scholar
  13. 13.
    Ablimit, M., Hamdull, A., Kawahara, T.: Morpheme concatenation approach in language modeling for large-vocabulary Uyghur speech recognition. In: International Conference on Speech Database and Assessments (Oriental COCOSDA), pp. 112–115 (2011)Google Scholar
  14. 14.
    Zhang, Y.J.: Study on N-gram language model of Uygur language. Comput. Knowl. Technol. 7(17), 4177–4179 (2011)Google Scholar
  15. 15.
    Song, F., Croft, W.B.: A general language model for information retrieval. In: Proceedings of the Eighth International Conference on Information and Knowledge Management, pp. 316–321 (1999)Google Scholar
  16. 16.
    Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 334–342 (2001)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Turdi Tohti
    • 1
    • 2
  • Lirui Xu
    • 1
  • Jimmy Huang
    • 2
  • Winira Musajan
    • 1
  • Askar Hamdulla
    • 1
  1. 1.School of Information Science and EngineeringXinjiang UniversityÜrümqiChina
  2. 2.Information Retrieval and Knowledge Management Research LabYork UniversityTorontoCanada

Personalised recommendations