Advertisement

Improving Bilingual Search Performance Using Compact Full-Text Indices

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9041)

Abstract

Machine Translation tasks must tackle the ever-increasing sizes of parallel corpora, requiring space and time efficient solutions to support them. Several approaches were developed based on full-text indices, such as suffix arrays, with important time and space achievements. However, for supporting bilingual tasks, the search time efficiency of such indices can be improved using an extra layer for the text alignment. Additionally, their space requirements can be significantly reduced using more compact indices. We propose a search procedure on top of a compact bilingual framework that improves bilingual search response time, while having a space efficient representation of aligned parallel corpora.

Keywords

Machine Translation Translation Model Statistical Machine Translation Computational Linguistics Parallel Corpus 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bourdaillet, J., Huet, S., Langlais, P., Lapalme, G.: Transsearch: from a bilingual concordancer to a translation finder. Machine Translation 24(3-4), 241–271 (2010)CrossRefGoogle Scholar
  2. 2.
    Brisaboa, N.R., Fariña, A., Navarro, G., Places, Á.S., Rodríguez, E.: Self-indexing natural language. In: Amir, A., Turpin, A., Moffat, A. (eds.) SPIRE 2008. LNCS, vol. 5280, pp. 121–132. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  3. 3.
    Brisaboa, N.R., Fariña, A., Ladra, S., Navarro, G.: Reorganizing compressed text. In: Proceedings of the 31th Annual International ACM SIGIR Conference, pp. 139–146 (2008)Google Scholar
  4. 4.
    Callison-Burch, C., Bannard, C., Schroeder, J.: A compact data structure for searchable translation memories. In: European Association for Machine Translation (2005)Google Scholar
  5. 5.
    Callison-Burch, C., Bannard, C., Schroeder, J.: Scaling phrase-based statistical machine translation to larger corpora and longer phrases. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 255–262 (2005)Google Scholar
  6. 6.
    Chiang, D.: Hierarchical phrase-based translation. Computational Linguistics 33(2) (2007)Google Scholar
  7. 7.
    Ferragina, P., Fischer, J.: Suffix arrays on words. In: Ma, B., Zhang, K. (eds.) CPM 2007. LNCS, vol. 4580, pp. 328–339. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  8. 8.
    Gomes, L., Aires, J., Lopes, G.P.: Parallel Texts Alignment. In: New Trends in Artificial Intelligence: 14th Portuguese Conference on Artificial Intelligence, pp. 513–524 (2009)Google Scholar
  9. 9.
    Grossi, R., Gupta, A., Vitter, J.S.: High-order entropy-compressed text indexes. In: Proceedings of the 14th ACM-SIAM Symposium on Discrete Algorithms, pp. 841–850 (2003)Google Scholar
  10. 10.
    Koehn, P.: Europarl: A parallel corpus for statistical machine translation. In: Machine Translation Summit X, vol. 5 (2005)Google Scholar
  11. 11.
    Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 48–54 (2003)Google Scholar
  12. 12.
    Lopez, A.: Hierarchical phrase-based translation with suffix arrays. In: Proceedings of the Joint Meeting EMNLP-CoNLL, pp. 976–985 (2007)Google Scholar
  13. 13.
    Lopez, A.: Tera-scale translation models via pattern matching. In: Proceedings of the 22nd International Conference on Computational Linguistics, vol. 1, pp. 505–512 (2008)Google Scholar
  14. 14.
    Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22(5), 935–948 (1993)CrossRefzbMATHMathSciNetGoogle Scholar
  15. 15.
    Navarro, G., Mäkinen, V.: Compressed full-text indexes. ACM Computing Surveys 39(1) (2007)Google Scholar
  16. 16.
    Ribeiro, A., Dias, G., Lopes, G., Mexia, J.: Cognates alignment. In: Machine Translation Summit VIII, Machine Translation in The Information Age, pp. 287–292 (2001)Google Scholar
  17. 17.
    Tiedemann, J.: News from opus-a collection of multilingual parallel corpora with tools and interfaces. In: Recent Advances in Natural Language Processing, vol. 5, pp. 237–248 (2009)Google Scholar
  18. 18.
    Yamamoto, M., Church, K.W.: Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics 27(1), 1–30 (2001)CrossRefGoogle Scholar
  19. 19.
    Zhang, Y., Vogel, S.: An efficient phrase-to-phrase alignment model for arbitrarily long phrase and large corpora. In: Proceedings of the 10th European Association for Machine Translation Conference, pp. 294–301 (2005)Google Scholar
  20. 20.
    Zobel, J., Moffat, A.: Inverted files for text search engines. ACM Computing Surveys 38(2),  6 (2006)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Faculdade de Ciências e TecnologiaUniversidade Nova de LisboaLisboaPortugal
  2. 2.INESC-ID / Instituto Superior TécnicoUniversidade de LisboaLisboaPortugal

Personalised recommendations

Cite paper

Buy options