Compact and Fast Indexes for Translation Related Tasks

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8154)


Translation tasks, including bilingual concordancing, demand an efficient space/time trade-off, which is not always easy to get due to the usage of huge text collections and the space consuming nature of time efficient text indexes. We propose a compact representation for monotonically aligned parallel texts, based on known compressed text indexes for representing the texts and additional uncompressed structures for the alignment. The proposed framework is able to index a collection of texts in main memory, occupying less space than the text size and with efficient query response time. The proposal supports any type of alignment granularity, a novelty in concordancing applications, allowing a flexible environment for linguistics working in all phases of a translation process. We present two alternatives for self-indexing the texts, and two alternatives for supporting the alignment, comparing the alternatives in terms of space/time performance.


Text compression Machine Translation bilingual concordancer parallel text alignment alignment granularity 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aires, J., Lopes, G., Gomes, L.: Phrase translation extraction from aligned parallel corpora using suffix arrays and related structures. In: Lopes, L.S., Lau, N., Mariano, P., Rocha, L.M. (eds.) EPIA 2009. LNCS, vol. 5816, pp. 587–597. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  2. 2.
    Apostolico, A.: The myriad virtues of subword trees. In: Combinatorial Algorithms on Words. NATO ISO Series, pp. 85–96. Springer (1985)Google Scholar
  3. 3.
    Bilbao, V.D., Lopes, J.P., Ildefonso, T.: Measuring the impact of cognates in parallel text alignment. In: Portuguese Conference on Artificial Intelligence, EPIA 2005, pp. 338–343. IEEE (2005)Google Scholar
  4. 4.
    Bourdaillet, J., Huet, S., Langlais, P., Lapalme, G.: Transsearch: from a bilingual concordancer to a translation finder. Machine Translation 24(3), 241–271 (2010)CrossRefGoogle Scholar
  5. 5.
    Brisaboa, N.R., Fariña, A., Navarro, G., Places, Á.S., Rodríguez, E.: Self-indexing natural language. In: Amir, A., Turpin, A., Moffat, A. (eds.) SPIRE 2008. LNCS, vol. 5280, pp. 121–132. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Brisaboa, N., Fariña, A., Ladra, S., Navarro, G.: Reorganizing compressed text. In: Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 139–146. ACM (2008)Google Scholar
  7. 7.
    Callison-Burch, C., Bannard, C.: A compact data structure for searchable translation memories. In: EAMT 2005 (2005)Google Scholar
  8. 8.
    Gomes, L., Aires, J., Lopes, G.: Parallel texts alignment. In: Proceedings of the New Trends in Artificial Intelligence, 14th Portuguese Conference in Artificial Intelligence, EPIA 2009, Aveiro, pp. 513–524. Universidade de Aveiro (October 2009)Google Scholar
  9. 9.
    González, R., Grabowski, S., Mäkinen, V., Navarro, G.: Practical implementation of rank and select queries. In: Poster Proc. Volume of 4th Workshop on Efficient and Experimental Algorithms, pp. 27–38. CTI Press and Ellinika Grammata (2005)Google Scholar
  10. 10.
    Grossi, R., Gupta, A., Vitter, J.: High-order entropy-compressed text indexes. In: Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 841–850. Society for Industrial and Applied Mathematics (2003)Google Scholar
  11. 11.
    Kilgarriff, A., Rychly, P., Smrz, P., Tugwell, D.: Itri-04-08 the sketch engine. Information Technology 105, 116 (2004)Google Scholar
  12. 12.
    Koehn, P.: Europarl: A parallel corpus for statistical machine translation. In: MT summit. 5 (2005)Google Scholar
  13. 13.
    Kremer, G., Hartung, M., Padó, S., Riezler, S.: Statistical machine translation support improves human adjective translation. Translation: Computation, Corpora, Cognition 2(1) (2012)Google Scholar
  14. 14.
    Lopez, A.: Hierarchical phrase-based translation with suffix arrays. In: Proc. of EMNLP-CoNLL, pp. 976–985 (2007)Google Scholar
  15. 15.
    Macklovitch, E., Simard, M., Langlais, P.: Transsearch: A free translation memory on the world wide web. In: Second International Conference On Language Resources and Evaluation, LREC, vol. 3, pp. 1201–1208 (2000)Google Scholar
  16. 16.
    Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22(5), 935–948 (1993)CrossRefzbMATHMathSciNetGoogle Scholar
  17. 17.
    Puglisi, S.J., Smyth, W.F., Turpin, A.: Inverted files versus suffix arrays for locating patterns in primary memory. In: Crestani, F., Ferragina, P., Sanderson, M. (eds.) SPIRE 2006. LNCS, vol. 4209, pp. 122–133. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Sadakane, K.: New text indexing functionalities of the compressed suffix arrays. Journal of Algorithms 48(2), 294–313 (2003)CrossRefzbMATHMathSciNetGoogle Scholar
  19. 19.
    Tiedemann, J.: News from opus-a collection of multilingual parallel corpora with tools and interfaces. Recent Advances in Natural Language Processing 5, 237–248 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Faculdade de Ciências e TecnologiaUniversidade Nova de LisboaCaparicaPortugal
  2. 2.Instituto Superior TécnicoUniversidade Técnica de LisboaLisboaPortugal
  3. 3.Database LabUniversity of A CoruñaA CoruñaSpain

Personalised recommendations