Using SRX Standard for Sentence Segmentation

  • Marcin Miłkowski
  • Jarosław Lipski
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6562)

Abstract

In this paper, we evaluate using the SRX (Segmentation Rules eXchange) standard for specifying sentence segmentation rules. The rules were originally created for a proofreading tool called LanguageTool. As proofreading tools are quite sensitive to segmentation errors, the underlying segmentation mechanisms must be sufficiently reliable. Even though SRX allows only regular expressions as a means for specifying sentence breaks and exceptions to those breaks, our evaluation shows that it is sufficient for the task, both in terms of the performance of the algorithm used and correctness of results. Moreover, it offers interoperability with different tools, which in turn allows maintaining consistent segmentation in a whole language-processing pipeline. Our rules are available on an open-source license (LGPL), which also helped in receiving valuable community input from our users.

Keywords

segmentation standards computer-aided translation grammar-checking 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Gale, W.A., Church, K.W.: A Program for Aligning Sentences in Bilingual Corpora. Computational Linguistics 19(1), 75–102 (1993)Google Scholar
  2. 2.
    Mikheev, A.: Text segmentation. In: The Oxford Handbook of Computational Linguistics, ch. 10, pp. 201–218. Oxford University Press, Oxford (2003)Google Scholar
  3. 3.
    Palmer, D.D., Hearst, M.A.: Adaptive multilingual sentence boundary disambiguation. Computational Linguistics 23(2), 241–269 (1997)Google Scholar
  4. 4.
    Schmid, H.: Unsupervised learning of period disambiguation for tokenization. Internal Report, University of Stuttgart (2000)Google Scholar
  5. 5.
    Przepiórkowski, A.: The IPI PAN Corpus: Preliminary version. IPI PAN, Warszawa (2004)Google Scholar
  6. 6.
    Miłkowski, M.: Developing an open-source, rule-based proofreading tool. Software – Practice and Experience 40(7), 543–566 (2010)Google Scholar
  7. 7.
  8. 8.
    TMX 1.4b Standard Specification, http://www.lisa.org/tmx/tmx.htm
  9. 9.
    RFC4646: Tags for Identifying Languages, http://www.ietf.org/rfc/rfc4646.txt
  10. 10.
    Rudolf, M.: Metody automatycznej analizy korpusu tekstów polskich. Uniwersytet Warszawski, Warszawa (2004)Google Scholar
  11. 11.
    Mazur, P.P.: Text segmentation in Polish. In: Proceedings of the 5th International Conference on Intelligent Systems Design and Applications, pp. 43–48. IEEE Computer Society, Washington DC (2005)Google Scholar
  12. 12.
    Kim, J., Ohta, T., Teteisi, Y., Tsujii, J.: GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics 19, suppl.1, i180– i182 (2003)CrossRefGoogle Scholar
  13. 13.
    Nazarczuk, M.: Wstępne przygotowanie korpusu Słownika frekwencyjnego polszczyzny współczesnej do dystrybucji na CD-ROM. Praca magisterska napisana pod kierunkiem dra hab. Janusza S. Bienia. Warszawa (1997)Google Scholar
  14. 14.
    Przepiórkowski, A., Górski, R.G., Lewandowska-Tomaszczyk, B., Łaziński, M.: Towards the National Corpus of Polish. In: Proceedings of the 6th Language Resources and Evaluation Conference, Marrakesh (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Marcin Miłkowski
    • 1
  • Jarosław Lipski
    • 2
  1. 1.Institute of Philosophy and SociologyPolish Academy of SciencesWarszawaPoland
  2. 2.Institute of Philosophy and SociologyPolish Academy of SciencesLiverpoolUK

Personalised recommendations