Advertisement

WiSeBE: Window-Based Sentence Boundary Evaluation

  • Carlos-Emiliano González-Gallardo
  • Juan-Manuel Torres-Moreno
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11289)

Abstract

Sentence Boundary Detection (SBD) has been a major research topic since Automatic Speech Recognition transcripts have been used for further Natural Language Processing tasks like Part of Speech Tagging, Question Answering or Automatic Summarization. But what about evaluation? Do standard evaluation metrics like precision, recall, F-score or classification error; and more important, evaluating an automatic system against a unique reference is enough to conclude how well a SBD system is performing given the final application of the transcript? In this paper we propose Window-based Sentence Boundary Evaluation (WiSeBE), a semi-supervised metric for evaluating Sentence Boundary Detection systems based on multi-reference (dis)agreement. We evaluate and compare the performance of different SBD systems over a set of Youtube transcripts using WiSeBE and standard metrics. This double evaluation gives an understanding of how WiSeBE is a more reliable metric for the SBD task.

Keywords

Sentence Boundary Detection Evaluation Transcripts Human judgment 

Notes

Acknowledgments

We would like to acknowledge the support of CHIST-ERA for funding this work through the Access Multilingual Information opinionS (AMIS), (France - Europe) project.

We also like to acknowledge the support given by the Prof. Hanifa Boucheneb from VERIFORM Laboratory (École Polytechnique de Montréal).

References

  1. 1.
    Bohac, M., Blavka, K., Kucharova, M., Skodova, S.: Post-processing of the recognized speech for web presentation of large audio archive. In: 2012 35th International Conference on Telecommunications and Signal Processing (TSP), pp. 441–445. IEEE (2012)Google Scholar
  2. 2.
    Brum, H., Araujo, F., Kepler, F.: Sentiment analysis for Brazilian portuguese over a skewed class corpora. In: Silva, J., Ribeiro, R., Quaresma, P., Adami, A., Branco, A. (eds.) PROPOR 2016. LNCS (LNAI), vol. 9727, pp. 134–138. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-41552-9_14CrossRefGoogle Scholar
  3. 3.
    Che, X., Wang, C., Yang, H., Meinel, C.: Punctuation prediction for unsegmented transcript based on word vector. In: LREC (2016)Google Scholar
  4. 4.
    Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378 (1971)CrossRefGoogle Scholar
  5. 5.
    Fohr, D., Mella, O., Illina, I.: New paradigm in speech recognition: deep neural networks. In: IEEE International Conference on Information Systems and Economic Intelligence (2017)Google Scholar
  6. 6.
    González-Gallardo, C.E., Hajjem, M., SanJuan, E., Torres-Moreno, J.M.: Transcripts informativeness study: an approach based on automatic summarization. In: Conférence en Recherche d’Information et Applications (CORIA), Rennes, France, May (2018)Google Scholar
  7. 7.
    González-Gallardo, C.E., Torres-Moreno, J.M.: Sentence boundary detection for French with subword-level information vectors and convolutional neural networks. arXiv preprint arXiv:1802.04559 (2018)
  8. 8.
    Gotoh, Y., Renals, S.: Sentence boundary detection in broadcast speech transcripts. In: ASR2000-Automatic Speech Recognition: Challenges for the new Millenium ISCA Tutorial and Research Workshop (ITRW) (2000)Google Scholar
  9. 9.
    Hinton, G., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82–97 (2012)CrossRefGoogle Scholar
  10. 10.
    Jamil, N., Ramli, M.I., Seman, N.: Sentence boundary detection without speech recognition: a case of an under-resourced language. J. Electr. Syst. 11(3), 308–318 (2015)Google Scholar
  11. 11.
    Kiss, T., Strunk, J.: Unsupervised multilingual sentence boundary detection. Comput. Linguist. 32(4), 485–525 (2006)CrossRefGoogle Scholar
  12. 12.
    Klejch, O., Bell, P., Renals, S.: Punctuated transcription of multi-genre broadcasts using acoustic and lexical approaches. In: 2016 IEEE Spoken Language Technology Workshop (SLT), pp. 433–440. IEEE (2016)Google Scholar
  13. 13.
    Kolář, J., Lamel, L.: Development and evaluation of automatic punctuation for French and english speech-to-text. In: Thirteenth Annual Conference of the International Speech Communication Association (2012)Google Scholar
  14. 14.
    Kolář, J., Švec, J., Psutka, J.: Automatic punctuation annotation in Czech broadcast news speech. In: SPECOM 2004 (2004)Google Scholar
  15. 15.
    Liu, Y., Chawla, N.V., Harper, M.P., Shriberg, E., Stolcke, A.: A study in machine learning from imbalanced data for sentence boundary detection in speech. Comput. Speech Lang. 20(4), 468–494 (2006)CrossRefGoogle Scholar
  16. 16.
    Lu, W., Ng, H.T.: Better punctuation prediction with dynamic conditional random fields. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. pp. 177–186. Association for Computational Linguistics (2010)Google Scholar
  17. 17.
    Meteer, M., Iyer, R.: Modeling conversational speech for speech recognition. In: Conference on Empirical Methods in Natural Language Processing (1996)Google Scholar
  18. 18.
    Mrozinski, J., Whittaker, E.W., Chatain, P., Furui, S.: Automatic sentence segmentation of speech for automatic summarization. In: 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, vol. 1, p. I. IEEE (2006)Google Scholar
  19. 19.
    Palmer, D.D., Hearst, M.A.: Adaptive sentence boundary disambiguation. In: Proceedings of the Fourth Conference on Applied Natural Language Processing, pp. 78–83. ANLC 1994. Association for Computational Linguistics, Stroudsburg, PA, USA (1994)Google Scholar
  20. 20.
    Palmer, D.D., Hearst, M.A.: Adaptive multilingual sentence boundary disambiguation. Comput. Linguist. 23(2), 241–267 (1997)Google Scholar
  21. 21.
    Pearson, K.: Note on regression and inheritance in the case of two parents. Proc. R. Soc. Lond. 58, 240–242 (1895)CrossRefGoogle Scholar
  22. 22.
    Peitz, S., Freitag, M., Ney, H.: Better punctuation prediction with hierarchical phrase-based translation. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), South Lake Tahoe, CA, USA (2014)Google Scholar
  23. 23.
    Rott, M., Červa, P.: Speech-to-text summarization using automatic phrase extraction from recognized text. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2016. LNCS (LNAI), vol. 9924, pp. 101–108. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-45510-5_12CrossRefGoogle Scholar
  24. 24.
    Shriberg, E., Stolcke, A.: Word predictability after hesitations: a corpus-based study. In: Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 1996, vol. 3, pp. 1868–1871. IEEE (1996)Google Scholar
  25. 25.
    Stevenson, M., Gaizauskas, R.: Experiments on sentence boundary detection. In: Proceedings of the sixth conference on Applied natural language processing, pp. 84–89. Association for Computational Linguistics (2000)Google Scholar
  26. 26.
    Stolcke, A., Shriberg, E.: Automatic linguistic segmentation of conversational speech. In: Proceedings of the Fourth International Conference on Spoken Language, 1996. ICSLP 1996, vol. 2, pp. 1005–1008. IEEE (1996)Google Scholar
  27. 27.
    Strassel, S.: Simple metadata annotation specification v5. 0, linguistic data consortium (2003). http://www.ldc.upenn.edu/projects/MDE/Guidelines/SimpleMDE_V5.0.pdf
  28. 28.
    Tilk, O., Alumäe, T.: Bidirectional recurrent neural network with attention mechanism for punctuation restoration. In: Interspeech 2016 (2016)Google Scholar
  29. 29.
    Treviso, M.V., Shulby, C.D., Aluisio, S.M.: Evaluating word embeddings for sentence boundary detection in speech transcripts. arXiv preprint arXiv:1708.04704 (2017)
  30. 30.
    Ueffing, N., Bisani, M., Vozila, P.: Improved models for automatic punctuation prediction for spoken and written text. In: Interspeech, pp. 3097–3101 (2013)Google Scholar
  31. 31.
    Wang, W., Tur, G., Zheng, J., Ayan, N.F.: Automatic disfluency removal for improving spoken language translation. In: 2010 IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 5214–5217. IEEE (2010)Google Scholar
  32. 32.
    Xu, C., Xie, L., Huang, G., Xiao, X., Chng, E.S., Li, H.: A deep neural network approach for sentence boundary detection in broadcast news. In: Fifteenth Annual Conference of the International Speech Communication Association (2014)Google Scholar
  33. 33.
    Yu, D., Deng, L.: Automatic Speech Recognition. Springer, London (2015).  https://doi.org/10.1007/978-1-4471-5779-3CrossRefzbMATHGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Carlos-Emiliano González-Gallardo
    • 1
    • 2
  • Juan-Manuel Torres-Moreno
    • 1
    • 2
  1. 1.LIA - Université d’Avignon et des Pays de VaucluseAvignonFrance
  2. 2.Département de GIGL, École Polytechnique de MontréalMontréalCanada

Personalised recommendations