Skip to main content

Automatic Text Preprocessing for Intelligent Dialog Agents

  • Conference paper
  • First Online:
Web, Artificial Intelligence and Network Applications (WAINA 2019)

Abstract

The paper describes a new Text Preprocessing Pipeline based on a Hybrid approach which involve rule-based and stochastic approaches. The presented pipeline is part of a larger project titled Big Data for Multi-Agent Specialized System developed by Network Contacts in collaboration with University of Salerno and other institutional partners. The aim of the project is to build an Hybrid Question Answering System composed by sets of Dialog Bots able to process great volumes of data. Due to the importance of unstructured textual data, a particular focus of the project is on automatic processing of Text. The paper will describe the three main modules of the preprocessing pipeline, which involve a Style Correction Module, a Clitic Decomposition Module and a POS Tagging and Lemmatization Module.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Network Contacts is one of the major national player in BPO services (Business Process Outsourcing), CRM (Customer Relationship Management), Digital Interaction and Call & Contact Center.

  2. 2.

    With MWE we make reference both to lemmatized MWE, e.g. carta di credito, which are listed in the Knowledge base, and to a set of MWE which are domain specific, e.g. all inclusive unlimited, for the domain of telecommunications.

References

  1. Amato, F., Colace, F., Greco, L., Moscato, V., Picariello, A.: Semantic processing of multimedia data for e-government applications. J. Vis. Lang. Comput. 32, 35–41 (2016). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84949036166&doi=10.1016%2fj.jvlc.2015.10.012&partnerID=40&md5=77d8fbfd4293af91d382e770553e159c.

  2. Amato, F., Mazzeo, A., Penta, A., Picariello, A.: Knowledge representation and management for e-government documents. In: IFIP International Federation for Information Processing, vol. 280, pp. 31–40 (2008). https://www.scopus.com/inward/record.uri?eid=2-s2.0-47249122655&doi=10.1007%2f978-0-387-09712-1_4&partnerID=40&md5=d8e43aeb328d1fdcb45aa323689f7010.

  3. Amato, F., Moscato, F.: A model driven approach to data privacy verification in e-health systems. Trans. Data Privacy 8(3), 273–296 (2015). https://www.scopus.com/inward/record.uri?eid=2-s2.0-84949775424&partnerID=40&md5=2ca02581d8482fbc2f8489a423a80290.

    Google Scholar 

  4. Attardi, G., Fuschetto, A., Tamberi, F., Simi, M., Vecchi, E.M.: Experiments in tagger combination: arbitrating, guessing, correcting, suggesting. In: Proceedings of Workshop Evalita, p. 10 (2009)

    Google Scholar 

  5. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  6. Choi, J.D.: Dynamic feature induction: the last gist to the state-of-the-art. In: Proceedings of NAACL-HLT, pp. 271–281 (2016)

    Google Scholar 

  7. Chrupała, G., Dinu, G., Van Genabith, J.: Learning morphology with Morfette (2008)

    Google Scholar 

  8. Collins, M.: Discriminative training methods for hidden Markov models: theory and experiments with perceptron algorithms. In: Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, vol. 10, pp. 1–8. Association for Computational Linguistics (2002)

    Google Scholar 

  9. Constant, M., Tellier, I., Duchier, D., Dupont, Y., Sigogne, A., Billot, S.: Intégrer des connaissances linguistiques dans un crf: application à l’apprentissage d’un segmenteur-étiqueteur du français. In: TALN, vol. 1, p. 321 (2011)

    Google Scholar 

  10. Dahlmeier, D., Ng, H.T.: Better evaluation for grammatical error correction. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 568–572. Association for Computational Linguistics (2012)

    Google Scholar 

  11. Dell’Orletta, F.: Ensemble system for part-of-speech tagging. In: Proceedings of EVALITA, vol. 9, pp. 1–8 (2009)

    Google Scholar 

  12. Delmonte, R.: Generating and parsing clitics with getarun. In: CLIN (1999)

    Google Scholar 

  13. Favretti, R.R., Tamburini, F., De Santis, C.: CORIS/CODIS: a corpus of written Italian based on a defined and a dynamic model. A Rainbow of Corpora: Corpus Linguistics and the Languages of the World. Munich: Lincom-Europa (2002)

    Google Scholar 

  14. Felice, M., Yuan, Z., Andersen, Ø.E., Yannakoudakis, H., Kochmar, E.: Grammatical error correction using hybrid systems and type filtering. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 15–24 (2014)

    Google Scholar 

  15. Giménez, J., Marquez, L.: Svmtool: a general POS tagger generator based on support vector machines. In: Proceedings of the 4th International Conference on Language Resources and Evaluation. Citeseer (2004)

    Google Scholar 

  16. Han, N.R., Chodorow, M., Leacock, C.: Detecting errors in English article usage with a maximum entropy classifier trained on a large, diverse corpus. In: LREC (2004)

    Google Scholar 

  17. Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)

  18. Ingason, A.K., Helgadóttir, S., Loftsson, H., Rögnvaldsson, E.: A mixed method lemmatization algorithm using a hierarchy of linguistic identities (HOLI). In: Advances in Natural Language Processing, pp. 205–216. Springer, Heidelberg (2008)

    Google Scholar 

  19. Junczys-Dowmunt, M., Grundkiewicz, R.: The AMU system in the CoNLL-2014 shared task: grammatical error correction by data-intensive and feature-rich statistical machine translation. In: Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pp. 25–33 (2014)

    Google Scholar 

  20. Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Soviet Phys. doklady 10, 707–710 (1966)

    MathSciNet  Google Scholar 

  21. Lyding, V., Stemle, E., Borghetti, C., Brunello, M., Castagnoli, S., Dell’Orletta, F., Dittmann, H., Lenci, A., Pirrelli, V.: The PAISA corpus of Italian web texts. In: Proceedings of the 9th Web as Corpus Workshop (WaC-9), pp. 36–43 (2014)

    Google Scholar 

  22. Manning, C.D.: Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In: International Conference on Intelligent Text Processing and Computational Linguistics, pp. 171–189. Springer, Heidelberg (2011)

    Google Scholar 

  23. Mizumoto, T., Hayashibe, Y., Komachi, M., Nagata, M., Matsumoto, Y.: The effect of learner corpus size in grammatical error correction of ESL writings. In: Proceedings of COLING 2012: Posters, pp. 863–872 (2012)

    Google Scholar 

  24. Monachesi, P.: A grammar of Italian clitics (1996)

    Google Scholar 

  25. Monachesi, P.: A lexical approach to Italian cliticization (1999)

    Google Scholar 

  26. Palmero Aprosio, A., Moretti, G.: Italy goes to Stanford: a collection of CoreNLP modules for Italian. ArXiv e-prints (2016)

    Google Scholar 

  27. Pianta, E., Zanoli, R.: TagPro: a system for Italian PoS tagging based on SVM. Intelligenza Artificiale 4(2), 8–9 (2007)

    Google Scholar 

  28. Porter, M.F.: An algorithm for suffix stripping. Program 14(3), 130–137 (1980)

    Article  Google Scholar 

  29. Rozovskaya, A., Roth, D.: Algorithm selection and model adaptation for ESL correction tasks. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 924–933. Association for Computational Linguistics (2011)

    Google Scholar 

  30. Russi, C.: Italian clitics: an empirical study, vol. 193. Walter de Gruyter (2008)

    Google Scholar 

  31. Schmid, H.: Treetagger—a language independent part-of-speech tagger. Institut für Maschinelle Sprachverarbeitung, Universität Stuttgart 43, 28 (1995)

    Google Scholar 

  32. Shen, L., Satta, G., Joshi, A.: Guided learning for bidirectional sequence classification. In: ACL, vol. 7, pp. 760–767. Citeseer (2007)

    Google Scholar 

  33. Smedt, T.D., Daelemans, W.: Pattern for python. J. Mach. Learn. Res. 13(Jun), 2063–2067 (2012)

    Google Scholar 

  34. Srinivasan, A., Compton, P., Malor, R., Edwards, G., Lazarus, L.: Knowledge acquisition in context for a complex domain. Pre-print of Proceedings of the Fifth EKAW91 (1991)

    Google Scholar 

  35. Sun, C., Jin, X., Lin, L., Zhao, Y., Wang, X.: Convolutional neural networks for correcting English article errors. In: Natural Language Processing and Chinese Computing, pp. 102–110. Springer, Heidelberg (2015)

    Google Scholar 

  36. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)

    Google Scholar 

  37. Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 173–180. Association for Computational Linguistics (2003)

    Google Scholar 

  38. Vietri, S.: Dizionari elettronici e grammatiche a stati finiti: metodi di analisi formale della lingua italiana. Plectica (2008)

    Google Scholar 

  39. Vietri, S.: The Italian module for NooJ. In: Proceedings of the First Italian Conference on Computational Linguistics, CLiC-it, pp. 389–393 (2014)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Alessandro Maisto .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Maisto, A., Pelosi, S., Polito, M., Stingo, M. (2019). Automatic Text Preprocessing for Intelligent Dialog Agents. In: Barolli, L., Takizawa, M., Xhafa, F., Enokido, T. (eds) Web, Artificial Intelligence and Network Applications. WAINA 2019. Advances in Intelligent Systems and Computing, vol 927. Springer, Cham. https://doi.org/10.1007/978-3-030-15035-8_78

Download citation

Publish with us

Policies and ethics