Abstract
Deep learning transformers have drastically improved systems that automatically answer questions in natural language. However, different questions demand different answering techniques; here we propose, build and validate an architecture that integrates different modules to answer two distinct kinds of queries. Our architecture takes a free-form natural language text and classifies it to send it either to a Neural Question Answering Reasoner or a Natural Language parser to SQL. We implemented a complete system for the Portuguese language, using some of the main tools available for the language and translating training and testing datasets. Experiments show that our system selects the appropriate answering method with high accuracy (over 99%), thus validating a modular question answering strategy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Relation-Aware Transformer SQL Generation-Augmented Pretraining.
- 3.
The instructions to download can be found in the project github: https://github.com/ibm-aur-nlp/domain-specific-QA.
- 4.
Spider dataset is a popular resource that contains 200 databases with multiples tables under 138 domains: https://yale-lily.github.io/spider.
- 5.
hospital_1 100 questions (test), protein_institute 20 questions (train), medicine_enzyme_interaction 44 questions (train), scientist_1 48 questions (train).
- 6.
Text-to-SQL Generation for Question Answering on Electronic Medical Records Github: https://github.com/wangpinggl/TREQS.
- 7.
We used well-known implementations of naive Bayes [22] and transformers. In particular, the tranformers HuggingFace library, at https://huggingface.co/transformers/, and also simpletransformers at https://simpletransformers.ai/docs/installation/.
- 8.
Trained using 5 epochs, learning rate of 5e-5, batch size of 32 and maximum sequence length of 512.
- 9.
Trained using 25 epochs, learning rate of 2e-5, batch size of 32 and maximum sequence length of 512.
- 10.
This is the standard F1-score for classification, not the Macro Average F1-Score.
References
Bender, E.M., Gebru, T., McMillan-Major, A., Shmitchell, S.: On the dangers of stochastic parrots: can language models be too big? In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3442188.3445922
Cação, F.N., José, M.M., Oliveira, A.S., Spindola, S., Costa, A.H.R., Cozman, F.G.: Deepagé: answering questions in Portuguese about the Brazilian environment. In: Britto, A., Valdivia, D.K. (eds.) Intelligent Systems. BRACIS 2021. Lecture Notes in Computer Science, vol. 13074. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-91699-2_29
Carmo, D., Piau, M., Campiotti, I., Nogueira, R., de Alencar Lotufo, R.: PTT5: pretraining and validating the T5 model on brazilian portuguese data. CoRR abs/2008.09144 (2020). https://arxiv.org/abs/2008.09144
Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to answer open-domain questions. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879. Association for Computational Linguistics, Vancouver, Canada (2017). https://doi.org/10.18653/v1/P17-1171
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2–7, 2019, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1423
Ferrucci, D., et al.: Building watson: an overview of the deepQA project. AI Mag. 31(3), 59–79 (2010). https://doi.org/10.1609/aimag.v31i3.2303
José, M.A., Cozman, F.G.: mRAT-SQL\(+\)GAP: a Portuguese text-to-SQL transformer. In: Britto, A., Valdivia Delgado, K. (eds.) BRACIS 2021. LNCS (LNAI), vol. 13074, pp. 511–525. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-91699-2_35
Lewis, P., et al.: Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Advances in Neural Information Processing Systems. vol. 33, pp. 9459–9474. Curran Associates, Inc. (2020). https://doi.org/10.48550/arXiv.2005.11401
Lewis, P., et al.: Paq: 65 million probably-asked questions and what you can do with them. Trans. Assoc. Comput. Linguis. 9, 1098–1115 (2021). https://doi.org/10.1162/tacl_a_00415
Li, A.H., Ng, P., Xu, P., Zhu, H., Wang, Z., Xiang, B.: Dual reader-parser on hybrid textual and tabular evidence for open domain question answering. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1–6, 2021, pp. 4078–4088. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.acl-long.315
McCallum, A., Penn, G., Munteanu, C., Zhu, X.: Ecological validity and the evaluation of speech summarization quality. In: Proceedings of the 2012 IEEE Workshop on Spoken Language Technology, SLT 2012, pp. 467–472 (2012). https://doi.org/10.1109/SLT.2012.6424269
Nguyen, T., et al.: MS MARCO: a human generated Machine reading Comprehension dataset. In: CEUR Workshop Proceedings, vol. 1773, pp. 1–10 (2016)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ questions for machine comprehension of text. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383–2392. Association for Computational Linguistics, Austin, Texas (2016). https://doi.org/10.18653/v1/D16-1264
Robertson, S., Zaragoza, H.: The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3(4), 333–389 (2009). https://doi.org/10.1561/1500000019
Shi, P., et al.: Learning contextual representations for semantic parsing with generation-augmented pre-training. Proc. AAAI Conf. Artif. Intell. 35(15), 13806–13814 (2021). https://doi.org/10.48550/arXiv.2012.10309
Souza, F., Nogueira, R., Lotufo, R.: Bertimbau: pretrained BERT models for Brazilian Portuguese. In: Cerri, R., Prati, R.C. (eds.) Intelligent Systems. BRACIS 2020. Lecture Notes in Computer Science, vol. 12319. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000–6010. Curran Associates Inc., Red Hook, NY, USA (2017)
Wang, P., Shi, T., Reddy, C.K.: Text-to-SQL generation for question answering on electronic medical records. In: WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20–24, 2020, pp. 350–361. ACM/IW3C2 (2020). https://doi.org/10.1145/3366423.3380120
Xu, Y., Zhong, X., Yepes, A.J.J., Lau, J.H.: Forget me not: reducing catastrophic forgetting for domain adaptation in reading comprehension. In: 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2020). https://doi.org/10.1109/IJCNN48605.2020.9206891
Xue, L., et al.: mT5: a massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483–498. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.naacl-main.41
Yu, T., et al.: Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911–3921. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/v1/D18-1425
Zhang, H.: The optimality of Naive Bayes. In: Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2004, vol. 2, pp. 562–567 (2004)
Zhang, Z., Yang, J., Zhao, H.: Retrospective reader for machine reading comprehension. Proc. AAAI Conf. Artif. Intell. 35(16), 14506–14514 (2021). https://doi.org/10.48550/arXiv.2001.09694
Acknowledgment
This work was partly supported by Itaú Unibanco S.A. through the Programa de Bolsas Itaú (PBI) of the Centro de Ciência de Dados da Universidade de São Paulo (C\(^2\)D-USP); by the Center for Artificial Intelligence (C4AI) through support from the São Paulo Research Foundation (FAPESP grant #2019/07665-4) and from the IBM Corporation; by CNPq grants no. 312180/2018-7 and 304012/2019-0, and CAPES Finance Code 001.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
José, M.M., José, M.A., Mauá, D.D., Cozman, F.G. (2022). Integrating Question Answering and Text-to-SQL in Portuguese. In: Pinheiro, V., et al. Computational Processing of the Portuguese Language. PROPOR 2022. Lecture Notes in Computer Science(), vol 13208. Springer, Cham. https://doi.org/10.1007/978-3-030-98305-5_26
Download citation
DOI: https://doi.org/10.1007/978-3-030-98305-5_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98304-8
Online ISBN: 978-3-030-98305-5
eBook Packages: Computer ScienceComputer Science (R0)