Skip to main content
SpringerLink
Log in

UniSAr: a unified structure-aware autoregressive language model for text-to-SQL semantic parsing

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Existing text-to-SQL semantic parsers are typically designed for particular settings such as handling queries that span multiple tables, domains, or turns which makes them ineffective when applied to different settings. We present UniSAr (Unified Structure-Aware Autoregressive Language Model), which benefits from directly using an off-the-shelf language model architecture and demonstrates consistently high performance under different settings. Specifically, UniSAr extends existing autoregressive language models to incorporate two non-invasive extensions to make them structure-aware: (1) adding structure mark to encode database schema, conversation context, and their relationships; (2) constrained decoding to decode well-structured SQL for a given database schema. On seven well-known text-to-SQL datasets covering multi-domain, multi-table, and multi-turn, UniSAr demonstrates highly comparable or better performance to the most advanced specifically-designed text-to-SQL models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The dataset listed in Table 1 are publicly available. WikiSQL: https://github.com/salesforce/WikiSQL. TableQA: https://github.com/ZhuiyiTechnology/TableQA. Spider: https://yale-lily.github.io/spider. DuSQL: https://github.com/luge-ai/luge-ai/. CoSQL: https://yale-lily.github.io/cosql. SParC: https://yale-lily.github.io/sparc. Chase: https://xjtu-intsoft.github.io/chase/.

Notes

  1. https://lucene.apache.org/.

  2. https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html.

References

  1. Zhong V, Xiong C, Socher R (2017) Seq2sql: generating structured queries from natural language using reinforcement learning. arXiv:1709.00103

  2. Yu T, Zhang R, Yang K, Yasunaga M, Wang D, Li Z, Ma J, Li I, Yao Q, Roman S, Zhang Z, Radev D (2018) Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. EMNLP2018. https://aclanthology.org/D18-1425

  3. Guo J, Zhan Z, Gao Y, Xiao Y, Lou J-G, Liu T, Zhang D (2019) Towards complex text-to-SQL in cross-domain database with intermediate representation. Association for Computational Linguistics, Florence, Italy. https://aclanthology.org/P19-1444. Accessed 2 Aug 2019

  4. Yu T, Zhang R, Yasunaga M, Tan YC, Lin XV, Li S, Er H, Li I, Pang B, Chen T, Ji E, Dixit S, Proctor D, Shim S, Kraft J, Zhang V, Xiong C, Socher R, Radev D (2019) SParC: cross-domain semantic parsing in context. Association for Computational Linguistics, Florence, Italy. https://aclanthology.org/P19-1443. Accessed 2 Aug 2019

  5. Sun N, Yang X, Liu Y (2020) TableQA: a large-scale Chinese text-to-SQL dataset for table-aware SQL generation. arXiv:2006.06434

  6. Wang L, Zhang A, Wu K, Sun K, Li Z, Wu H, Zhang M, Wang H (2020) DuSQL: a large-scale and pragmatic Chinese text-to-SQL dataset. Association for Computational Linguistics, Online. https://aclanthology.org/2020.emnlp-main.562. Accessed 20 Nov 2020

  7. Yu T, Zhang R, Er H, Li S, Xue E, Pang B, Lin XV, Tan YC, Shi T, Li Z, Jiang Y, Yasunaga M, Shim S, Chen T, Fabbri A, Li Z, Chen L, Zhang Y, Dixit S, Zhang V, Xiong C, Socher R, Lasecki W, Radev D (2019) CoSQL: a conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. Association for Computational Linguistics, Hong Kong, China. https://aclanthology.org/D19-1204. Accessed 7 Nov 2019

  8. Guo J, Si Z, Wang Y, Liu Q, Fan M, Lou J-G, Yang Z, Liu T (2021) Chase: a large-scale and pragmatic Chinese dataset for cross-database context-dependent text-to-SQL. Association for Computational Linguistics, Online. https://doi.org/10.18653/v1/2021.acl-long.180. https://aclanthology.org/2021.acl-long.180

  9. Yin P, Neubig G (2018) TRANX: a transition-based neural abstract syntax parser for semantic parsing and code generation. Association for Computational Linguistics, Brussels, Belgium. https://aclanthology.org/D18-2002. Accessed 4 Nov 2018

  10. Wang B, Shin R, Liu X, Polozov O, Richardson M (2020) RAT-SQL: relation-aware schema encoding and linking for text-to-SQL parsers. Association for Computational Linguistics, Online. https://doi.org/10.18653/v1/2020.acl-main.677. https://aclanthology.org/2020.acl-main.677

  11. Bogin B, Gardner M, Berant J (2019) Global reasoning over database structures for text-to-SQL parsing. Association for Computational Linguistics, Hong Kong, China. https://aclanthology.org/D19-1378. Accessed 7 Nov 2019

  12. Cai Y, Wan X (2020) IGSQL: database schema interaction graph based neural model for context-dependent text-to-SQL generation. Association for Computational Linguistics, Online. https://doi.org/10.18653/v1/2020.emnlp-main.560. https://aclanthology.org/2020.emnlp-main.560

  13. Zhang R, Yu T, Er H, Shim S, Xue E, Lin XV, Shi T, Xiong C, Socher R, Radev D (2019) Editing-based SQL query generation for cross-domain context-dependent questions. Association for Computational Linguistics, Hong Kong, China. https://aclanthology.org/D19-1537. Accessed 7 Nov 2019

  14. Hwang W, Yim J, Park S, Seo M (2019) A comprehensive exploration on wikisql with table-aware word contextualization. arXiv:1902.01069

  15. Xu X, Liu C, Song D (2017) Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint. arXiv:1711.04436

  16. Zhang X, Yin F, Ma G, Ge B, Xiao W (2020) F-sql: fuse table schema and table content for single-table text2sql generation. IEEE Access 8:136409–136420. https://doi.org/10.1109/ACCESS.2020.3011747

    Article  Google Scholar 

  17. Shin R, Lin CH, Thomson S, Chen C, Roy S, Platanios EA, Pauls A, Klein D, Eisner J, Durme BV (2021) Constrained language models yield few-shot semantic parsers. CoRR. arXiv:2104.08768

  18. Scholak T, Schucher N, Bahdanau D, Bahdanau D (2021) PICARD: parsing incrementally for constrained auto-regressive decoding from language models. EMNLP2021. https://aclanthology.org/2021.emnlp-main.779

  19. Xie T, Wu CH, Shi P, Zhong R, Scholak T, Yasunaga M, Wu C-S, Zhong M, Yin P, Wang SI, Zhong V, Wang B, Li C, Boyle C, Ni A, Yao Z, Radev D, Xiong C, Kong L, Zhang R, Smith NA, Zettlemoyer L, Yu T (2022) UnifiedSKG: unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint. arXiv:2201.05966

  20. Xuan K, Wang Y, Wang Y, Wen Z, Dong Y (2021) Sead: End-to-end text-to-sql generation with schema-aware denoising. CoRR. arXiv:2105.07911

  21. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L (2020) BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Association for Computational Linguistics, Online. https://aclanthology.org/2020.acl-main.703. Accessed 10 Jul 2020

  22. Liu Y, Wan Y, He L, Peng H, Yu PS (2020) KG-BART: knowledge graph-augmented BART for generative commonsense reasoning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 35, no 7, pp 6418–6425

  23. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(140):1–67

    MathSciNet  Google Scholar 

  24. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Part of Advances in Neural Information Processing Systems 30 (NIPS 2017)

  25. Lei W, Wang W, Ma Z, Gan T, Lu W, Kan M-Y, Chua T-S (2020) Re-examining the role of schema linking in text-to-SQL. Association for Computational Linguistics, Online. https://aclanthology.org/2020.emnlp-main.564. Accessed 20 Nov 2020

  26. Aghajanyan A, Okhonko D, Lewis M, Joshi M, Xu H, Ghosh G, Zettlemoyer L (2022) Htlm: hyper-text pre-training and prompting of language models. ICLR

  27. Chen X, Zhang N, Xie X, Deng S, Yao Y, Tan C, Huang F, Si L, Chen H (2021) Knowprompt: knowledge-aware prompt-tuning with synergistic optimization for relation extraction. arXiv:2104.07650

  28. Liu Q, Yang D, Zhang J, Guo J, Zhou B, Lou J-G (2021) Awakening latent grounding from pretrained language models for semantic parsing. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. Association for Computational Linguistics, pp 1174–1189, Online. https://aclanthology.org/2021.findings-acl.100

  29. Sun Y, Tang D, Duan N, Ji J, Cao G, Feng X, Qin B, Liu T, Zhou M (2018) Semantic parsing with syntax- and table-aware SQL generation. Association for Computational Linguistics, Melbourne, Australia. https://aclanthology.org/P18-1034. Accessed 20 Jul 2018

  30. Zheng Y, Wang H, Dong B, Wang X, Li C (2022) HIE-SQL: history information enhanced network for context-dependent text-to-SQL semantic parsing. ACL2022. arXiv:2203.07376

  31. Cao ND, Izacard G, Riedel S, Petroni F (2021) Autoregressive entity retrieval. ICLR2021. https://openreview.net/forum?id=5k8F6UU39V

  32. Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to algorithms. The MIT Press, Cambridge

  33. Dong L, Lapata M (2018) Coarse-to-fine decoding for neural semantic parsing. Association for Computational Linguistics, Melbourne, Australia. https://aclanthology.org/P18-1068. Accessed 20 Jul 2018

  34. He P, Mao Y, Chakrabarti K, Chen W (2019) X-SQL: reinforce schema representation with context. arXiv:1908.08113

  35. Lyu Q, Chakrabarti K, Hathi S, Kundu S, Zhang J, Chen Z (2020) Hybrid ranking network for text-to-SQL. arXiv:2008.04759

  36. Lin XV, Socher R, Xiong C (2020) Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. Association for Computational Linguistics, Online. https://aclanthology.org/2020.findings-emnlp.438. Accessed 20 Nov 2020

  37. Choi D, Shin M, Kim E, Shin DR (2020) RYANSQL: recursively applying sketch-based slot fillings for complex text-to-SQL in cross-domain databases. CoRR. arXiv:2004.03125

  38. Ott M, Edunov S, Baevski A, Fan A, Gross S, Ng N, Grangier D, Auli M (2019) fairseq: A fast, extensible toolkit for sequence modeling. Association for Computational Linguistics, Minneapolis, Minnesota. https://aclanthology.org/N19-4009. Accessed 7 Jun 2019

  39. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958

    MathSciNet  MATH  Google Scholar 

  40. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. CoRR. arXiv:1412.6980

  41. Rasley J, Rajbhandari S, Ruwase O, He Y (2020) DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3394486.3406703

  42. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M, Davison J, Shleifer S, von Platen P, Ma C, Jernite Y, Plu J, Xu C, Le Scao T, Gugger S, Drame M, Lhoest Q, Rush A (2020) Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Association for Computational Linguistics, pp 38–45, Online. https://aclanthology.org/2020.emnlp-demos.6

  43. Shazeer N, Stern M (2018) Adafactor: adaptive learning rates with sublinear memory cost. PMLR. https://proceedings.mlr.press/v80/shazeer18a.html

  44. Loshchilov I, Hutter F (2019) Decoupled weight decay regularization. ICLR 2019. https://openreview.net/forum?id=Bkg6RiCqY7

  45. Cao R, Chen L, Chen Z, Zhao Y, Zhu S, Yu K (2021) LGESQL: line graph enhanced text-to-SQL model with mixed local and non-local relations. ACL 2021

Download references

Acknowledgements

We thank all anonymous reviewers for their constructive comments. Wanxiang Che was supported via the grant 2020AAA0106501, 62236004 and 61976072.

Author information

Author notes

  1. Longxu Dou: This work was done during the internship of Microsoft Research Asia.

Authors and Affiliations

  1. Harbin Institute of Technology, Harbin, 150001, Heilongjiang, China

    Longxu Dou, Mingyang Pan, Dingzirui Wang, Wanxiang Che & Dechen Zhan

  2. Microsoft Research Asia, Beijing, 100089, China

    Yan Gao & Jian-Guang Lou

Authors
  1. Longxu Dou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yan Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingyang Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dingzirui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wanxiang Che
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jian-Guang Lou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dechen Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dechen Zhan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dou, L., Gao, Y., Pan, M. et al. UniSAr: a unified structure-aware autoregressive language model for text-to-SQL semantic parsing. Int. J. Mach. Learn. & Cyber. 14, 4361–4376 (2023). https://doi.org/10.1007/s13042-023-01898-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-023-01898-3

Keywords

Search

Navigation