Skip to main content
View expanded cover

International Conference on Database Systems for Advanced Applications

DASFAA 2022: Database Systems for Advanced Applications pp 197–212Cite as

FalCon: A Faithful Contrastive Framework for Response Generation in TableQA Systems

  • 435 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13247)

Abstract

In a practical TableQA system, response generation is a critical module to generate a natural language description of the SQL and the execution result. Due to the complex syntax of SQL and matching issues with table content, this task is prone to produce factual errors. In this paper, we propose FalCon, a Faithful Contrastive generation framework to improve the factual correctness of generated responses. FalCon forces the generation model to identify examples with factual errors in the latent space during training and takes contrastive examples into consideration during inference. We also propose two new automatic metrics to further evaluate faithfulness specialized to this task. Experimental results show FalCon brings a favorable performance improvement on both automatic and human evaluation amongst various baseline methods (The code of FalCon is released at https://github.com/whuFSN/FalCon).

Keywords

  • Response generation
  • Factual correctness
  • Contrastive learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-00129-1_13
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-031-00129-1
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    Synchronously modify headers and values in the result table.

  2. 2.

    We only use the first three rules for constructing source imposters when inference.

  3. 3.

    We report the average best performance observed in 3 runs on the development set of CoSQL since its test set are not public. All improvements of FalCon are significant with \(p<0.01\) compared to backbone models.

References

  1. Burges, C.J.C., et al.: Learning to rank using gradient descent. In: Proceedings of ICML. ACM International Conference Proceeding Series (2005)

    Google Scholar 

  2. Cai, D., Lam, W.: Graph transformer for graph-to-sequence learning. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020 (2020)

    Google Scholar 

  3. Cai, R., Liang, Z., Xu, B., Li, Z., Hao, Y., Chen, Y.: TAG : type auxiliary guiding for code comment generation. In: Proceedings of ACL (2020)

    Google Scholar 

  4. Cao, R., Chen, L., Chen, Z., Zhao, Y., Zhu, S., Yu, K.: LGESQL: line graph enhanced text-to-SQL model with mixed local and non-local relations. In: Proceedings of ACL (2021)

    Google Scholar 

  5. Cao, Z., Wei, F., Li, W., Li, S.: Faithful to the original: fact aware neural abstractive summarization. In: Proceedings of AAAI (2018)

    Google Scholar 

  6. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: Proceedings of ICML. Proceedings of Machine Learning Research (2020)

    Google Scholar 

  7. Chen, W., Chen, J., Su, Y., Chen, Z., Wang, W.Y.: Logical natural language generation from open-domain tables. In: Proceedings of ACL (2020)

    Google Scholar 

  8. Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)

    CrossRef  Google Scholar 

  9. Falke, T., Ribeiro, L.F.R., Utama, P.A., Dagan, I., Gurevych, I.: Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In: Proceedings of ACL (2019)

    Google Scholar 

  10. Gabriel, S., Celikyilmaz, A., Jha, R., Choi, Y., Gao, J.: GO FIGURE: a meta evaluation of factuality in summarization. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (2021)

    Google Scholar 

  11. Goodrich, B., Rao, V., Liu, P.J., Saleh, M.: Assessing the factual accuracy of generated text. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2019, 4–8 August 2019, Anchorage (2019)

    Google Scholar 

  12. Guo, J., et al.: Towards complex text-to-SQL in cross-domain database with intermediate representation. In: Proceedings of ACL (2019)

    Google Scholar 

  13. Hu, X., Li, G., Xia, X., Lo, D., Jin, Z.: Deep code comment generation. In: 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). IEEE (2018)

    Google Scholar 

  14. Iyer, S., Konstas, I., Cheung, A., Zettlemoyer, L.: Summarizing source code using a neural attention model. In: Proceedings of ACL (2016)

    Google Scholar 

  15. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of ICLR (2015)

    Google Scholar 

  16. Koncel-Kedziorski, R., Bekal, D., Luan, Y., Lapata, M., Hajishirzi, H.: Text generation from knowledge graphs with graph transformers. In: Proceedings of NAACL-HLT (2019)

    Google Scholar 

  17. Kryscinski, W., Keskar, N.S., McCann, B., Xiong, C., Socher, R.: Neural text summarization: a critical evaluation. In: Proceedings of EMNLP (2019)

    Google Scholar 

  18. Lee, S., Lee, D.B., Hwang, S.J.: Contrastive learning with adversarial perturbations for conditional text generation. In: Proceedings of ICLR (2021)

    Google Scholar 

  19. Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of ACL (2020)

    Google Scholar 

  20. Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out (2004)

    Google Scholar 

  21. Lin, X.V., Socher, R., Xiong, C.: Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In: Findings of the Association for Computational Linguistics: EMNLP 2020 (2020)

    Google Scholar 

  22. Liu, Y., et al.: RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint (2019)

    Google Scholar 

  23. Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: Proceedings of ICLR (2018)

    Google Scholar 

  24. Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint (2018)

    Google Scholar 

  25. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of ACL (2002)

    Google Scholar 

  26. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint (2019)

    Google Scholar 

  27. Scholak, T., Schucher, N., Bahdanau, D.: PICARD: parsing incrementally for constrained auto-regressive decoding from language models. arXiv preprint (2021)

    Google Scholar 

  28. Sellam, T., Das, D., Parikh, A.: BLEURT: learning robust metrics for text generation. In: Proceedings of ACL (2020)

    Google Scholar 

  29. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4–9 December 2017, Long Beach (2017)

    Google Scholar 

  30. Verma, V., Luong, T., Kawaguchi, K., Pham, H., Le, Q.V.: Towards domain-agnostic contrastive learning. In: Proceedings of ICML. Proceedings of Machine Learning Research (2021)

    Google Scholar 

  31. Wang, B., Shin, R., Liu, X., Polozov, O., Richardson, M.: RAT-SQL: relation-aware schema encoding and linking for text-to-SQL parsers. In: Proceedings of ACL (2020)

    Google Scholar 

  32. Wang, Z., Wang, X., An, B., Yu, D., Chen, C.: Towards faithful neural table-to-text generation with content-matching constraints. In: Proceedings of ACL (2020)

    Google Scholar 

  33. Xu, K., Wu, L., Wang, Z., Feng, Y., Sheinin, V.: SQL-to-text generation with graph-to-sequence model. In: Proceedings of EMNLP (2018)

    Google Scholar 

  34. Yang, Z., Cheng, Y., Liu, Y., Sun, M.: Reducing word omission errors in neural machine translation: a contrastive learning approach. In: Proceedings of ACL (2019)

    Google Scholar 

  35. Yu, T., et al.: CoSQL: a conversational text-to-SQL challenge towards cross-domain natural language interfaces to databases. In: Proceedings of EMNLP (2019)

    Google Scholar 

  36. Yu, T., et al.: Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In: Proceedings of EMNLP (2018)

    Google Scholar 

  37. Zeng, J., et al.: Photon: a robust cross-domain text-to-SQL system. In: Proceedings of ACL (2020)

    Google Scholar 

  38. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: BERTScore: evaluating text generation with BERT. In: Proceedings of ICLR (2020)

    Google Scholar 

  39. Zhang, Y., Merck, D., Tsai, E., Manning, C.D., Langlotz, C.: Optimizing the factual correctness of a summary: a study of summarizing radiology reports. In: Proceedings of ACL (2020)

    Google Scholar 

  40. Zhong, V., Xiong, C., Socher, R.: Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint (2017)

    Google Scholar 

  41. Zhong, W., et al.: LogicalFactChecker: leveraging logical operations for fact checking with graph module network. In: Proceedings of ACL (2020)

    Google Scholar 

Download references

Acknowledgements

We thank anonymous reviewers for their comments and suggestions. This work was supported by National Key Research and Development Project (No. 2020AAA0109302), Shanghai Science and Technology Innovation Action Plan (No. 19511120400) and Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZX0103).

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai, China

    Shineng Fang, Jiangjie Chen, Xinyao Shen & Yanghua Xiao

  2. Fudan-Aishu Cognitive Intelligence Joint Research Center, Shanghai, China

    Yanghua Xiao

  3. DataGrand Inc., Shanghai, China

    Yunwen Chen

Authors
  1. Shineng Fang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiangjie Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyao Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunwen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanghua Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanghua Xiao .

Editor information

Editors and Affiliations

  1. Indian Institute of Technology Kanpur, Kanpur, India

    Dr. Arnab Bhattacharya

  2. National University of Singapore, Singapore, Singapore

    Janice Lee Mong Li

  3. University of California, Santa Barbara, Santa Barbara, CA, USA

    Dr. Divyakant Agrawal

  4. IIIT Hyderabad, Hyderabad, India

    P. Krishna Reddy

  5. Indraprastha Institute of Information Technology Delhi, New Delhi, India

    Mukesh Mohania

  6. Ashoka University, Sonepat, Haryana, India

    Anirban Mondal

  7. Indraprastha Institute of Information Technology Delhi, New Delhi, India

    Vikram Goyal

  8. University of Aizu, Aizu, Japan

    Rage Uday Kiran

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Fang, S., Chen, J., Shen, X., Chen, Y., Xiao, Y. (2022). FalCon: A Faithful Contrastive Framework for Response Generation in TableQA Systems. In: , et al. Database Systems for Advanced Applications. DASFAA 2022. Lecture Notes in Computer Science, vol 13247. Springer, Cham. https://doi.org/10.1007/978-3-031-00129-1_13

Download citation