Skip to main content
SpringerLink
Log in

Towards Explaining Shortcut Learning Through Attention Visualization and Adversarial Attacks

  • Conference paper
  • First Online:
Engineering Applications of Neural Networks (EANN 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1826))

  • 680 Accesses

Abstract

Since its introduction, the attention-based Transformer architecture has become the de facto standard for building models with state-of-the-art performance on many Natural Language Processing tasks. However, it seems that the success of these models might have to do with their exploitation of dataset artifacts, rendering them unable to generalize to other data and vulnerable to adversarial attacks. On the other hand, the attention mechanism present in all models based on the Transformer, such as BERT-based ones, has been seen by many as a potential way to explain these deep learning models: by visualizing attention weights, it might be possible to gain insights on the reasons behind these opaque models’ decisions. This paper introduces AttentiveBERT, an interactive attention weights visualization tool for diagnosing BERT-based models, focusing on explaining the occurrence of shortcut learning. The distinctive feature of this tool is enabling the visual comparison of attention weights before and after a change to the model’s input, in order to visually analyse adversarial attacks. Some illustrations of this use case are explored in this paper.

This research is supported by Calouste Gulbenkian Foundation and by Fundação para a Ciência e a Tecnologia, through LIACC (UIDB/00027/2020).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Available on Github: https://github.com/Goncalerta/AttentiveBERT.

  2. 2.

    https://huggingface.co/boychaboy/SNLI_distilbert-base-cased.

References

  1. Chowdhery, A. et al.: PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311 (2022)

  2. Bahdanau, D., Cho, K., Bengio, Y.: Neural Machine Translation by Jointly Learning to Align and Translate. In: Bengio, Y., LeCun, Y. (eds.) 3rd Int. Conf. on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conf. Track Proceedings (2015)

    Google Scholar 

  3. Bekoulis, G., Papagiannopoulou, C., Deligiannis, N.: A Review on Fact Extraction and Verification. ACM Comput. Surv. 55(1) (nov 2021). https://doi.org/10.1145/3485127

  4. Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632–642. ACL, Lisbon, Portugal (Sep 2015). https://doi.org/10.18653/v1/D15-1075

  5. Branco, R., Branco, A., António Rodrigues, J., Silva, J.R.: Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning. In: Proceedings of 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1504–1521. ACL (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.113

  6. Buhrmester, V., Münch, D., Arens, M.: Analysis of Explainers of Black Box Deep Neural Networks for Computer Vision: a survey. Mach. Learn. Knowl. Extract. 3(4), 966–989 (2021). https://doi.org/10.3390/make3040048

    Article  Google Scholar 

  7. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proc. 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186. ACL, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1423

  8. Du, M., et al.: owards Interpreting and Mitigating Shortcut Learning Behavior of NLU models. In: Proc. 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 915–929. ACL (Jun 2021). https://doi.org/10.18653/v1/2021.naacl-main.71

  9. Feng, S., Wallace, E., Grissom II, A., Iyyer, M., Rodriguez, P., Boyd-Graber, J.: Pathologies of Neural Models Make Interpretations Difficult. In: Proc. 2018 Conf. on Empirical Methods in Natural Language Processing, pp. 3719–3728. ACL, Brussels, Belgium (Oct-Nov 2018). https://doi.org/10.18653/v1/D18-1407

  10. Galassi, A., Lippi, M., Torroni, P.: Attention in Natural Language Processing. IEEE Trans. Neural Netw. Learn. Syst. 32(10), 4291–4308 (10 2021). https://doi.org/10.1109/tnnls.2020.3019893

  11. Garg, S., Ramakrishnan, G.: BAE: BERT-based Adversarial Examples for Text Classification. In: Proc. 2020 Conf. on Empirical Methods in Natural Language Processing (EMNLP), pp. 6174–6181. ACL (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.498

  12. Geirhos, R., et al.: Shortcut learning in deep neural networks. Nature Mach. Intell. 2(11), 665–673 (11 2020). https://doi.org/10.1038/s42256-020-00257-z

  13. Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. In: Bengio, Y., LeCun, Y. (eds.) 3rd Int. Conf. on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings (2015)

    Google Scholar 

  14. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., Smith, N.A.: Annotation Artifacts in Natural Language Inference Data. In: Proc. 2018 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107–112. ACL, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-2017

  15. Han, X., Wallace, B.C., Tsvetkov, Y.: Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5553–5563. ACL (Jul 2020). 10.18653/v1/2020.acl-main.492

    Google Scholar 

  16. Jain, S., Wallace, B.C.: Attention is not Explanation. In: Proceedings 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3543–3556. ACL, Minneapolis, Minnesota (Jun 2019). https://doi.org/10.18653/v1/N19-1357

  17. Jin, D., Jin, Z., Zhou, J.T., Szolovits, P.: Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 8018–8025 (Apr 2020). https://doi.org/10.1609/aaai.v34i05.6311

  18. Koh, P.W., Liang, P.: Understanding Black-Box Predictions via Influence Functions. In: Proceedings of the 34th International Conference on Machine Learning - Volume 70, pp. 1885–1894. JMLR.org (2017)

    Google Scholar 

  19. Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the Dark Secrets of BERT. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Int. J. Conf. on Natural Language Processing (EMNLP-IJCNLP), pp. 4365–4374. ACL, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1445

  20. Kuleshov, V., Thakoor, S., Lau, T., Ermon, S.: Adversarial Examples for Natural Language Classification Problems (2018). https://openreview.net/forum?id=r1QZ3zbAZ

  21. Lee, J., Shin, J.H., Kim, J.S.: Interactive Visualization and Manipulation of Attention-based Neural Machine Translation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 121–126. ACL, Copenhagen, Denmark (Sep 2017). https://doi.org/10.18653/v1/D17-2021

  22. Lei, D., Chen, X., Zhao, J.: Opening the black box of deep learning. arXiv:1805.08355 (2018)

  23. Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-ATTACK: Adversarial Attack Against BERT Using BERT. In: Proceedings 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6193–6202. ACL (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.500

  24. MacCartney, B., Manning, C.D.: Modeling Semantic Containment and Exclusion in Natural Language Inference. In: Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp. 521–528. Coling 2008 Organizing Committee, Manchester, UK (Aug 2008)

    Google Scholar 

  25. Mnih, V., Heess, N., Graves, A., Kavukcuoglu, K.: Recurrent Models of Visual Attention. In: Proceedings of the 27th International Conference on Neural Information Processing Systems - vol. 2, pp.. 2204–2212. NIPS’14, MIT Press, Cambridge, MA, USA (2014)

    Google Scholar 

  26. Morris, J., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: TextAttack: A framework for adversarial attacks, data augmentation, and adversarial training in NLP. In: Proceedings 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 119–126. ACL (Oct 2020). https://doi.org/10.18653/v1/2020.emnlp-demos.16

  27. Niven, T., Kao, H.Y.: Probing Neural Network Comprehension of Natural Language Arguments. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4658–4664. ACL, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1459

  28. Peldszus, A., Stede, M.: Joint prediction in MST-style discourse parsing for argumentation mining. In: Proceedings of the 2015 Conference. on Empirical Methods in Natural Language Processing, pp. 938–948. ACL, Lisbon, Portugal (Sep 2015). https://doi.org/10.18653/v1/D15-1110

  29. Ren, S., Deng, Y., He, K., Che, W.: Generating Natural Language Adversarial Examples through Probability Weighted Word Saliency. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1085–1097. ACL, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1103

  30. Ribeiro, M.T., Wu, T., Guestrin, C., Singh, S.: Beyond accuracy: Behavioral testing of NLP models with CheckList. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4902–4912. ACL (Jul 2020). https://doi.org/10.18653/v1/2020.acl-main.442

  31. Rocha, G., Stab, C., Lopes Cardoso, H., Gurevych, I.: Cross-lingual argumentative relation identification: from English to Portuguese. In: Proceedings of the 5th Workshop on Argument Mining, pp. 144–154. ACL, Brussels, Belgium (Nov 2018). https://doi.org/10.18653/v1/W18-5217

  32. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv abs/1910.01108 (2019)

    Google Scholar 

  33. Serrano, S., Smith, N.A.: Is Attention Interpretable? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2931–2951. ACL, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-1282

  34. Strobelt, H., Gehrmann, S., Behrisch, M., Perer, A., Pfister, H., Rush, A.M.: Seq2seq-Vis: a visual debugging tool for sequence-to-sequence models. IEEE Trans. Visual Comput. Graph. 25(1), 353–363 (2019). https://doi.org/10.1109/TVCG.2018.2865044

    Article  Google Scholar 

  35. Brown, T., et al.: Language Models are Few-Shot Learners. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (eds.) In: Advances in Neural Information Processing Systems. vol. 33, pp. 1877–1901. Curran Associates, Inc. (2020)

    Google Scholar 

  36. Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: FEVER: a Large-scale Dataset for Fact Extraction and VERification. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 809–819. ACL, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-1074

  37. Vaswani, A., et al.: Attention is All You Need. In: Proc. Int. Conf. on Neural Information Processing Systems, pp. 6000–6010. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)

    Google Scholar 

  38. Vig, J.: A Multiscale Visualization of Attention in the Transformer Model. In: Proceedings 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 37–42. ACL, Florence, Italy (Jul 2019). https://doi.org/10.18653/v1/P19-3007

  39. Wagner, R.A., Fischer, M.J.: The String-to-String Correction Problem. J. ACM 21(1), 168–173 (1 1974). https://doi.org/10.1145/321796.321811

  40. Wiegreffe, S., Pinter, Y.: Attention is not not Explanation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Journal Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 11–20. ACL, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1002

  41. Wolf, T., et al.: Transformers: State-of-the-Art Natural Language Processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38–45. ACL (Oct 2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6

Download references

Acknowledgements

This research is supported by the Calouste Gulbenkian Foundation and by LIACC (FCT/UID/CEC/0027/2020), funded by Fundação para a Ciência e a Tecnologia (FCT).

Author information

Authors and Affiliations

  1. Faculdade de Engenharia, Universidade do Porto, Rua Dr. Roberto Frias, 4200-465, Porto, Portugal

    Pedro Gonçalo Correia & Henrique Lopes Cardoso

Authors
  1. Pedro Gonçalo Correia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrique Lopes Cardoso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pedro Gonçalo Correia .

Editor information

Editors and Affiliations

  1. Democritus University of Thrace, Xanthi, Greece

    Lazaros Iliadis

  2. University of Piraeus, Piraeus, Greece

    Ilias Maglogiannis

  3. University of Leon, León, Spain

    Serafin Alonso

  4. Teesside University, Middlesbrough, UK

    Chrisina Jayne

  5. University of the West of England, Bristol, UK

    Elias Pimenidis

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Correia, P.G., Lopes Cardoso, H. (2023). Towards Explaining Shortcut Learning Through Attention Visualization and Adversarial Attacks. In: Iliadis, L., Maglogiannis, I., Alonso, S., Jayne, C., Pimenidis, E. (eds) Engineering Applications of Neural Networks. EANN 2023. Communications in Computer and Information Science, vol 1826. Springer, Cham. https://doi.org/10.1007/978-3-031-34204-2_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34204-2_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34203-5

  • Online ISBN: 978-3-031-34204-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation