Skip to main content
SpringerLink
Log in

The Duet of Representations and How Explanations Exacerbate It

  • Conference paper
  • First Online:
Explainable Artificial Intelligence (xAI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1902))

Included in the following conference series:

Abstract

An algorithm effects a causal representation of relations between features and labels in the human’s perception. Such a representation might conflict with the human’s prior belief. Explanations can direct the human’s attention to the conflicting feature and away from other relevant features. This leads to causal overattribution and may adversely affect the human’s information processing. In a field experiment we implemented an XGBoost-trained model as a decision-making aid for counselors at a public employment service to predict candidates’ risk of long-term unemployment. The treatment group of counselors was also provided with SHAP. The results show that the quality of the human’s decision-making is worse when a feature on which the human holds a conflicting prior belief is displayed as part of the explanation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Setting X to \(x_0\), Y would be \(y_0\).

  2. 2.

    Had X been \(x_0\), Y would have been \(y_0\).

References

  1. Bundorf, K., Polyakova, M., Tai-Seale, M.: How do humans interact with algorithms? experimental evidence from health insurance. Technical report, National Bureau of Economic Research (2019)

    Google Scholar 

  2. Colin, J., Fel, T., Cadène, R., Serre, T.: What i cannot predict, i do not understand: a human-centered evaluation framework for explainability methods. Adv. Neural. Inf. Process. Syst. 35, 2832–2845 (2022)

    Google Scholar 

  3. Cyert, R.M., March, J.G., et al.: A Behavioral Theory of the Firm, vol. 2. Englewood Cliffs, NJ (1963)

    Google Scholar 

  4. DeLanda, M.: Materialist Phenomenology: A Philosophy of Perception. Bloomsbury Publishing (2021)

    Google Scholar 

  5. Dietvorst, B.J., Simmons, J.P., Massey, C.: Algorithm aversion: people erroneously avoid algorithms after seeing them err. J. Exp. Psychol. Gen. 144(1), 114 (2015)

    Article  Google Scholar 

  6. Dietvorst, B.J., Simmons, J.P., Massey, C.: Overcoming algorithm aversion: people will use imperfect algorithms if they can (even slightly) modify them. Manage. Sci. 64(3), 1155–1170 (2018)

    Article  Google Scholar 

  7. Fiske, S.T., Kenny, D.A., Taylor, S.E.: Structural models for the mediation of salience effects on attribution. J. Exp. Soc. Psychol. 18(2), 105–127 (1982)

    Article  Google Scholar 

  8. Fügener, A., Grahl, J., Gupta, A., Ketter, W.: Will humans-in-the-loop become borgs? merits and pitfalls of working with AI. Manage. Inf. Syst. Q. (MISQ)-Vol 45 (2021)

    Google Scholar 

  9. Fügener, A., Grahl, J., Gupta, A., Ketter, W.: Cognitive challenges in human-artificial intelligence collaboration: investigating the path toward productive delegation. Inf. Syst. Res. 33(2), 678–696 (2022)

    Article  Google Scholar 

  10. Galhotra, S., Pradhan, R., Salimi, B.: Explaining black-box algorithms using probabilistic contrastive counterfactuals. In: Proceedings of the 2021 International Conference on Management of Data, pp. 577–590 (2021)

    Google Scholar 

  11. Gao, R., Saar-Tsechansky, M., De-Arteaga, M., Han, L., Lee, M.K., Lease, M.: Human-AI collaboration with bandit feedback. arXiv preprint arXiv:2105.10614 (2021)

  12. Gigerenzer, G., Goldstein, D.G.: Reasoning the fast and frugal way: models of bounded rationality. Psychol. Rev. 103(4), 650 (1996)

    Article  Google Scholar 

  13. Gigerenzer, G., Todd, P., Group, A.: Simple heuristics that make us smart (1999)

    Google Scholar 

  14. Glikson, E., Woolley, A.W.: Human trust in artificial intelligence: review of empirical research. Acad. Manag. Ann. 14(2), 627–660 (2020)

    Article  Google Scholar 

  15. Guidotti, R.: Evaluating local explanation methods on ground truth. Artif. Intell. 291, 103428 (2021)

    Article  MathSciNet  Google Scholar 

  16. Han, T., Srinivas, S., Lakkaraju, H.: Which explanation should i choose? a function approximation perspective to characterizing post hoc explanations. arXiv preprint arXiv:2206.01254 (2022)

  17. Jacovi, A., Marasović, A., Miller, T., Goldberg, Y.: Formalizing trust in artificial intelligence: prerequisites, causes and goals of human trust in AI. In: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 624–635 (2021)

    Google Scholar 

  18. Kawaguchi, K.: When will workers follow an algorithm? a field experiment with a retail business. Manage. Sci. 67(3), 1670–1695 (2021)

    Article  Google Scholar 

  19. Lagnado, D.A.: Explaining the Evidence: How the Mind Investigates the World. Cambridge University Press, Cambridge (2021). https://doi.org/10.1017/9780511794520

  20. Lebovitz, S., Lifshitz-Assaf, H., Levina, N.: To engage or not to engage with AI for critical judgments: how professionals deal with opacity when using AI for medical diagnosis. Organ. Sci. 33(1), 126–148 (2022)

    Article  Google Scholar 

  21. Lombrozo, T.: Simplicity and probability in causal explanation. Cogn. Psychol. 55(3), 232–257 (2007)

    Article  Google Scholar 

  22. Lu, H., Yuille, A.L., Liljeholm, M., Cheng, P.W., Holyoak, K.J.: Bayesian generic priors for causal learning. Psychol. Rev. 115(4), 955 (2008)

    Article  Google Scholar 

  23. Lundberg, S.M., Lee, S.I.: A unified approach to interpreting model predictions. In: Advances in Neural Information Processing Systems, pp. 4765–4774 (2017)

    Google Scholar 

  24. Nauta, M., et al.: From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai. arXiv preprint arXiv:2201.08164 (2022)

  25. Pearl, J., Mackenzie, D.: The Book of Why: The New Science of Cause and Effect. Basic books (2018)

    Google Scholar 

  26. Ribeiro, M.T., Singh, S., Guestrin, C.: " why should i trust you?" explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)

    Google Scholar 

  27. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)

    Google Scholar 

  28. Simon, H.A.: Administrative Behavior. Simon and Schuster, New York (2013)

    Google Scholar 

  29. Slack, D.Z., Hilgard, S., Singh, S., Lakkaraju, H.: Reliable post hoc explanations: modeling uncertainty in explainability. In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021). https://openreview.net/forum?id=rqfq0CYIekd

  30. Sun, J., Zhang, D.J., Hu, H., Van Mieghem, J.A.: Predicting human discretion to adjust algorithmic prescription: a large-scale field experiment in warehouse operations. Manage. Sci. 68(2), 846–865 (2022)

    Article  Google Scholar 

  31. Taylor, S.E., Crocker, J., Fiske, S.T., Sprinzen, M., Winkler, J.D.: The generalizability of salience effects. J. Pers. Soc. Psychol. 37(3), 357 (1979)

    Article  Google Scholar 

  32. Taylor, S.E., Fiske, S.T.: Point of view and perceptions of causality. J. Pers. Soc. Psychol. 32(3), 439 (1975)

    Article  Google Scholar 

  33. Taylor, S.E., Fiske, S.T.: Salience, attention, and attribution: top of the head phenomena. In: Advances in Experimental Social Psychology, vol. 11, pp. 249–288. Elsevier (1978)

    Google Scholar 

  34. Ullman, D., Malle, B.F.: What does it mean to trust a robot? steps toward a multidimensional measure of trust. In: Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction, pp. 263–264 (2018)

    Google Scholar 

  35. Wan, C., Belo, R., Zejnilovic, L.: Explainability’s gain is optimality’s loss? how explanations bias decision-making. In: Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 778–787 (2022)

    Google Scholar 

  36. Woodward, J.: Causation with A Human Face: Normative Theory and Descriptive Psychology. Oxford University Press, Oxford (2021)

    Book  Google Scholar 

  37. Xiang, Y., Vélez, N., Gershman, S.J.: Collaborative decision making is grounded in representations of other people’s competence and effort (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Rotterdam School of Management, Erasmus University, Rotterdam, The Netherlands

    Charles Wan

  2. Nova School of Business and Economics, Universidade NOVA de Lisboa, Lisbon, Portugal

    Rodrigo Belo, Leid Zejnilović & Susana Lavado

Authors
  1. Charles Wan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rodrigo Belo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Leid Zejnilović
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Susana Lavado
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charles Wan .

Editor information

Editors and Affiliations

  1. Technological University Dublin, Dublin, Ireland

    Luca Longo

Appendices

A Empirical Setting

(See Table 4).

Table 4. Treatment assignment to job centers
Full size table

B User Interface

(See Figs. 1 and 2)

Fig. 1.
figure 1

User Interface for the Control Group. In the “Modelo Atual” panel the risk score and the risk assessment are shown. In the “Justificação da Segemntação Atribuída” panel the counselor has to select her own risk assessment on the top and her confidence level on the right.

Full size image
Fig. 2.
figure 2

User Interface for the Treatment Group. SHAP are shown in the “Principais Factores” panel with their respective effect on the risk of LTU.

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wan, C., Belo, R., Zejnilović, L., Lavado, S. (2023). The Duet of Representations and How Explanations Exacerbate It. In: Longo, L. (eds) Explainable Artificial Intelligence. xAI 2023. Communications in Computer and Information Science, vol 1902. Springer, Cham. https://doi.org/10.1007/978-3-031-44067-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-44067-0_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-44066-3

  • Online ISBN: 978-3-031-44067-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation