Abstract
With automation of routine decisions coupled with more intricate and complex information architecture operating this automation, concerns are increasing about the trustworthiness of these systems. These concerns are exacerbated by a class of artificial intelligence (AI) that uses deep learning (DL), an algorithmic system of deep neural networks, which on the whole remain opaque or hidden from human comprehension. This situation is commonly referred to as the black box problem in AI. Without understanding how AI reaches its conclusions, it is an open question to what extent we can trust these systems. The question of trust becomes more urgent as we delegate more and more decision-making to and increasingly rely on AI to safeguard significant human goods, such as security, healthcare, and safety. Models that “open the black box” by making the non-linear and complex decision process understandable by human observers are promising solutions to the black box problem in AI but are limited, at least in their current state, in their ability to make these processes less opaque to most observers. A philosophical analysis of trust will show why transparency is a necessary condition for trust and eventually for judging AI to be trustworthy. A more fruitful route for establishing trust in AI is to acknowledge that AI is situated within a socio-technical system that mediates trust, and by increasing the trustworthiness of these systems, we thereby increase trust in AI.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Opacity and the black box problem are not exclusive to DL as other forms of machine learning also can be opaque. Because DL is paradigmatic of the black box problem, it is the focus of this paper.
ProPublica’s analysis of COMPAS is useful for heuristic purposes but not without criticism. Subsequent analyses have raised questions about ProPublica’s conclusions regarding racial bias but uncovered other serious concerns that remain hidden due to a lack of transparency (Fisher et. al., 2019; Rudin et. al., 2020). Other analysis has suggested that it is no more fair or accurate than human judgment (Dressel & Farid, 2018).
References
Baier, A. (1986). Trust and antitrust. Ethics, 96(2), 231–260. https://doi.org/10.1086/292745
Bleicher, A. (2017). Demystifying the Black Box that is AI. Scientific American. https://www.scientificamerican.com/article/demystifying-the-black-box-that-is-ai/. Accessed 6/4/2020.
Burrell, J. (2016). How the machine ‘thinks’: Understanding opacity in machine learning algorithms. Big Data & Society, 3(1). https://doi.org/10.1177/2053951715622512
Castelvecchi, D. (2016). Can we open the black box of AI? Nature, 538, 21–23. https://doi.org/10.1038/538020a
D’Agostino, M., & Durante, M. (2018). Introduction: The governance of algorithms. Philosophy and Technology, 31(4), 499–505. https://doi.org/10.1007/s13347-018-0337-z
Dahl, E. S. (2018). Appraising black-boxed technology: The positive prospects. Philosophy and Technology, 31(4), 571–591. https://doi.org/10.1007/s13347-017-0275-1
Danaher, J. (2016). The threat of algocracy: Reality, resistance and accommodation. Philosophy and Technology, 29, 245–268. https://doi.org/10.1007/s13347-015-0211-1
Dressel, J., & Farid, H. (2018). The accuracy, fairness, and limits of predicting recidivism. Science Advances, 4(1). https://doi.org/10.1126/sciadv.aao5580
Durante, M. (2010). What is the model of trust for multi-agent systems? Whether or not e-trust applies to autonomous agents. Knowledge Technology & Policy, 23, 347–366.
Edelman Trust Barometer. (2020). Special report: Trust in Technology.
Eubanks, V. (2018). Automating inequality : How high-tech tools profile, police, and punish the poor (1st ed.). St. Martin’s Press.
Fisher, A., Rudin, C., & Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable’s importance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20, 1–8.
Flores, F., & Solomon, R. C. (1998). Creating trust. Business Ethics Quarterly, 8(2), 205–232.
Floridi, L., & Sanders, J. W. (2004). On the morality of artificial agents. Minds and Machine, 14, 349–379.
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys, 51(5), 1–42. https://doi.org/10.1145/3236009
Hardin, R. (2004). Trust and Trustworthiness. Vol. 4. The Russell Sage Foundation. https://doi.org/10.4324/9781315542294-2
Humphreys, P. (2009). The philosophical novelty of computer simulation methods. Synthese, 169(3), 615–626. https://doi.org/10.1007/s11229-008-9435-2
Jones, K. (1996). Trust as an affective attitude. Ethics, 107(1), 4–25. https://doi.org/10.1086/233694
Jones, K. (2012). Trustworthiness. Ethics, 123(1), 61–85. https://doi.org/10.1086/667838
Kiran, A. H., & Verbeek, P-P. (2010). Trusting our selves to technology. Knowledge Technology & Policy, 23, 409–27.
Larson, J., Mattu, S., Kirchner, L., & Angwin, J. (2016). How we analyzed the COMPAS recidivism algorithm. ProPublica. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm. Accessed 6/4/2020.
Miotto, R., Li, L., Kidd, B. A., & Dudley, J. T. (2016). Deep patient: An unsupervised representation to predict the future of patients from the electronic health records. Scientific Reports, 6(May), 1–10. https://doi.org/10.1038/srep26094
Mökander, J., & Floridi, L. (2021). Ethics-based auditing to develop trustworthy AI. Minds and Machines. Springer Science and Business Media B.V. https://doi.org/10.1007/s11023-021-09557-8
Nickel, P. J., Franssen, M., & Kroes, P. (2010). Can we make sense of the notion of trustworthy technology? Knowledge Technology & Policy, 23, 429–44.
O’Neill, O. (2020). Trust and accountability in a digital age. Philosophy, 95(1), 3–17. https://doi.org/10.1017/S0031819119000457
Páez, A. (2019). The pragmatic turn in explainable artificial intelligence (XAI). Minds and Machines, 29(3), 441–459. https://doi.org/10.1007/s11023-019-09502-w
Pettit, P. (1995). The cunning of trust. Philosophy & Public Affairs, 24(3), 202–225. https://doi.org/10.1111/j.1088-4963.1995.tb00029.x
Pitt, J. C. (2010). It’s not about technology. Knowledge Technology & Policy, 23, 445–54.
Rai, A. (2020). Explainable AI: From black box to glass box. Journal of the Academy of Marketing Science, 48, 137–141. https://doi.org/10.1007/s11747-019-00710-5
Robbins, S. (2019). A misdirected principle with a catch: Explicability for AI. Minds and Machines, 29(4), 495–514. https://doi.org/10.1007/s11023-019-09509-3
Rudin, C., Wang, C., & Coker, B. (2020). The age of secrecy and unfairness in recidivism prediction. Harvard Data Science Review, 2(1), 1–54. https://doi.org/10.1162/99608f92.6ed64b30
Simpson, T. W. (2012). What is trust? Pacific Philosophical Quarterly, 93(4), 550–569. https://doi.org/10.1111/j.1468-0114.2012.01438.x
Taddeo, M., & Floridi, L. (2011). The case for e-trust. Ethics and Information Technology, 13, 1–3. https://doi.org/10.1007/s10676-010-9263-1
von Eschenbach, W. J. (2019). Trust as a public virtue. In J. Arthur (ed.), Virtues in the public sphere: Citizenship, Friendship, Public Duty. Routledge. https://doi.org/10.4324/9780429505096
Zednik, C. (2019). Solving the black box problem: A normative framework for explainable artificial intelligence. Philosophy & Technology, 34, 265–288. https://doi.org/10.1007/s13347-019-00382-7
Zerilli, J., J. Maclaurin, A. Knott, and C. Gavaghan. (2019). Transparency in algorithmic and human decision-making: Is there a double standard? Philosophy and Technology, 32(4), 661–683. https://doi.org/10.1007/s13347-018-0330-6
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
von Eschenbach, W.J. Transparency and the Black Box Problem: Why We Do Not Trust AI. Philos. Technol. 34, 1607–1622 (2021). https://doi.org/10.1007/s13347-021-00477-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13347-021-00477-0