Advertisement

Profiling and Automated Decision-Making: Legal Implications and Shortcomings

  • Stefanie Hänold
Chapter
Part of the Perspectives in Law, Business and Innovation book series (PLBI)

Abstract

The increased use of profiling and automated decision-making systems raises a number of challenges and concerns. The underlying algorithms embody a considerable potential for discrimination and unfair treatment. Furthermore, individuals are treated as passive objects of algorithmic evaluation and decision tools and are unable to present their values and positions. They are no longer perceived as individuals in their own right: all that matters is the group they are assigned to. Profiling and automated decision-making techniques also depend on the processing of personal data, and a significant number of the available applications are highly privacy-intrusive. This article analyses how the European General Data Protection Regulation (GDPR) responds to these challenges. In particular, Art. 22 GDPR, which provides the right not to be subject to automated individual decision-making, as well as the information obligations under Art. 13 (2) (f) and Art. 14 (2) (g) GDPR and the access right under Art. 15 (1) (h) GDPR, will be examined in detail. General data protection principles, particularly the principle of fairness, as well as specific German scoring provisions and anti-discrimination rules, are looked at, too. In conclusion, various shortcomings of the present legal framework are identified and discussed and a short outlook for potential future steps presented.

Keywords

Profiling Automated decision-making Algorithm Explanation General Data Protection Regulation (GDPR) 

Notes

Acknowledgements

This work has been supported by the EU project SoBigData (http://www.sobigdata.eu/) which receives funding from the European Union’s Horizon 2020 research and innovation program under grant agreement No. 654024 and the German national project ABIDA (http://www.abida.de/) which has been funded by the Bundesministerium für Bildung und Forschung (BMBF). The author would like to thank Marc Stauch and Julia Pfeiffenbring for their valuable support.

References

  1. Article 29 Data Protection Working Party. (2018). Guidelines on automated individual decision-making and profiling for the purposes of regulation 2016/679. 17/EN WP251rev.01.Google Scholar
  2. Bäcker, M. (2018). In J. Kühling & B. Buchner (Eds.), Datenschutz-Grundverordnung BDSG Kommentar (2nd ed.). Munich: C.H. Beck.Google Scholar
  3. Buchner, B. (2018). In J. Kühling & B. Buchner (Eds.), Datenschutz-Grundverordnung BDSG Kommentar (2nd ed.). Munich: C.H. Beck.Google Scholar
  4. Clifford Chance. (2017). Me, myself and AI: When AI meets personal data. Available at: https://talkingtech.cliffordchance.com/en/cybersecurity/me–myself-and-ai–when-ai-meets-personal-data-.html. Accessed May 10, 2018.
  5. Datatilsynet, The Norwegian Data Protection Authority. (2018). Artificial intelligence and privacy. Report, January 2018. Available at: https://www.datatilsynet.no/globalassets/global/english/ai-and-privacy.pdf. Accessed May 16, 2018.
  6. Der Tagesspiegel. (2018). Rekrutierung beim Versicherungskonzern Talanx. Wo Roboter Manager testen. Available at: https://www.tagesspiegel.de/politik/rekrutierung-beim-versicherungskonzern-talanx-wo-roboter-manager-testen/21147540.html. Accessed 16 May 2018.
  7. Edwards, L., & Veale, M. (2017). Slave to the algorithm? Why a ‘right to an explanation’ is probably not the remedy you are looking for [draft, August 17, 2017]. Duke Law & Technology Review, 16(18), 18–84 (forthcoming).Google Scholar
  8. Ernst, C. (2017). Algorithmische Entscheidungsfindung und personenbezogene Daten. JuristenZeitung, 72(21), 1026–1036.CrossRefGoogle Scholar
  9. European Data Protection Board. (2018). Endorsement 1/2018. Available at: https://edpb.europa.eu/sites/edpb/files/files/news/endorsement_of_wp29_documents_en_0.pdf. Accessed June 07, 2018.
  10. Forgó, N., Hänold, S., & Schütze, B. (2017). The principle of purpose limitation and Big Data. In M. Corrales, M. Fenwick, & N. Forgó (Eds.), New technology, Big Data and the law. Singapore: Springer.Google Scholar
  11. Greve, H. (2017). Das neue Bundesdatenschutzgesetz. NVwZ, 36(11), 737–744.Google Scholar
  12. Hacker, P., & Petkovka, B. (2017). Reining in the big promise of Big Data: Transparency, inequality, and new regulatory frontiers. Northwestern Journal of Technology and Intellectual Property, 15(1), 1–42.Google Scholar
  13. Helfrich, M. (2017). In G. Sydow (Ed.), Europäische Datenschutzgrundverordnung Handkommentar. Baden-Baden: Nomos.Google Scholar
  14. Hildebrandt, M. (2009). Who is profiling who? Invisible visibility. In S. Gutwirth, et al. (Eds.), Reinventing data protection?. Dordrecht: Springer.Google Scholar
  15. Hladjk, J. (2017). In E. Ehmann & M. Selmayr (Eds.), Datenschutz-Grundverordnung Kommentar. Munich: C.H. Beck.Google Scholar
  16. Hoffmann-Riem, W. (2017). Verhaltenssteuerung durch Algorithmen—Eine Herausforderung für das Recht. Archiv des öffentlichen Rechts, 142(1), 1–42.CrossRefGoogle Scholar
  17. Hofmann, K. (2016). Smart factory—Arbeitnehmerdatenschutz in der Industrie 4.0—Datenschutzrechtliche Besonderheiten und Herausforderungen. ZD, 6(1), 12–17.Google Scholar
  18. Holzinger, A., et al. (2017). What do we need to build explainable AI systems for the medical domain? Available at: http://arxiv.org/pdf/1712.09923v1. Accessed May 10, 2018.
  19. IT Finanzmagazin. (2017). N26 Echtzeit-Kredit: Per smartphone-app automatisiert zum Kredit bis 25.000 €. Available at: https://www.it-finanzmagazin.de/n26-echtzeit-kredit-in-wenigen-minuten-per-smartphone-app-automatisiert-zum-kredit-bis-25-000-e-44535/. Accessed May 16, 2018.
  20. James, K. (2015). 6 ways to avoid sneaky online price changes. WISEBREAD. Available at: http://www.wisebread.com/6-ways-to-avoid-sneaky-online-price-changes. Accessed May 16, 2018.
  21. Jandt, S. (2015). Big Data und die Zukunft des Scoring. Kommunikation und Recht, 18(6, Beihefter 2), 6–8.Google Scholar
  22. Kamlah, W. (2003). Das Scoring-Verfahren der SCHUFA. Multimedia und Recht, 6(2), V–VII.Google Scholar
  23. Kamlah, W. (2016). In K. U. Plath (Ed.), BDSG/DSGVO Kommentar zum BDSG und zur DSGVO sowie den Datenschutzbestimmungen von TMG und TKG (2nd ed.). Cologne: Otto Schmidt.Google Scholar
  24. Knight, W. (2017). The dark secret at the heart of AI. MIT Technology Review. Available at: https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/. Accessed May 10, 2018.
  25. Knoche, K. (2018). KI: In kritischen Anwendungen muss die künstliche Intelligenz nachvollziehbare Ergebnisse liefern. IT Finanzmagazin. Available at: https://www.it-finanzmagazin.de/ki-kuenstliche-intelligenz-nachvollziehbare-ergebnisse-67860/. Accessed May 16, 2018.
  26. Kramer, B. (2018). Der Algorithmus diskriminiert nicht”—Interview. ZEITONLINE. Available at: https://www.zeit.de/arbeit/2018-01/roboter-recruiting-bewerbungsgespraech-computer-tim-weitzel-wirtschaftsinformatiker. Accessed May 16, 2018.
  27. Kühling, J., & Martini, M. (2016). Die Datenschutz-Grundverordnung: Revolution oder evolution im europäischen und deutschen Datenschutzrecht? EuZW, 27(12), 448–454.Google Scholar
  28. Lüdemann, V., Sengstacken, C., & Vogelpohl, K. (2014). Pay as you drive: Datenschutz in der Telematikversicherung. RDV, 6, 302–306.Google Scholar
  29. Malgieri, G., & Comandé, G. (2017). Why a right to legibility of automated decision-making exists in the general data protection regulation. International Data Privacy Law, 7(4), 243–265.CrossRefGoogle Scholar
  30. Martini, M. (2017). Algorithmen als Herausforderung für die Rechtsordnung. JuristenZeitung, 72(21), 1017–1025.CrossRefGoogle Scholar
  31. Martini, M. (2018). In B. Paal & D. A. Pauly (Eds.), Datenschutz-Grundverordnung Bundesdatenschutzgesetz (2nd ed.). Munich: C.H. Beck.Google Scholar
  32. McLellan, C. (2016). Inside the black box: Understanding AI decision-making. ZDNet. Available at: https://www.zdnet.com/article/inside-the-black-box-understanding-ai-decision-making/. Accessed May 16, 2018.
  33. Paal, B., & Hennemann, M. (2018). In B. Paal & D. A. Pauly (Eds.), Datenschutz-Grundverordnung Bundesdatenschutzgesetz (2nd ed.). Munich: C.H. Beck.Google Scholar
  34. Schaar, P. (2016). Algorithmentransparenz. ALGORITHMWATCH. Available at: https://algorithmwatch.org/de/algorithmentransparenz/. Accessed May 25, 2018.
  35. Schermer, B. W. (2011). The limits of privacy in automated profiling and data mining. Computer Law & Security Review, 27(1), 45–52.CrossRefGoogle Scholar
  36. Schmidt-Wudy, F. (2018). In H. A. Wolff & S. Brink (Eds.), Beck OK Datenschutzrecht (23rd ed.). Munich: C.H. Beck.Google Scholar
  37. Schönhaar, L. (2018). Bewerbung: Unternehmen nutzen eine neue recruiting-Methode, die vor allem jungen und unerfahrenen Bewerbern helfen kann. Available at: https://www.businessinsider.de/bewerbung-neue-recruiting-methode-koennte-jungen-und-unerfahrenen-bewerben-helfen-2018-2. Accessed May 16, 2018.
  38. Schulz, S. (2017). In P. Gola (Ed.), Datenschutz-Grundverordnung VO (EU) 2016/679 Kommentar. Munich: C.H. Beck.Google Scholar
  39. Schwichtenberg, S. (2015). “Pay as you drive”—Neue und altbekannte probleme. Datenschutz Datensich, 39(6), 378–382.CrossRefGoogle Scholar
  40. Steppe, R. (2017). Online price discrimination and personal data. A general data protection regulation perspective. Computer Law & Security Review, 33(6), 768–785.CrossRefGoogle Scholar
  41. Stolberg, M., Ceccotti, C. (2018). XAI—Explainable AI: Wissen was die KI wirklich macht—So bleibt künstliche Intelligenz erklärbar. IT Finanzmagazin. Available at: https://www.it-finanzmagazin.de/xai-wissen-was-die-ki-wirklich-macht-kuenstliche-intelligenz-erklaerbar-67609/. Accessed May 10, 2018.
  42. Taeger, J. (2017). Verbot des profiling nach Art. 22 DS-GVO und die Regulierung des Scoring ab Mai 2018. RDV, 33(1), 3–9.Google Scholar
  43. Vedder, A., & Naudts, L. (2017). Accountability for the use of algorithms in a Big Data environment. International Review of Law, Computers & Technology, 31(2), 206–224.CrossRefGoogle Scholar
  44. Von Lewinski, K. (2018). In H. A. Wolff & S. Brink (Eds.), Beck OK Datenschutzrecht (23rd ed.). Munich: C.H. Beck.Google Scholar
  45. Wachter, S., Mittelstadt, B., & Floridi, L. (2017). Why a right to explanation of automated decision-making does not exist in the general data protection regulation. International Data Privacy Law, 7(2), 76–99.CrossRefGoogle Scholar
  46. Weichert, T. (2014). Scoring in Zeiten von Big Data. ZRP, 47(6), 168–171.Google Scholar
  47. Zuiderveen Borgesius, F., & Poort, J. (2017). Online price discrimination and EU data privacy law. Journal of Consumer Policy, 40(3), 347–366.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.Institute for Legal Informatics, Leibniz Universität HannoverHannoverGermany

Personalised recommendations