Artificial Intelligence and Autonomy: Self-Determination in the Age of Automated Systems

Ernst, Christian

doi:10.1007/978-3-030-32361-5_3

Christian Ernst³

5410 Accesses
5 Citations
3 Altmetric

Abstract

The use of automated (decision-making) systems is becoming increasingly widespread in everyday life. By, for example, producing tailor-made decisions or individual suggestions, these systems increasingly penetrate—intentionally or unintentionally, openly or covertly—a sphere that has long been reserved for individual self-determination. With the advancing digitalisation of everyday life and the increasing proliferation of such systems, it can become more and more difficult for those affected to recognize the impact of these systems or to avoid their influence. This Chapter illustrates the risks that such systems may pose for individual self-determination and possible ways out.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Chapter: USD 29.95; Price excludes VAT (USA)

eBook: USD 119.00; Price excludes VAT (USA)

Softcover Book: USD 159.99; Price excludes VAT (USA)

Hardcover Book: USD 159.99; Price excludes VAT (USA)

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

1.
See also Tischbirek, paras 10 et seq.
2.
See Krohn and Linder (2018).
3.
See www.generalivitality.de/vmp/.
4.
See www.johnhancockinsurance.com/vitality-program.html.
5.
Krohn and Linder (2018).
6.
See the definition in Section 31 German Federal Data Protection Act (BDSG). According to the GDPR, scoring is a subset of profiling pursuant to Article 4(4) GDPR. The GDPR follows a comprehensive approach, see Selmayr and Ehmann (2018), para 75 et seq.; Schantz (2018), para 8; Pötters (2018), para 24. So, it is doubtful whether the national legislator is allowed to enact a rule like Section 31(1) BDSG that is more restrictive for scoring than Article 22 GDPR. Whether or not Section 31 BDSG can still be considered a ‘suitable measure to safeguard the data subject’s rights and freedoms and legitimate interests’ in the sense of Article 22(2)(b) GDPR, is an open question, see Krämer (2018), para 3; skeptical Buchner (2018), paras 4 et seq.; Moos and Rothkegel (2016), p. 567.
7.
Hockling (2013).
8.
Sachverständigenrat für Verbraucherfragen (2018), p. 62; Hartlmaier (2018).
9.
Seibel (2015).
10.
See also Hoffmann-Riem, para 10.
11.
See comprehensive State Council (2014), State Council (2016).
12.
Holland (2018).
13.
State Council (2014).
14.
Botsman (2017).
15.
Botsman (2017).
16.
Meissner (2017), p. 3.
17.
See Ernst (2017a), pp. 1032 et seq.
18.
For a general definition of artificial intelligence see Wischmeyer and Rademacher, paras 5–6.
19.
Broemel and Trute (2016), p. 59.
20.
See paras 38 et seq. and paras 41 et seq.
21.
von Grafenstein et al. (2018), p. 46.
22.
Ebers (2018), pp. 423 et seq.
23.
O’Neil (2016), pp. 194 et seq.; Calo (2014), pp. 1015 et seq.; Ebers (2018), p. 423; Wolff (2015), p. 202.
24.
Sunstein (2016), pp. 124 et seq.; Hansen and Jespersen (2013), pp. 14 et seq.; Lindstrom (2014); von Grafenstein et al. (2018), p. 69; Wolff (2015), pp. 202 et seq.; Seckelmann and Lamping (2016), pp. 191 et seq.
25.
Sunstein (2016), pp. 124 et seq.; Hansen and Jespersen (2013), pp. 14 et seq.; Lindstrom (2014); von Grafenstein et al. (2018), p. 69; Weber and Schäfer (2017), pp. 578 et seq.; Seckelmann and Lamping (2016), pp. 191 et seq.
26.
von Grafenstein et al. (2018), p. 45.
27.
German Federal Constitutional Court 1 BvR 209/83 et al. ‘Volkszählung’ (15 December 1983), BVerfGE 65 p. 43; 1 BvR 256/08 et al. ‘Vorratsdatenspeicherung’ (2 March 2010), BVerfGE 125 p. 232.
28.
In detail von Grafenstein et al. (2018), p. 20; see also Weber and Schäfer (2017), pp. 569 et seq.; Purnhagen and Reisch (2016), pp. 637 et seq.
29.
See Viseu and Suchman (2010).
30.
Thaler and Sunstein (2009), pp. 53 et seq.; Wolff (2015), pp. 201 et seq.; Zillien et al. (2015), p. 87.
31.
See von Grafenstein et al. (2018), pp. 35 et seq.
32.
Schultz (2007), and von Grafenstein et al. (2018), pp. 39 et seq.
33.
Ernst (2017b), pp. 72 et seq.
34.
Ebers (2018), p. 423; Broemel and Trute (2016), p. 59.
35.
See Tischbirek, paras 9 et seq.
36.
See Constitutional Court ‘Volkszählung’ (see note 27), p. 42; Martini (2014), p. 1483; Broemel and Trute (2016), p. 57.
37.
See Wischmeyer; see Hoffmann-Riem, paras 51 et seq.
38.
Hermstrüwer (2016), pp. 227 et seq.; Radlanski (2016), pp. 11 et seq.; Broemel and Trute (2016), p. 53.
39.
See Hoffmann-Riem, paras 54 et seq.
40.
Ebers (2018), p. 423.
41.
Di Fabio (2001), para 147; Broemel and Trute (2016), pp. 57 et seq.
42.
Britz (2007), p. 24; von Grafenstein et al. (2018), p. 54.
43.
Petrlic (2016), p. 96; see Jandt (2016), p. 571; Di Fabio (2001), para 107.
44.
See von Grafenstein et al. (2018), p. 54 with further references.
45.
See von Grafenstein et al. (2018), p. 54 with further references.
46.
Ernst (2017b), p. 72.
47.
See Huster (2015), pp. 24 et seq.; Wolff (2015), pp. 209 et seq.; Weber and Schäfer (2017), pp. 572 et seq.
48.
Thaler and Sunstein (2009), pp. 5 et seq.; see Weber and Schäfer (2017), pp. 575 et seq.; Wolff (2015), pp. 209 et seq. with further references.
49.
Sunstein (2016).
50.
Ebers (2018), p. 423.
51.
See Kloepfer (2002), § 1 para 60; Vesting (2001), pp. 220 et seq.; Druey (1995), pp. 26 et seq.
52.
See von Grafenstein et al. (2018), p. 82.
53.
See von Grafenstein et al. (2018), p. 111.
54.
See Duttweiler and Passoth (2016), pp. 12 et seq.; Crawford et al. (2015), p. 492; von Grafenstein et al. (2018), p. 38.
55.
In Germany see Section 4a UWG.
56.
Köhler (2019), para 1.59; Picht (2016), para 27; Götting (2016), para 14; Ebers (2018), p. 424.
57.
Ebers (2018), p. 424.
58.
CJEU C-178/84 ‘Reinheitsgebot für Bier’ (12 March 1987), 1987 E.C.R. 1227, paras 31–36; C-362/88 ‘GB-Inno-BM/Confédération du commerce luxembourgeois’ (7 March 1990), 1990 E.C.R. I-667, paras 13–19; C-220/98 ‘Estée Lauder’ (13 January 2000), 2000 E.C.R. I-117, paras 27 et seq.; recital 18 of Directive 2005/29/EC; see Weber and Schäfer (2017), p. 579.
59.
Ebers (2018), pp. 423 et seq.
60.
Sosnitza (2016), para 36.
61.
See para 22.
62.
German Federal Supreme Court X ZR 37/12 (16 October 2012), BGHZ 195, 126 para 17; VI ZR 269/12 (14 May 2013), BGHZ 197, 213 para 17; District Court Cologne 9 S 289/02 (16 April 2003), MMR 2003, 481 et seq.; Schulz (2018), para 18. See also Schirmer, paras 16 et seq.
63.
In German law regarding General Terms and Conditions, Section 305c BGB (Civil Code) contains a rule according to which clauses which are surprising for the contractual partner are not valid. Only such contractual conditions shall be legally effective that are expected. This ensures the protection of confidence and takes into account that general terms and conditions are regularly so complex that they are not completely comprehensible for an average consumer, see Basedow (2019), para 1. This protective effect is comparable to the proposal made here. See Hoffmann-Riem, para 23.
64.
An illustrative example of the delimitation difficulties that may be associated with this is the consideration of the font in connection with the granting of loans, as described above see para 5.
65.
Hermstrüwer (2016), p. 227; Radlanski (2016), pp. 11 et seq.; Broemel and Trute (2016), p. 53.
66.
German Federal Constitutional Court 1 BvR 567/89, 1044/89 ‘Bürgschaftsverträge’ (19 October 1993), BVerfGE 89, p. 233.
67.
See German Federal Constitutional Court 1 BvR 26/84 ‘Handelsvertreter’ (7 February 1990), BVerfGE 81, p. 242; ‘Bürgschaftsverträge’ (see note 66), BVerfGE 89 p. 214; 1 BvR 2248/01 ‘Erbvertrag’ (22 March 2004), NJW 2004, p. 2008.
68.
See Hoffmann-Riem, para 14; see Marsch, para 8.
69.
Bachmeier (1995), p. 51; Wuermeling (1996), p. 668; Kamlah (2013), para 2; Ernst (2017a), p. 1031.
70.
See Wischmeyer, paras 25 et seq.

References

Bachmeier R (1995) EG-Datenschutzrichtlinie – Rechtliche Konsequenzen für die Datenschutzpraxis. RDV 11:49–52
Google Scholar
Basedow J (2019) § 305c BGB. In: Säcker FJ et al (eds) Münchener Kommentar zum Bürgerlichen Gesetzbuch, 8th edn. C.H. Beck, München
Google Scholar
Botsman R (2017) Big data meets Big Brother as China moves to rate its citizens. WIRED. 21 October 2017. www.wired.co.uk/article/chinese-government-social-credit-score-privacy-invasion
Britz G (2007) Freie Entfaltung durch Selbstdarstellung. Mohr Siebeck, Tübingen
Google Scholar
Broemel R, Trute H-H (2016) Alles nur Datenschutz? Zur rechtlichen Regulierung algorithmenbasierter Wissensgenerierung. Berliner Debatte Initial 27(4):50–65
Google Scholar
Buchner B (2018) § 31 BDSG. In: Kühling J, Buchner B (eds) DS-GVO, 2nd edn. C.H. Beck, München
Google Scholar
Calo R (2014) Digital Market Manipulation. George Wash Law Rev 82:995–1051
Google Scholar
Crawford K et al (2015) Our metrics, ourselves: a hundred years of self-tracking from the weight scale to the wrist wearable device. Eur J Cult Stud 18:479–496
Article Google Scholar
Di Fabio U (2001) Art. 2(1) GG. In: Maunz T et al (eds) Grundgesetz. Kommentar, 39th edn. C.H. Beck, München
Google Scholar
Druey JN (1995) Information als Gegenstand des Rechts. Schulthess, Zürich
Google Scholar
Duttweiler S, Passoth J-H (2016) Self-Tracking als Optimierungsprojekt? In: Duttweiler S et al (eds) Leben nach Zahlen: Self-Tracking als Optimierungsprojekt?, pp 9–42
Google Scholar
Ebers M (2018) Beeinflussung und Manipulation von Kunden durch Behavioral Microtargeting – Verhaltenssteuerung durch Algorithmen aus der Sicht des Zivilrechts. MMR 21:423–428
Google Scholar
Ernst C (2017a) Algorithmische Entscheidungsfindung und personenbezogene Daten. JuristenZeitung 72:1026–1036
Article Google Scholar
Ernst C (2017b) Die Gefährdung der individuellen Selbstentfaltung durch den privaten Einsatz von Algorithmen. In: Klafki A et al (eds) Digitalisierung und Recht. Bucerius Law School Press, Hamburg, pp 63–81
Google Scholar
Götting H-P (2016) 4a UWG. In: Götting H-P, Nordemann A (eds) UWG. Kommentar, 3rd edn. Nomos, Baden-Baden
Google Scholar
Hansen PG, Jespersen AM (2013) Nudge and the manipulation of choice: a framework for the responsible use of the nudge approach to behaviour change in public policy. Eur J Risk Reg 4:3–28
Article Google Scholar
Hartlmaier B (2018) Überwachung: Umfassendes Social Scoring wie in China ist auch bei uns möglich. WIRED. 9 November 2018. www.wired.de/article/ueberwachung-umfassendes-social-scoring-wie-in-china-ist-auch-bei-uns-moeglich
Hermstrüwer Y (2016) Informationelle Selbstgefährdung. Zur rechtsfunktionalen, spieltheoretischen und empirischen Rationalität der datenschutzrechtlichen Einwilligung und des Rechts auf informationelle Selbstbestimmung. Mohr Siebeck, Tübingen
Google Scholar
Hockling S (2013) Manche Namen senken Scorewert für Kreditwürdigkeit. Die Welt. 23 March 2013. www.welt.de/finanzen/verbraucher/article114709366/Manche-Namen-senken-Scorewert-fuer-Kreditwuerdigkeit.html
Holland M (2018) “Social Scoring”: Ab 2020 Punktesystem für vorbildliches Verhalten in Peking. heise online. 23 November 2018. www.heise.de/newsticker/meldung/Social-Scoring-Ab-2020-Punktesystem-fuer-vorbildliches-Verhalten-in-Peking-4231644.html
Huster S (2015) Selbstbestimmung, Gerechtigkeit und Gesundheit. Nomos, Baden-Baden
Book Google Scholar
Jandt S (2016) Smart Health. Wird die DSGVO den dynamischen Herausforderungen gerecht? DuD 40:571–574
Article Google Scholar
Kamlah W (2013) § 6a BDSG. In: Plath K-U (ed) BDSG. Kommentar, 2nd edn. Dr. Otto Schmidt, Köln
Google Scholar
Kloepfer M (2002) Informationsrecht. C.H. Beck, München
Google Scholar
Köhler H (2019) § 4a UWG. In: Köhler H et al (eds) Gesetz gegen den unlauteren Wettbewerb. Kommentar, 39th edn. C.H. Beck, München
Google Scholar
Krämer W (2018) § 31 BDSG. In: Wolff HA, Brink S (eds) BeckOK Datenschutzrecht, 25th edn. C.H. Beck, München
Google Scholar
Krohn P, Linder R (2018) Lebensversicherer verlangt, dass Kunden Fitness-Tracker nutzen. FAZ. 20 September 2018. www.faz.net/aktuell/wirtschaft/diginomics/lebensversicherung-bei-john-hancock-nur-mit-fitness-tracker-15798146.html
Lindstrom L (2014) So, when is it legitimate to nudge? The Swedish Nudging Network. 5 November 2014, theswedishnudgingnetwork.com/2014/11/05/so-when-is-it-legitimate-to-nudge
Martini M (2014) Big Data als Herausforderung für den Persönlichkeitsschutz und das Datenschutzrecht. DVBl 129:1481–1489
Google Scholar
Meissner M (2017) Chinas gesellschaftliches Bonitätssystem. www.merics.org/sites/default/files/2017-09/China%20Monitor_39_SOCS_DE.pdf
Moos F, Rothkegel T (2016) Nutzung von Scoring-Diensten im Online-Versandhandel. ZD 9:561–567
Google Scholar
O’Neil C (2016) Weapons of math destruction. Allen Lane, Penguin, London
Google Scholar
Petrlic R (2016) Das vermessene Selbst. Von der Selbst-Überwachung zur Fremd-Bestimmung. DuD 40:94–97
Article Google Scholar
Picht P (2016) § 4a UWG. In: Harte-Bavendamm H, Henning-Bodewig F (eds) Gesetz gegen den unlauteren Wettbewerb (UWG). Kommentar, 4th edn. C.H. Beck, München
Google Scholar
Pötters S (2018) Art. 1 DS-GVO. In: Gola P (ed) DS-GVO, 2nd edn. C.H. Beck, München
Google Scholar
Purnhagen K, Reisch L (2016) “Nudging Germany”? Herausforderungen für eine verhaltensbasierte Regulierung in Deutschland. ZEuP 24:629–655
Google Scholar
Radlanski P (2016) Das Konzept der Einwilligung in der datenschutzrechtlichen Realität. Mohr Siebeck, Tübingen
Google Scholar
Sachverständigenrat für Verbraucherfragen (2018) Verbrauchergerechtes Scoring. Berlin 2018
Google Scholar
Schantz P (2018) Art. 1 DSGVO. In: Wolff HA, Brink S (eds), BeckOK Datenschutzrecht, 25th edn. C.H. Beck, München
Google Scholar
Schultz PW et al (2007) The constructive, destructive, and reconstructive power of social norms. Psychol Sci 18:429–434
Article Google Scholar
Schulz S (2018) Art. 22 DS-GVO. In: Gola P (ed) Datenschutz-Grundverordnung. Kommentar, 2nd edn. C.H. Beck, München
Google Scholar
Seckelmann M, Lamping W (2016) Verhaltensökonomischer Experimentalismus im Politik-Labor. DÖV 69:189–200
Google Scholar
Seibel K (2015) Gegen Kreditech ist die Schufa ein Schuljunge. Die Welt. 17 April 2015. www.welt.de/finanzen/verbraucher/article139671014/Gegen-Kreditech-ist-die-Schufa-ein-Schuljunge.html
Selmayr M, Ehmann E (2018) Einführung. In: Selmayr M, Ehmann E (eds) DS-GVO, Kommentar, 2nd edn. C.H. Beck, München
Google Scholar
Sosnitza O (2016) § 4a UWG. In: Ohly A, Sosnitza O (eds) Gesetz gegen den unlauteren Wettbewerb. Kommentar, 7th edn. C.H. Beck, München
Google Scholar
State Council (2014) Planning Outline for the Construction of a Social Credit System (2014–2020). http://chinacopyrightandmedia.wordpress.com/2014/06/14/planning-outline-for-the-construction-of-a-social-credit-system-2014-2020
State Council (2016) State Council Guiding Opinions concerning Establishing and Perfecting Incentives for Promise-keeping and Joint Punishment Systems for Trust-Breaking, and Accelerating the Construction of Social Sincerity. http://chinacopyrightandmedia.wordpress.com/2016/05/30/state-council-guiding-opinions-concerning-establishing-and-perfecting-incentives-for-promise-keeping-and-joint-punishment-systems-for-trust-breaking-and-accelerating-the-construction-of-social-sincer
Sunstein C (2016) People prefer System 2 nudges (kind of). Duke Law J 66:121–168
Google Scholar
Thaler RH, Sunstein C (2009) Nudge: improving decisions about health, wealth and happiness. Penguin Books, London
Google Scholar
Vesting T (2001) Die Bedeutung von Information und Kommunikation für die verwaltungsrechtliche Systembildung. In: Hoffmann-Riem W et al. (eds). Grundlagen des Verwaltungsrechts, Band. 2, Informationsordnung – Verwaltungsverfahren – Handlungsformen, 2nd edn. C.H. Beck, München
Google Scholar
Viseu A, Suchman L (2010) Wearable augmentations: imaginaries of the informed body. In: Edwards J, Penny H, Wade P (eds) Technologized images, technologized bodies. Berghahn, New York, pp 161–184
Google Scholar
von Grafenstein M et al. (2018) Nudging – Regulierung durch Big Data und Verhaltenswissenschaften, ABIDA Gutachten. Berlin, 2018
Google Scholar
Weber F, Schäfer H-B (2017) “Nudging”, Ein Spross der Verhaltensökonomie. Der Staat 56:561–592
Google Scholar
Wolff J (2015) Eine Annäherung an das Nudge-Konzept. Rechtswissenschaft 6:194–222
Article Google Scholar
Wuermeling U (1996) Umsetzung der Europäischen Datenschutzrichtlinie, DB 49:663–671
Google Scholar
Zillien N et al (2015) Zahlenkörper. Digitale Selbstvermessung als Verdinglichung des Körpers. In: Hahn K, Stempfhuber M (eds) Präsenzen 2.0. Medienkulturen im digitalen Zeitalter. Springer, Wiesbaden, pp 77–94
Google Scholar

Download references

Author information

Authors and Affiliations

Helmut-Schmidt-Universität/Universität der Bundeswehr Hamburg, Hamburg, Germany
Christian Ernst

Authors

Christian Ernst
View author publications
You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christian Ernst .

Editor information

Editors and Affiliations

Faculty of Law, University of Bielefeld, Bielefeld, Germany
Thomas Wischmeyer
Faculty of Law, University of Hannover, Hannover, Germany
Timo Rademacher

Rights and permissions

Reprints and permissions

Copyright information

About this chapter

Cite this chapter

Ernst, C. (2020). Artificial Intelligence and Autonomy: Self-Determination in the Age of Automated Systems. In: Wischmeyer, T., Rademacher, T. (eds) Regulating Artificial Intelligence. Springer, Cham. https://doi.org/10.1007/978-3-030-32361-5_3

Download citation

DOI: https://doi.org/10.1007/978-3-030-32361-5_3
Published: 30 November 2019
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32360-8
Online ISBN: 978-3-030-32361-5
eBook Packages: Law and CriminologyLaw and Criminology (R0)

Publish with us

Policies and ethics