Artificial intelligence as a challenge for the law: the example of “Doctor Algorithm”

Society increasingly relies on algorithms to improve decisions with the aid of artificial intelligence. This is particularly true in the field of medicine where, based on algorithms, ever increasing volumes of data are to be analysed and provide the basis for tailored medical treatment. This leads to the question of how the law should react to the adoption of such algorithm-based decisions. Will new regulatory approaches be required if algorithms are employed for treatment decisions, or can one resort to the basic principles applied to determine the standard of medical treatment? Who will be liable if the use of algorithms leads to malpractice, in particular if it is the programming of the algorithms that has caused the fault?


AI-based algorithms
Algorithms are anything but new in medicine, where they are not associated with artificial intelligence (AI) or autonomous information technology (IT) systems but refer to the inner logic of medical guidelines. These guidelines, intended to support doctors in the search for the right treatment decision, are also structured like algorithms; they start out from a certain clinical problem and lead step by step with if-then conditions to a certain solution, i.e. the concrete treatment decision.
What is new today if we talk about a "Doctor Algorithm" in medicine is not the fact that medical treatment decisions are based on an algorithmic logic but, rather, the ways in which these algorithms are created: they are no longer based on "human" expertise that is transformed into written guidelines; instead, they basically constitute software that is fed by a potentially unlimited data pool ("Big Data") and, if the AI is not static but dynamic, constantly develops further and improves itself through "machine learning".
In particular in the field of diagnostics, these AI-based algorithms provide more reliable diagnostic results of screenings than their human counterparts even today [13]. Yet there are also cases of algorithms that get it wrong when diagnostic or therapeutic decisions are concerned-such as in the frequently quoted example of an algorithm that was used by numerous US clinics to determine which patients should be chosen for so-called high-risk care management [12]. This algorithm made more than an average number of recommendations for white patients for the programmes, although Afro-American patients should have been chosen much more frequently due to their medical histories. It was finally established that the reason was a relatively simple error in the programming of the algorithm, which was thus not somehow "racist" but simply "stupid"-however, with grave consequences for the patients concerned [11].

Algorithms and the law
In view of such errors-and their grave consequences-of supposedly intelligent AI, one can ask what is or should be the position of the law as to the question of under which conditions machine programmes may decide about the destiny of human beings. To begin with, it can be stated that the general AI regulation tends to be open to such AI-based decisions-in contrast to medical law, which takes a considerably more cautious view.

Basic regulatory approaches
The legislator has long been aware of the problem of AI-based decisions. The legislative materials on the EU Data Protection Directive (EDPD) of 1995 document that, already back then, the Commission viewed the danger of a misuse of informatics in decision-making as one of the "main dangers of the future" since the results generated by the machine had a "seemingly objective and indisputable character". 1 In the end, however, the European legislator decided to adopt a regulation that does not generally exclude even an exclusively automated decision at the expense of the person concerned (Art. 15 EDPD)-and the same applies for the similarly worded regulation of Art. 22 General Data Protection Regulation (GDPR) replacing it. Even an automated decision that is not backed by an individual person, but only by an IT system, is not necessarily inadmissible, provided that the person affected by the decision can subsequently appeal against the decision or, in cases of particularly sensitive decisions, that the person concerned has consented to it in advance.
The proposal of the European Commission on a regulatory framework for AI of 21 April 2021 (AI Act) 2 goes in a very similar direction: AI-based decisions are in principle admissible; only in the case of a few AI applications, the Commission perceives unacceptable risks and thus recommends prohibiting AI, e.g. with regard to the notorious social scoring practised in China or biometrical long-range realtime identification.
Apart from these and a few other carefully defined exceptions, AI-based decisions are to be admissible in principle if accompanied by a graduated system of safety instruments depending on the associated risks. 3 For so-called high-risk AI systems, these safety instruments include adequate "human oversight" in order to minimise risks. 4 This reflects once again the basic idea that AI decisions are acceptable if they are not altogether beyond human control but can at least be understood and questioned by human beings, and if necessary controlled, overruled or turned off, as regulated in detail by Art. 14 of the draft regulation.

The medical law perspective
If one compares the general European AI Regulation with the prevailing medical perspective for example in Germany, the latter appears to be considerably more reticent. The idea that decisions can be made by algorithms alone, provided that the doctor may revise an AI decision in the individual case, or the patient can "resist" it, is difficult to reconcile with the traditional role of the physician. 5 It is still the basic assumption of medical law that the doctor is at the centre of medical treatment. According to the case law of the German Federal Court of Justice, it is "primarily for the doctor" to choose the concrete method of treatment, ideally in a personal and immediate exchange with the patient. 6 It is for this very reason that German medical professional law found it difficult to accept telemedicine, i.e. remote treatment and consultation, for many years and only a reform of the law in 2018 has made it possible to provide medical advice or treatment via so-called communications media in individual cases and under strict conditions. 7 If the use of a simple communications medium as an intermediary in the exchange between doctor and patient is already such a big step away from the model doctor-patient relationship, this must be all the more true for the use of a decision-making medium, i.e. a medium directly involved in medical decision-making. Consequently, this specific decision-making dimension of AI is only discussed at an abstract level, if at all, and physicians and medical ethics experts generally agree that algorithms must never replace but can only support the doctor, claiming that the doctor must not be degraded to a "computer operator" or a "robo-doc" who announces and executes a machine code. 8 This basic attitude is reflected by the present approach where liability requirements for doctors employing AI systems are concerned. Such a constellation is equated with the conventional use of "medical-technical devices", an anaesthesia device, a tube, or another treatment device [8]. Consequently, the doctor is required to ensure regular maintenance or updates, to be trained in the use of technology, to pay close attention to the manuals and to monitor correct functioning. The specific 4 Artificial Intelligence Act of the European Commission, Art. 14 No. 1. 5 Also in favour of a cautious use of artificial intelligence at present: Artificial intelligence in health care: within touching distance (Editorial), The Lancet 390 (2017) 2739. 6  decision-making dimension of the use of algorithms in everyday medical practice is more or less ignored by equating AI with other technical devices. 9 However, it will be difficult to maintain such a narrow view in the long term. The more extensive the potential knowledge basis, the more complex the decision, and the smarter the AI, the less fitting the image of a supposedly clear division of labour between the doctor as decision-maker and the machine as a mere supporting system will appear-and the more urgent is the need to adjust the legal point of view and broaden the perspective on the interaction of "man" and "machine" or doctor and AI.

Algorithms and standard-setting
The latter question of the interaction of doctor and AI is, in fact, not all that new, but relates to the well-known question of medical standard-setting, i.e. the question of how to define the medical standard which is owned by doctors lege artis and for which they are held liable.

Practical experience versus scientific knowledge: the "machine medicine" accusation
The development of the medical standard has always been dynamic-and determined by the challenge of how medical standard-setting should react to the increasing complexity and knowledge explosion in medicine. Typically, the required standard of treatment is defined as "what is accepted within the profession according to medical-scientific knowledge and/or practical medical experience". 10 This definition reflects the fundamental tension between so-called practical medical experience, which stands for the primacy of therapeutic freedom and the doctor as key actor on the one hand, and so-called medical-scientific knowledge, which stands for standardsetting on the basis of external knowledge such as the medical guidelines mentioned above, on the other. Over time, the balance between the two has tended to shift towards medicalscientific knowledge at the expense of practical medical experience. In particular the establishment of medical guidelines has promoted an increasing scientification of standard-setting, which from the outset has been the subject of controversy and sometimes met with strong resistance, as it was not perceived as a "boost of professionalisation" but the "dissolution of medical autonomy". Critics have referred (and still do so) to "cookbook medicine" and an "expertocracy" that ignores the individuality of the patient and attempts to degrade medical treatment to a purely technical undertaking. 11 What is so illuminating about this debate on the introduction of guidelines into medical practice is that the advance of AI in medicine meets with the same reser- 9 On the fundamental differences between AI and robotics: Naveen [10]. 10 On German medical law cf. Carstensen [2], Hart [4], Ertl [6]. 11 Cf. Kienzle [9, p. 93] for detailed references to this discussion.
vations. The criticism of guideline medicine described here is mirrored by the accusations directed at AI medicine; "cookbook medicine" is replaced by "machine medicine", the rise of expertocracy is evoked and there are fears that medicine is increasingly becoming a purely technical undertaking.
Medical law, however, is not concerned with the human or technocratic character of standard-setting, but only with how to make the right treatment decision. If and insofar as AI decisions are superior to the average competence of a doctor, this circumstance has to be taken into account by standard-setting, i.e. standardsetting must be-and is-open to innovation. Just as doctors are not free to ignore an evidence-based (S3) guideline in the conviction that the stipulations of so-called theoreticians do not make sufficient allowance for their practical medical experience, they will not be able to disregard an intelligent algorithm in future simply because they feel that IT systems restrict their therapeutic freedom.

AI-based algorithms as medical products
From the legal point of view, the decisive question is thus simply if and under which conditions an AI-based algorithm can set the standard to the same extent as is the case with evidence-based medical guidelines. Due to the complexity and impenetrability of algorithms, this decision, however, cannot be left to the individual physician in the individual treatment situation, but has to be made at a preceding safety regulation level-in the case of AI-based algorithms at the level of medical devices law. AI-based algorithms influencing therapeutic decisions have to be classified as medical devices-at least as class IIa devices-and as such are subject to the Medical Devices Regulation (MDR). The use of algorithms in medical practice is therefore only permitted if they comply with the applicable safety and performance requirements (Art. 5 para. 2 MDR). They have to be certified and undergo a conformity assessment procedure, 12 in the framework of which a so-called Notified Body verifies that conformity with the relevant basic safety and performance requirements has been sufficiently proved (Annex IX-XI MDR).
Such a certification in accordance with the MDR does not comprehensively "anticipate" the doctor's treatment decision [3]. However, as the German Federal Court of Justice once stated with respect to the approval of drugs, it gives rise to the "presumption" that a medical device may be used in the actual therapy and thus relieves the doctor of the duty of verification. 13 He or she is not required to assess the general safety of an algorithm but can assume that the manufacturer has complied with the safety and performance requirements of the MDR and that this has been verified by the Notified Body.

Liability for algorithms
If it is assumed that the manufacturer and the Notified Body are responsible for the safety and performance of an AI product, the treating doctor cannot be held liable for a wrong treatment decision caused by a faulty algorithm such as the "discriminating" algorithm referred to earlier on. This applies at least in cases where the faulty algorithm has been certified and has been used within the scope of its intended purpose. Physicians are entitled to rely on the certification-in fact they must be able to rely on it since this is the only practicable way in view of the complexity of algorithms. 14

No operator liability
All approaches aimed at so-called operator liability for AI should therefore be rejected, at least where medical liability is concerned. However, such operator liability has been proposed by the European Parliament (EP) in its resolution on a civil liability regime for AI of October 2020. 15 The EP envisages a regulatory regime for AI where liability is first of all imposed on the operator of an AI system. This would include the doctor as a so-called front-end operator of AI. Furthermore, insofar as AI systems in medicine are associated with a high inherent risk, doctors would even be subject to strict liability. 16 From the EP's perspective, the primary responsibility of the operator of an AI system is justified by the fact that he or she "is controlling a risk associated with the AI-system, comparable to the owner of a car". In addition, in view of the complexity and connectivity of the system, the operator is seen as the "first visible contact point" for the person concerned. 17 For the above-named reasons, such considerations do not fit the specific constellation of medical liability. In particular, the parallel drawn by the Parliament to the use of a motor vehicle is not appropriate: if doctors use a-certified-algorithm within the scope of its intended purpose, they do not open up risks on their own accord but fulfil their medical obligation of treatment in accordance with the rules of medical care, and it could hardly be justified if this performance of duties were subject to strict operator liability.
For the same reason, all other approaches that aim to extend the physician's liability to the safety and performance of AI-based algorithms, for example by drawing parallels to animal owner liability or vehicle holder liability, or by classifying AI as a kind of "digital vicarious agent" whose errors would be attributable to the doctor [1], must therefore also be rejected. Any extension of the doctor's scope of responsibility in the direction of product liability fails to take into account the legal division between product-related safety responsibility on the one hand and treatment-related medical responsibility on the other and is therefore not appropriate.
14 Only the use of non-certified software by the physician gives rise to liability: Helle [5, p. 998

Primary liability of manufacturer and Notified body
It is not the doctor who is responsible for the performance and safety profile of an AI-based algorithm, but rather the two main actors involved in placing these medical devices on the market, i.e. the manufacturer and the Notified Body: Even in the times of AI, the manufacturer is still the central figure where liability is concerned, now with respect to liability for AI-based algorithms. The manufacturer is the one who determines or programmes the decision-making processes of these algorithms in the first place and thus the one who is responsible according to product liability law. Although product liability law defines products as "all movables" 18 -which would not include the algorithm as mere software-, de lege lata it has already been argued with good reason that at least an analogous application would fall into this category, and de lege ferenda there is wide agreement that product liability law requires further clarification in this respect [5]. Another possibility would be the liability of the Notified Body, in view of the fact that this body is subject to an extensive catalogue of obligations within the framework of the conformity assessment procedure under medical devices law. In a decision from 2020 relating to the liability for defective breast implants of a French manufacturer of medical devices, the German Federal Court of Justice explicitly stated that, under medical devices law, the "protection of the final recipients of medical devices ... must be ensured not only by the manufacturer, but also by the notified body". An overall assessment would show that the regulations on the conformity assessment procedure with the involvement of a Notified Body focused on the individual protection of each patient. 19

Fault-based liability of the operator
In the case of AI-based algorithms as medical devices, we therefore have a distribution of liability which is primarily attributed to the manufacturer and the Notified Body. Conversely, however, this does not automatically imply that physicians-as "operators"-can never be held accountable in the event of faulty AI. Particularly, if the decisions of the algorithm used contradict the clinical experience of the treating physicians, these physicians have the duty-a treatment-related duty-to refrain from the use of such algorithms if there are reasonable doubts. However, this obligation is not based on any form of operator liability for a defective product, but on the classic-action-related-liability of doctors violating their medical duty of care, specifically because they could have recognized on the basis of their medical experience that the therapy decision taken was not lege artis [5]. Insofar as practical medical experience is still important as a standard-setting criterion, it may lead to liability in the complex system of responsibilities that characterises AI-based medicine-however, not in the sense of product responsibility. 18 Council Directive 85/374/EEC of 25 July 1985 on the approximation of the laws, regulations and administrative provisions of the Member States concerning liability for defective products, Art. 2. 19 Federal Court of Justice (Bundesgerichtshof), NJW (2020) 1514, 1518.

Conclusion
There is thus a differentiated liability regime with clear and graduated responsibilities for the use of artificial intelligence (AI) in medicine. The preventive function of this liability regime is essential in order to ensure that AI does not prematurely enter everyday practice under the frequently used label of "medical and technological progress". There are more than enough examples of apparent innovations in medicine-be they new treatment methods, new drugs or new medical devices-where the expected benefits subsequently turned out to be much lower than the associated risks. A preventive liability regime with clearly defined responsibilities has an important "braking function" in this respect, especially in the case of AI-based medicine, to ensure that a supposedly intelligent algorithm does not subsequently turn out to be quite stupid or stupidly programmed and thus undeserving of the title "Doctor Algorithm" in the first place.