1 Introduction: how artificial intelligence (AI) can help in medical decisions about telemedicine?

Many health care procedures, such as inpatient clinical information systems, antibiotics prescription, and risk assessment of pressure ulcers, are supported by computer aided decision making Almpani, [1])AI-based telephone emergency triage assessment systems are an example of how AI can support medical decisions, including the decision on whether to use telemedicine. An example is the system developed in Germany to facilitate the structured initial medical assessment procedure in the emergency department called "Strukturiertes medizinisches Ersteinschätzungs Verfahren für Deutschland" (SmED). The programme is used in the initial AI-assisted telephone assessment of emergencies by specifically trained staff and supports ambulance crews in deciding whether to send patients to an emergency department or to refer them to general care consultation (Graf von Stillfried ; Graf von Stillfried, Czihal, Meer [2, 3]).

This programme is inspired by the SMASS software already implemented in Switzerland and uses an AI system to assist in making decisions based on urgency and the appropriate level of care. The effectiveness of the system in optimising processes and improving the allocation of resources has been proven. The study by the Institut für Sozial- und Präventivmedizin (ISPM) of the University of Bern, analysed the system and revealed that 91% (153) of the consultations assigned with the system were considered correct, 5% (10) were considered early and 4% (7) late. More than 90% of the doctors surveyed rated both the consultation and the timing as fair (Meer, Simonin, Trapp, Niemann, Abel, [4]).

The software operations consist of a structured survey based on the 85 most common complaints of the International Classification of Primary Care (ICPC-2) and contains thematic lists of specific questions to describe the pattern of complaints that the software can process. The first questions are intended to determine whether there is an indication of urgency; if the answer is yes, the call is immediately transferred to the emergency telephone number (112). If the urgency is ruled out, the questions focus on the main complaint and the urgency levels are categorised in colours according to the level of urgency (red, orange, yellow-green, dark green). There are four levels of urgency: (1) Emergency, (2) Medical treatment as soon as possible, 3) Medical treatment within 24 h, and 4) Medical treatment not needed within 24 h, and four levels of care: (1) Emergency services (112), (2) Hospital emergency department, (3) Practising physician/on-call medical service, and 4) Medical teleconsultation.

The algorithm used by the AI system is based on a deep and machine learning system (ML) based on a neural network structure and allows for an almost infinite number of possible question–answer constellations. The recommendation of a level of urgency or level of care results from the respective weighting factors of the question–answer constellations, so that different combinations of complaints and risk factors may result in different recommendations (Graf von Stillfried, Czihal, Meer [5]). In addition, according to Meer (2019) two SmED developments have recently been driven [6]: (a) the use of natural language processing AI to automate the telephone dialogue with the patient to a recommendation that could be further communicated by a healthcare professional and (b) the implementation of SmED as a digital self-assessment via a chatbot with the subsequent option to find a suitable consultation and immediately arrange an appointment.

There is currently a discussion about whether ML-based initial assessment tools could also be effective in managing non-urgent patient demand (against Slagman, Greiner, Searle, et al. [7], in support Wang et al. [8]). They could be used as a back-up tool for decision-making on whether medical care can be provided by remote treatment (teleconsultation). Based on the data provided by the patients, the machine learning system will make a prediction of those lower risk cases whose non-urgent care can be provided remotely, and will distinguish them from those cases which, although not urgent, require a physical consultation. The increasing development of pilot programmes of ML (AI)-based triage tools for non-emergency triage is a clear indicator for its future use in supporting the decision on the use of telemedicine.

2 Why artificial intelligence is being implemented in telemedicine decisions?

In most of European countries, the implementation of telemedicine has not reached its full potential. Indeed, there is a greater tendency to promote the development of telemedicine in its most instrumental function, such as the use of teleconsultation between experts or telemonitoring, to the detriment of the implementation of its maximum expression, which is exclusively remote treatment (Thiel, Deimel 2020 [9]). The telemedicine is currently limited to its most simplified manifestation as a support tool for traditional medical advice and treatment. The question we pose is why? The doctor’s decision as to when a remote medical assistance can be carried out is very complex and therefore difficult to implement.

For example, countries such as Germany admits the exclusively remote medical treatment in certain cases. The amended § 7.4. of the Berufsordnung für die in Deutschland tätigen Ärztinnen und Ärzte (MBO-Ä) or professional code of medical practice, provides for the use of exclusively remote treatment by stating that "is permitted in individual cases if it is medically justified”. In which individual cases is the exclusively remote treatment medically justified? To determine the cases, the doctrine proposes the remission to norms that, in comparable cases, provide us with interpretation criteria (Siglmüller, Herkenrath [10, 11]).

For example, the German Medicines Marketing Act (Gesetz über den Verkehr mit Arzneimitteln (Arzneimittelgesetz—AMG) in relation to human clinical trials proposes criteria to identify when there is medical justification and this condition is given when “the foreseeable risks and inconveniences in comparison with the benefit for the person to whom it is to be performed (interested party) and the foreseeable importance of the medicinal product are medically justifiable”.Footnote 1 The criterion is based on a comparison between opportunity and risk (Rehmann [12]). If this reasoning is applied to exclusively remote treatment, it would be medically justified when the risk/benefit of its practice is assessed, and this equation results in a positive balance for the patient. Moreover, remote-only treatment will only be performed in individually assessed cases, which leads to consider its possible application in emergency situations, but also in those cases where the physician assesses its appropriateness according to the risk/benefit and the clinical picture and situation of the patient.

In other words, remote treatment could be used in those cases in which its implementation implies a lower risk because we are dealing with minor illnesses (e.g., a common cold, flu). The complexity then lies in medically distinguishing those cases of lower risk or minor illnesses from those that require urgent health care. How to distinguish those cases in which treatment can be carried out exclusively remotely? As we will analyses below, artificial intelligence plays an important role in answering this question.

3 Artificial intelligence system as a decision support for remote treatment in telemedicine: liability in case of mistakes

3.1 AI software used for medical triage under the new European AI framework

The use of ML-based AI systems as support tools in medical decision making in relation to the urgency and mode of patient care raises the legal question of who will be responsible for the damages caused by the decision taken. Obviously, to establish a priority of care, there must be a prior assessment of the patient and a presumptive diagnosis.. A medical decision based on clinically inappropriate AI system predictions regarding the order and mode of patient care could lead to damages, including the patient's worsening or even death. This part of the paper deals with the question whether the liability claim in case of a mistake of the AI-system can be based on the two recent Proposals for Directives: the Proposal for a Directive of the European Parliament and of the Council on adapting non-contractual civil liability rules to artificial intelligence [AI Liability Directive-COM (2022) 496 final] and the Proposal for a Directive on liability for defective products [COM(2022) 493 final]. We will first analyse how the AI system is qualified according to these proposals.

AI software used for the purpose of medical triage is qualified as a high-risk artificial intelligence system (according to art.3 and 6 Proposal of Artificial Intelligence Act)Footnote 2 which means that it must submit to an evaluation to check whether the requirements of Title III of the Artificial Intelligence Act are satisfied. AI software used in medical triage is also qualified as a medical device (art. 2.1. EU Regulation 2017/745 on medical devices or Medical Devices Regulation or MDR (DOUE L 117/1)) which means that to sell the AI software on the European Union market, like any other medical device, the software must have the CE marking and to obtain the CE marking it must pass a conformity assessment procedure.

In conclusion, once all evaluation processes have been carried out and all requirements have been met, the technical robustness of high-risk AI systems will reduce the risk of system mistakes (Thiermann, Böck [13]). In the following section, we will discuss whether the claim for liability for the damages caused in this case can be based on the two recently approved Proposals for Directives that affect aspects related to the damages that may be caused by the artificial intelligence (AI) systems in.

3.2 Is the physician liable for damages resulting from a clinical recommendation issued by the AI system?

In the event of non-contractual liability for damage caused by a high-risk AI system, the proposed AI Liability Directive provides a rebuttable presumption of causation between the fault of the defendant and the output produced by the AI system or the failure of the AI system to produce an output (Art. 4 AI Liability Directive).

However, the proposed Directive applies to fully automatic decisions and does not cover liability claims where the damage is caused by human assessment followed by a human act or omission, even if the AI system only provided information or advice considered by the relevant human actor. According to the proposed Directive (Recital 15), the clinical decision regarding emergencies or which model of medical care to follow based on the prediction of an AI system is considered a human decision. The initial pre-assessment of the patient is one of the first medical acts in a chain of decisions that the physician makes. The AI system functions as a mere support for the medical decision but does not replace the role of the physician who is the decision-maker and can therefore differ from the recommendation of the IA system (Staudenmayer, Heinrichs, Heinrichs J-H, Rüther [14, 15]). Therefore, tort claims for damage arising from clinical triage decisions based on AI system predictions will be excluded from the application of the proposed AI Liability Directive. The question that arises is whether an automated decision could exist without human intervention, as the automated decision will always be interpreted, implemented or processed by a human (Nišević et al. [16]).

However, if the physician follows the AI recommendation and damage is caused, the question of liability for medical malpractice is to be addressed according to the subjective liability rules in the Member States. In this case, whether the liability is resolved under contractual liability or under non-contractual liability, each Member State should consider what duty of care will the physician be required to observe: Does compliance with the lex artis ad hoc require clinical supervision of the decision of the AI system to ensure that the recommendations are safe and relevant for the patient? According to the following considerations such a high duty of care could not be demanded: 1º) a rule to the effect that human supervision is always required would most probably create an obstacle for the implementation of AI systems for clinical decision support; 2º) this kind of rule would also be hard to apply given the high number of decisions that the AI system can make; 3º) whenever the AI system has been correctly maintained and has been used correctly by the doctor, AI system mistake is a risk that is out of the doctor’s control (unless in case of an obvious mistake). For example, in Germany according to Sect. 630 h BGB (1): “An error is to be presumed to have been committed by the treating party if a general treatment risk has materialised that was fully manageable for the treating party and that led to injury to the life, limb or health of the patient”. As AI system mistake is a risk that is out of the doctor's control, the presumption of error does not apply (Schmidt [17]).However, the assessment of the error will depend on the explainability of the system. The opacity of ML systems poses a challenge. In the medical field, the explainability of AI systems is especially important for the physician to be able to verify decisions and trust the AI system. Therefore in practice, data-driven models that provide explanations for their decision are applied in the field of medical informatics to recommend a treatment, to make a diagnosis (with the help of an expert making the final decision), and image analysis to classify a disease, e.g. using magnetic resonance imaging to classify whether a person has some type of cancer (Vassiliades et al., Prentzas et al. [18], [19]).

3.3 The predictive AI system as a defective product

The question here is whether the victim (a natural person) may also claim damages against the 'economic operators' involved in the production chain of the AI system, following the strict liability regime regulated in the Proposal for a Directive products. The application of this proposal to this case raises the following questions:

  1. 1.

    The qualification as a product of the AI system is admitted by art. 4 of the proposed Directive on defective products.Footnote 3 In this case, we would be dealing with software used for medical purposes, which is qualified as a medical device, in accordance with the MDR (Also in case C-329/16, Snitem).Footnote 4 However, the proposal establishes a very general meaning of the term “software” which raises doubts about the fit of the machine learning AI system. But the proposed Directive, in recital 12, clearly states that software would include “operating systems, firmware, computer programs, applications or AI systems”.

  2. 2.

    Another question is whether we can consider as a defective product a machine learning AI system that gives erroneous advice and whose consequences can compromise people's health. In this case, the dangerousness of the product lies in an inaccurate prediction, therefore in the generation of an intellectual performance, but the prediction results from the parameters that the AI system itself changes according to the learning it performs and therefore we would be dealing with a prediction issued by the AI system autonomously. Contrary to what is established in the Case C-65/20 VI v KRONE,Footnote 5 there is in our case a fundamental difference that may justify a different qualification in the context of product liability for software and printed information. In our case software does not simply convey information but constitutes an entity that can be used for the specific purpose for which it has been designed. The inaccurate medical assessment or prediction/information issued by the AI system constitutes an intrinsic element of the purpose-built system itself and is therefore implicit in its use and could therefore qualify as a defective product.

On the other hand, Article 6(c) of the proposal, in a clear reference to machine learning AI system, recognises as a circumstance of the concept of a defect: the effect on the product of any ability to continue to learn after deployment. Moreover, in this case, the defectiveness of the product could be presumed according to art. 8 and 9.4 of the proposed Directive, given the technical complexity of the evidence, putting the burden of rebutting the presumption on the defendant.Footnote 6

  1. 3.

    Another question is who could claim damages under the proposed Directive. It applies to any natural person who suffers damage caused by a defective product (‘the injured person’) (Art. 5). Furthermore, the connecting factor for defining “Injured Person” in the context of product liability is that the person has suffered damage because of the defective product. In short, the term encompasses both the purchaser of the product, as well as any user of the product who is not the purchaser or also the completely uninvolved third party (bystander) (Junker, 2021 [20]). Another issue is that the proposal limits its application to damage suffered by natural persons (Art. 1) and excludes from the scope of application legal entities, companies and entities that suffer damage caused by defects in the products they use (e.g., hospitals).

  2. 4.

    According to article 7 of the proposal, the manufacturer will be liable (the manufacturer of the product and the manufacturer of its defective components).Footnote 7 However, in case of machine learning based AI systems, the clinical recommendation has been issued by the AI system autonomously, so the question is whether the manufacturer is liable for it. Since continuous learning is part of the system's design, one can consider any action by the system being tacitly authorized by the manufacturer (Dettling [21]).

4 Conclusion

The study of the implementation of artificial intelligence systems as medical decision support, e.g., on the use of telemedicine, poses a relevant challenge in terms of liability. The AI system used for healthcare purposes is subject to a strict evaluation process to check compliance with all safety and marketing requirements given its European qualification as a medical device and as a high-risk AI system. However, in the event of damage due to a prediction error of the AI system, the question arises as to who is liable.

According to Recital 15 of the proposed AI Liability Directive its application is excluded if the AI system is used to support the human decision and the decision is not automated Liability is subjective. The question is then settled in each Member State, in terms of the standard of care that can be required of the physician using an AI system as a tool. The regulatory framework proposed in Europe seems to point to the more peaceful solution that compensation for damages resulting from the failure of the AI system should be settled in accordance with the Proposal for a Directive for defective product. The defectiveness of the product could be presumed according to art. 8 and 9.4 of the proposed Directive, given the technical complexity of the evidence, putting the burden of rebutting the presumption on the defendant.