1 Introduction

Additive manufacturing (commonly known as 3D printing) is already used for a variety of healthcare applications, such as creating 3D anatomical models, medical aids (such as surgical guides, splints, and prostheses), tools and instruments, implants, and bioprinting for creating patient-specific drugs [1, 2]. It is used in dentistry to create implants and personalised orthodontic fixtures for teeth alignment [3]. Attachments for surgical robots may also be 3D printed [4]. 3D printing surgical instruments creates the possibility of producing custom-made tools for specific operations [5]. Medical imaging may also be used with 3D printing to design specific medical devices for the patient [6]. An automated design system has been described for designing 3D printed patient-specific tools for laparoscopic surgery [7].

Creating customised surgical tools that are tailored to individual patients and specific procedures has the potential to reduce the risk of surgical complications [1]. We will call these 3D printed customised surgical tools bespoke surgical tools. Like any new medical technology, a system for designing and producing bespoke surgical tools will introduce benefits and risks. The research question in this paper is whether these risks correspond to the usual risks associated with the development and introduction of new medical technologies, or whether there are risks unique to creating and using bespoke surgical tools that will require additional forms of mitigating risk.

Computational design is an application of artificial intelligence (AI) and machine learning (ML) that solves design problems algorithmically, and can be used either as an aide to a human designer or by fulfilling the role of designer itself. Computational design methods (such as evolutionary algorithms) have been used to design patient-specific snake-like surgical manipulators that attach to existing surgical robots [8]. The purpose of this paper is to better understand how stakeholders see the potential risks of using computational design and 3D printing to produce bespoke surgical tools. We argue that the risks stakeholders identified can be categorised as either being unique to these tools or as established risks of surgical technologies. By examining the unique risks posed using computational design systems in this context, the nature of these risks and potential mitigation measures for the application of bespoke surgical tools are identified.

2 Background

2.1 Risk and medical technology

In contrast to quantitative descriptions of risk as the probability or the expected utility of an event occurring, qualitative descriptions of risk refer to the nature of potential unwanted events, their causes and the consequences of those events for specific stakeholders [9]. These descriptions of risk may be called ‘objective’ and ‘subjective’, respectively [10]. Qualitative analysis of risk also incorporates the relative consequences of risk on different stakeholders, their exposure to that risk, and attitudes and expectations about identified risks and consequences [11].

For a medical technology, unwanted events are those that either prevent it from performing its intended purpose or cause unintended harm to its users or those affected by its use. Medical technology used in surgery will necessarily cause some harm to the patient, as surgical interventions cause intentional, short-term harm to the patient with the goal of treating their condition and providing them with a long-term benefit [12].

The intended purposes of medical technology may be diagnostic (identifying disease or health condition), therapeutic (treating disease or health condition), function enhancing (improving abilities beyond impairment from disease or health condition), enabling (improve abilities and capabilities impaired by disease or health condition), or preventive (reduce risk of disease or health condition) purposes [13]. Bespoke surgical tools are a therapeutic technology, as surgeons would use them to treat existing diseases or health conditions in patients [13]. Unwanted events would be unintended harm to patients caused by the tools, and the tool’s failure to perform as intended by the surgeon. The causes of unwanted events would be errors in using the tool, and characteristics of the tool or of the process necessary to create it that cause these harms to occur. For example, faults in the tool or in the 3D printing process may cause ‘unretrieved device fragments’ to remain in the patient after surgery, potentially causing harm to the patient [14]. In the process necessary to create the tool, the patient may be harmed during the required medical imaging due to excessive radiation or in the physical handling needed to perform the scan [15].

Ethical risk analysis (eRA) is a risk management approach that highlights the impact of unwanted events on agency, interpersonal relationships, and justice, and supplements traditional forms of risk analysis that focus on the probability and severity of such events [16, 17]. eRA uses three primary ‘risk roles’ (beneficiary, decision-maker, and risk-exposed) that may occur singly or in combination to identify differences between the parties connected to risk [16, 17]. Table 1 lists these roles

Table 1 Role-holders used in ethical risk analysis (eRA) (based on [16, 17])
Full size table

The differences between these roles are ethically significant as they describe differences in how benefits and burdens are distributed, and who decides who is exposed to risk and how that will occur. Someone exposed to risk without also being a beneficiary or having any say in taking that risk may lack autonomy over important aspects of their life. In extreme cases, this may even place individuals in the position of being exploited for the benefit of others. The relationship between the decision-maker and the risk-exposed beneficiary may be paternalistic, as the risk-exposed beneficiary is subject to decisions made by others [17, 18]. Those who benefit from risks without also being decision-makers or exposed to those risks may also be cause for ethical concern, as the beneficiary gains from risks being taken by and decided upon by others [17].

We will call situations where decision-makers are distinct from either the beneficiaries or the risk-exposed (or both) ethical risks for those involved. More specifically, an ethical risk is an unwanted event where one party decides whether the risk is taken, who it affects and how it affects them, and those affected by that risk (positively or negatively) are not involved in making that decision. Such risks are ethical concerns as they potentially undermine the autonomy of the beneficiary or the risk-exposed in any given circumstance. Respect for autonomy is a major component of bioethics [19] and in Western ethical theory [20].

We can further illustrate the ethical significance of these roles by examining how they relate to informed consent in a medical setting. For example, giving informed consent makes the patient a decision-maker in accepting the risk of having surgery. Informed consent gives the patient the risk roles of beneficiary, decision-maker, and risk-exposed. Without the surgeon first obtaining the patient’s informed consent, the surgeon would be taking the role of decision-maker for themselves and away from the patient. The latter case would be an example of clinical paternalism [17].

Autonomy is not the only ethical concern represented by ethical risk. Significant ethical concerns are also raised by decision-makers who are beneficiaries of a risk without also being exposed to it [16]. The risk in this case is an externality to the decision-maker and beneficiary. This is potentially concerning due to the possibility that the decision-maker will act recklessly (or even simply without due consideration) as they stand to benefit from the risk without being exposed to it [18].

2.2 Ethical issues with new surgical technology and healthcare AI

Informed consent is one of the core ethical issues involved in introducing new surgical technologies. To make an informed decision, patients need to be aware of the potential risks and benefits of a proposed surgical procedure. However, the risks and benefits of new surgical technologies may be uncertain, and this uncertainty needs to be communicated to the patient [21,22,23,24]. Surgeons and patients may also be biased towards new surgical technologies, which may affect how the risks and uncertainties are communicated and understood [21, 25]. A range of issues related to safety, timing and cost of introducing new technologies are relevant here.

New surgical technologies need to be safe for both surgeons and patients. The regulation of medical devices by regulatory agencies (such as the Food and Drug Administration (FDA) in the US and the Therapeutic Goods Administration (TGA) in Australia) is intended to ensure that new surgical technologies are safe to use [22]. New technologies may also require surgeons to learn new techniques with a learning curve that may affect their competence and increase the risk to patient safety until surgeons have mastered them [21, 26]. Effective training and credentialing of surgeons with new technologies is necessary to mitigate this risk [22, 23].

The timing of adopting a new surgical technology is also significant, as it requires making a judgement about whether respond to the needs of patients who may benefit from a surgeon being an early adopter of the technology or waiting until there is further evidence to support its use [21, 22]. Tracking the outcomes of surgeries performed using the new technology and sharing this information to other surgeons and regulators is important for informing this decision [22, 23].

Finally, the cost of new surgical technologies also should be considered, as the cost of a new technology may result in fewer patients being treated (and treatment being less accessible to patients) than if existing technologies and methods were used [21, 22, 24, 25].

Healthcare applications of AI also raise additional ethical issues [27]. The explainability of medical decisions and diagnoses made with the assistance of AI and of actions performed by AI systems is one concern [28]. The lack of explainability may lead to the AI system being a ‘black box’ to developers, regulators, clinicians, and patients, who may be unable to understand how it produced certain outcomes [29, 30]. The reliability and safety of these decisions is another concern [27]. While these concerns often refer to using AI as a diagnostic medical technology, issues such as explainability and reliability are also relevant to therapeutic healthcare AI applications. It is these ethical issues raised by the applications of AI in healthcare that are our focus.

2.3 Computational design

Broadly speaking, computational design involves the use of some computational tool to assist or augment a design process. These tools may include modelling software (e.g., COMSOL [31], ANSYS [32]), which allows for components to be assessed for performance prior to fabrication, CAD software (e.g., Solidworks [33]), which provides rapid iteration and visualisation of designs on a screen, and purpose-built design software such as Rhino [34], which allows users to manipulate a graph structure to generate a wide variety of designs in a generative manner. In all the above cases, a human is driving the design process, and the tool is providing some form of assistance.

The type of computational design that we focus on herein replaces the human with a piece of software, that similarly drives the design process. This software typically implements a form of machine learning, which removes human oversight from the design process whilst operating in problem spaces that may be too complex or unintuitive for a human to effectively work in. It is this definition of computational design that we will use in this study.

The most common form of computational design optimises the geometry, structure, or distribution of materials of a target component within some pre-defined bounds and belongs to a family of approaches known as shape, structure, and topology optimisation, respectively [35, 36]. These approaches heavily rely on a human designer to set appropriate conditions (loads and forces), and may (for instance) incrementally remove material from a component to make it as lightweight as possible whilst retaining a critical amount of structural strength. Due to the heavy reliance on a human to constrain and direct the optimisation (e.g., to set up the problem), plus the use of gradient-based (heavily directed) optimisation algorithms, the solutions produced tend to be relatively predictable.

Another popular approach uses evolutionary algorithms (EAs). These are stochastic, black-box, gradient-free optimisers that combine local and global search, via digital analogues of genetic mutation (local) and crossover (global) processes. Rather than operating on a pre-defined model with known, constrained starting conditions, EAs manipulate a population of digital genomes (string of numbers), where are then transformed into a candidate solution (representation) and tested. At each iteration (generation), a population of solutions are tested and receive a score from a ‘fitness’ function that measures how well the candidate component fulfils the specified goal [37]. The fittest solutions are preferentially selected to be ‘parents’ to create a new generation of ‘children’ via mutation and crossover, which are assessed for fitness in a similar manner. The process of evaluation, reproduction, and replacement continues until a termination condition is reached, which may be a specified number of generations or a specified amount of time elapsing, or a given fitness level being reached [37]. Classically, the highest fitness solution is then selected as the best ‘optimised’ component.

EAs are particularly useful for computational design, where it has been used to design a variety of industrial and mechanical components, such as wind turbines [38], granular materials for industrial applications [39], and soft robotic actuators [40]. Popular variants of EAs allow for optimal trade-offs between multiple competing objectives to be identified [41], as well as for algorithms that promote diversity, novelty, and surprise, as well as more closely mimicking the open-ended objective-free evolutionary processes seen in nature [42], in place of direct fitness optimisation. Interestingly, these approaches have been shown to passively generate higher fitness solutions than direct fitness optimisation, so are promising for use in design optimisation.

Together with the design freedom afforded by mapping a genotype to a candidate solution, the use of black-box optimisation, and overt promotion of novel and surprising solutions, evolutionary approaches provide a much more free-form designs to be discovered. However, this freedom, together with a general loss of oversight over the design process, engenders heightened ethical concerns from the use of modern EAs for computational design.

Using computational design has similar risks to other practical applications of AI and ML. Amodei et al. [43] describe five categories of AI safety concerns: avoiding negative side effects, avoiding reward hacking, scalable oversight, safe exploration, and robustness to distributional shift. Lehman [44] describes how these concerns appear in applications of EAs. A fitness function may cause negative side effects if the specification of the goal is too narrow, and it does not penalise potential solutions that will cause harm [44]. Reward hacking may occur if unexpected and undesirable solutions are given high evaluations by the fitness function [44]. Scalable oversight requires effectively combining simple fitness functions with human assessments of problem solutions that cannot be easily implemented algorithmically [44]. Safe exploration requires the EA to effectively search the solution space without performing searches that cause the system to fail, and robustness to distributional shift requires EAs to operate effectively in solution spaces that were not used in training the system [44].

Unlike the examples of AI-driven robots mentioned by Amodei et al. [43] and Lehman [44], AI safety in the context of computational design systems concerns the safety of the designs it produces rather than the safety of the actions performed by the system itself. The solutions produced by the AI need to be safe, rather than its actions. For computational design systems to produce safe product designs, the design requirements need to be accurately reflected in the fitness function that evaluates the potential designs it develops as it searches the design space to find an optimal design solution. The design space also needs to be limited so that potentially dangerous potential designs are identified and labelled as unsafe. The constraints and parameters that define the design space and the fitness function are therefore significant in ensuring the safety of the designs produced by the computational design system.

2.4 Medical 3D printing

The process for designing and fabricating bespoke surgical tools would build on existing techniques that are used to develop 3D printed anatomical models based on patient scans [45]. Patient scans are acquired via computed tomography (CT) or magnetic resonance imaging (MRI) scans and converted into DICOMFootnote 1 images for use in 3D visualisation [45]. For 3D printing, the DICOM image is segmented to isolate the area of interest from the rest of the acquired image, and a surface mesh is generated from the segmented image [46]. The mesh is used to create a 3D model as a Standard Tessellation Language (STL) file and post-processed to make it suitable for use by 3D printers [47]. After fabrication, the physical 3D model is post-processed to remove excess material [45]. Bespoke surgical tools would be fabricated using non-biodegradable materials, as they are single-use tools that would be used for a surgical intervention and would not be incorporated into the patient’s body. They would necessarily be single use as they are designed for a single patient and a single operation.

Using computational design also changes the preparation time for surgical uses of 3D printing. Martelli et al. [48] mention the time necessary for data processing and preparing 3D models (rather than 3D printing itself) as the most time-consuming aspects of using 3D printing in surgery. The time necessary for the computational design system to produce the tool design will also affect preparation time.

A systematic review of the advantages and disadvantages of 3D printing in surgery found accuracy to be the second most stated advantage, and the most stated disadvantage [48]. Image artefacts in the patient scan were identified as a source for inaccuracies in 3D models based on the scan [48].

3 Methods

The findings presented here are part of a project examining the responsibilities of stakeholders involved with creating and using bespoke surgical tools with AI, and how potential risks associated with those tools might be identified and mitigated. In-depth stakeholder interviews were used to collect data for this research project. In analysing the complete data set for the project, we identified adoption, process, responsibility, and risk as the four major themes. The adoption theme covered responses identifying barriers and enablers for the acceptance of bespoke surgical tools. The process theme covered responses describing specific details of the creation and use process for bespoke surgical tools. The responsibility theme covered responses relating to how responsibility may be allocated between stakeholders within the creation and use process for bespoke surgical tools and is presented in Douglas et al. [60]. The risk theme, which is the subject of this paper, covered responses relating to the identified perceived risks with creating and using bespoke surgical tools.

The analysis presented here is not intended to replace quantitative risk assessments of using computational design and 3D printing to produce bespoke surgical tools. Rather, our analysis is intended to identify how the risks apparent to relevant stakeholders are seen to distribute the benefits and burdens among those affected by the technology. Such qualitative risks may influence how willing stakeholders are to adopt this technology. While establishing the quantitative risks connected using computational design to produce bespoke surgical tools will be a necessary part of evaluating this technology before it is introduced, the qualitative assessment of risk is important for initially understanding the concerns stakeholders may have with a new technology, and how they perceive its benefits and burdens.

3.1 Participant selection

We used purposive sampling to identify and include a variety of relevant stakeholders in our sample group [49]. We identified an initial set of stakeholder groups through discussions with a research team working to develop bespoke surgical tools using computational design and 3D printing. These stakeholder groups were the designers of computational design systems, the fabricators operating 3D printers, medical insurers, patients, regulators, radiologists, and surgeons.

We identified potential participants from these stakeholder groups via an online search for organisations active in medical 3D printing, surgeons experienced with 3D printing, patient advocacy groups, professional organisations representing surgeons, medical regulatory agencies, and researchers in computational design. We limited our search for participants to Australia so that the experience and expertise of participants was within a common legal and regulatory context. We invited potential participants to take part in this study via email. We also used snowball sampling to identify further participants [49], which led us to include bioethicists as an additional stakeholder group. The experience of bioethicists in dealing with ethical questions about medical technologies make them a relevant group to include in this study, even though they are not stakeholders in creating and using bespoke surgical tools.

We conducted 21 interviews with representatives of stakeholder groups between August and November 2020. The participants, the stakeholder groups they represent, and their relevant experience, are listed in Table 2. No representatives of medical insurers responded to our requests to participate in this research.

Table 2 Interviewed stakeholders
Full size table

3.2 Data collection

During the interview, the interviewer shared the diagram of the creation and use process for bespoke surgical tools with the participant and used a script from the interview guide to verbally describe the process. The process shown in the diagram was based on discussions with a research team working on a computational design system for creating 3D printed bespoke surgical tools. Figure 1 presents this diagram.

Fig. 1
figure 1

Bespoke surgical tool creation and use process diagram shared with participants (reproduced from Douglas et al. [60])

Full size image

The interview questions covered four broad topics: (1) the participant’s current role and their experience with surgical robotics and 3D printing; (2) their comments on the stages of the process for creating and using bespoke surgical tools; (3) how the participant saw their own potential role and the responsibilities of themselves and others involved in the process; and (4) the potential risks in the process and how they might be mitigated.

Informed consent was obtained from each participant before the interview. Interviews were conducted over telephone or video call, and with the participant’s permission, they were recorded and transcribed for analysis. The average interview length was 29 min, and no follow up interviews were conducted. The interview transcripts were anonymised prior to analysis.

For coding the data, two researchers read the transcripts separately, and portions of the transcripts were assigned to codes that were the basis for identifying initial patterns or themes in the data [50]. Following this first round of coding, discussions among the research team were used to identify similarities or discrepancies and refine the framework for analysis. The interviews were coded using NVivo software (released in March 2020) [51] to support a structured, organised and transparent approach to our qualitative data analysis. Our analysis produced 143 unique codes, which were grouped and subsequently formed the basis of four major themes of adoption, process, responsibility, and risk. As noted above, this paper provides a detailed examination of the risk theme only.

4 Findings

Within the risk theme, we sought to determine whether the risks stakeholders identified correspond to known or well-established risks associated with new medical technologies, or whether bespoke surgical tools present new potential risks to be mitigated. To do this, we classified the data within the risk theme in the stakeholder interviews as either established risks that are known to exist for other medical technologies, or risks that are unique to bespoke surgical tools. Distinguishing between the unique risks posed by an emerging technology and the established risks of existing technologies is a useful means of highlighting where the greatest areas of uncertainty are around the application of a novel technology. Determining which risks are unique to an emerging technology also provides a clearer picture of where new means of risk mitigation may need to be implemented if the technology is adopted. Where the emerging technology shares common risks with existing technologies, the current means of risk mitigation may be examined to determine how they may be applied to the new technology.

We identified three categories within the established risk subtheme: 3D printing, medical imaging, and new medical technology. The risks from 3D printing relate to the 3D printed tool’s strength, the 3D printing method’s accuracy in producing the tool design, and using 3D printed tools in surgery. Medical imaging risks relate to the potential harms to the patient from performing the necessary scans, and potential errors in the scan that may affect the surgical tool’s design. Finally, the risks of new medical technologies relate to factors that influence the decision to use new technologies in surgery, and how using a new technology may affect surgical outcomes. Under the subtheme of unique risks of bespoke surgical tools, we identified two risk categories: risks posed using the computational design system itself, and risks arising from the process of creating these tools. The risks posed by the computational design system relate to how the system interprets the patient scan and generates a tool design based on it. The risks from the process of creating these tools relate to the potential interactions between individual process stages, and the risks posed by the process itself. Figure 2 illustrates the relationship between these subthemes and lists the risks and mitigations in each category.

Fig. 2
figure 2

Risk subthemes and categories identified in stakeholder interviews

Full size image

4.1 Established risks

4.1.1 3D printing

Five participants (Fabricators 1 and 5, Designers 1 and 3, and Radiologist 1) discussed risks associated with medical applications of 3D printing. These risks include the durability of 3D printed surgical tools, and that the fabrication process may produce an inaccurate recreation of the 3D model, introducing errors that are not present in the design. Fabricator 1 noted that the 3D printed tool will also require post-processing to be usable. Fabricator 5 mentioned that the method of sterilisation may also affect the strength of the tool.

Radiologist 1 used the example of implants 3D printed on site to explain that these risks may be mitigated by maintaining quality control measures to ensure that the tool produced accurately reflects the intended design.

4.1.2 Medical imaging

Seven participants (Designers 1 and 2, Fabricator 2, Regulator 1, and Surgeons 1, 2 and 3) raised risks relating to the medical imaging of the patient. Surgeon 3 mentioned that there may be errors in acquiring the patient images. Designer 1 noted that the patient’s condition may also create difficulties for performing the scan, which may affect the scan’s quality. Fabricator 2 and Surgeon 2 also noted that errors in the patient data (such as misinterpreting patient details or issues with the scan) may pose a risk. Such errors in the data used as input for the computational design system may result in an unusable surgical tool.

Designer 2 noted that “noise” in the patient scans creates difficulties for the processing needed to create a 3D model, as there is the possibility that the scan produced may contain distortions in the final image of the form or element being scanned, and therefore be inaccurate. A surgeon described this risk as an example of “garbage in, garbage out” (Surgeon 1) as automated methods of generating 3D models from patient scans may be unable to identify and recognise the differences between adjacent structures within the scan.

The scan itself also has potential risks for the patient. Regulator 1 observed that if additional scans were necessary for the process of creating a bespoke surgical tool, depending on the scanning method used, it may involve an additional exposure to radiation for the patient.

4.1.3 New medical technology

Eight participants (all three bioethicists, the patient advocacy organisation, Regulator 1, and Surgeons 1, 2 and 4) noted several risks relating to adopting new medical technologies. Bioethicists 2 and 3 noted the potential for conflicts of interest to exist if the surgeons using bespoke surgical tools were also involved in developing the process for creating them and owned intellectual property relating to it.

The costs of new medical technologies were identified as another risk to their potential adoption. Surgeon 4 noted that the costs of new technologies may be significantly higher than existing alternatives with patient outcomes remaining comparable. Surgeon 1 observed that the costs associated with the intellectual property contained within a new technology could potentially have an impact if one of the parties involved in developing the technology is unwilling to share their intellectual property or imposes a significant licensing fee for it.

Bioethicists 1 and 3 noted the potential novelty bias by patients in favour of new medical technologies. This novelty bias can be a form of “the technological imperative” (Bioethicist 1) that assumes that new technologies are inherently better than existing ones. Bioethicist 2 observed that novelty bias may also extend to the surgeon. In contrast, one surgeon argued that there is a strong status quo bias against new methods and technologies within surgery: “surgeons feel safe doing things that they’ve been doing for a long time” (Surgeon 1). This preference for existing tools and techniques is partly explained by the difficulty of mastering surgical procedures: “it may only take 10 patients to get up to proficiency or it might take 50 to 100 patients before you were really proficient at doing the new procedure” (Bioethicist 2).

Five participants (Bioethicists 2 and 3, Surgeons 2 and 4, and the patient advocacy organisation) highlighted the importance of the patient’s informed consent for the surgery to take place. Surgeon 2 and the patient advocacy organisation stated that the patient should be informed of the risks and benefits of bespoke surgical tools, what alternatives are available, and how the cost of using bespoke surgical tools compares to the costs of the standard procedure. This surgeon also explained that the patient needs to be informed that this is not a routine procedure, and to accept that using a bespoke surgical tool may be “the cause of it going well, or the cause of it going poorly” (Surgeon 2). Surgeon 4 noted that during the early trials of the creation and use process, the patient also needs to be clearly informed that the bespoke surgical tool is experimental, and that there may be unforeseen problems as well as potential benefits.

Bioethicist 3 discussed several potential risks with the surgical use of new medical technologies such as bespoke tools, such as the possibility of unexpected changes in the surgical procedure due to using the bespoke tool. The potential changes mentioned were changes to the roles of the surgical team (such as who would be present in the operating room, and who would be selecting and handling the tools during the operation), the duration of the procedure, and subsequently the amount of time the patient spends under anaesthetic. The bioethicist also noted that these changes should be recorded so that other users of bespoke surgical tools can take these changes into account. Another potential risk raised by this bioethicist was that the tool may not perform as expected during surgery.

Bioethicist 3 also noted the potential risk of reverting to using existing methods and tools if there was a problem with the bespoke tool during the surgical procedure. Another participant (Regulator 1) mentioned the possibility that the changes in the bespoke tool compared to the regular surgical tool may make the tool more difficult for the surgeon to use. The impact of the design changes may not be apparent until it is used in clinical trials.

4.2 Unique risks

4.2.1 Computational design

Six participants (Bioethicist 1, Designers 1, 3 and 4, Fabricator 2, and Regulator 1) discussed risks relating to computational design. They observed that assumptions about the intended output of the computational design can affect the quality of the system’s designs. As Designers 3 and 4 noted, the parameters used in a computational design system determine the characteristics of the bespoke tool designs it produces. There may also be limitations in the computational design system’s ability to represent the tolerances and margins for error associated with the 3D printing process. Designer 1 noted that the tool design produced by the system also needs to be verified (by testing the 3D model geometry of the design) so that it can be 3D printed.

Another identified risk relates to the potential unpredictability of computational design. Fabricator 2 noted that the unpredictability of computational design has the potential for users to produce tool designs that are unfit for purpose without realising it. As one designer described it:

let’s say I’ve developed … an algorithm and I’ve run it and the algorithm has a million different possibilities as output solutions and it’s comprised from 1000 various parameters. Now, the combination of these parameters when combined in a specific way will give you a specific output. You change the combination; you get another output and so on and so forth. For let’s say the first 500,000 outputs, everything looks fine. But if one parameter from this list of 1000 parameters, if combined in a specific order with the other parameters might give you a completely crappy result, one that you didn’t see, didn’t imagine, was lost and buried in the system and they’re all in the realm of possible solutions. My fear always is that when does that one parameter emerge or evolve or mutate in the algorithm and can you catch it? (Designer 3)

Despite this potential risk, Designer 3 argued that the inherent unpredictability of computational design is also a motivation for using this technology. Unexpected designs offer the possibility of creating solutions to design problems and tool optimisations that human designers were unlikely to consider. However, the designer noted that unexpected designs may also be the result of user error. Because of this, it is important to be able to confirm whether such designs are an unexpected solution to a design problem or the result of user error.

The complexity of the computational design system itself poses a potential risk. For example, the computational design system was described as a “black box” by two participants (Bioethicist 1 and Fabricator 2). Designer 3 noted that the computational design system is “inherently complex”. Regulator 1 observed that this complexity will make it difficult to verify and validate that the system will always work as intended. While designers were among those stakeholders who commented on the complexity of computational design systems, they were also likely to be the only stakeholders with expert knowledge of how those systems were developed and operated.

4.2.2 Tool creation process

Six participants (Bioethicist 2, Fabricator 5, Designers 1 and 3, Regulator 1, and Surgeon 1) mentioned unique risks posed by the process for creating and using bespoke surgical tools. Surgeon 1 noted how errors at individual stages of the process (scanning the patient, creating the optimised tool design using the computational design system, and fabricating the tool using 3D printing) may significantly affect later stages performed by other stakeholders, and ultimately affect the outcome of the process itself:

The risk at every step of the workflow relates to the end-product not being fit for purpose. There can be computational errors all the way along, and some of those computational errors may actually be human error … That's relevant, especially when you add multiple steps to your workflow because errors compound (Surgeon 1).

The interconnection between process stages also creates potential risks. Designer 3 stated that the computational design system designer should fully understand the input from the patient scans, and that those reviewing the produced designs should understand how the system works so that they can identify likely design faults. Bioethicist 2 also noted the risks of transmitting data between each process stage.

Fabricator 5 described the difficulty of performing clinical trials with surgical tools that are individually created for specific patients. Three participants (Designer 1, Fabricator 5, and Regulator 1) raised similar concerns about the difficulty of testing bespoke surgical tools before use. Regulator 1 observed that testing a tool for a day would be practical for a tool used to treat a patient with a life-threatening condition, but that time-consuming testing would be impractical if the bespoke tool had only minor impact on the surgery, such as making it shorter. However, this regulator elaborated on how data collection may reduce the need for testing individual tools:

...the more often that you did it the more data you'd accumulate … Because if you've modified these tools for, say 100 surgeons, you're then going to have a very good idea of whether the process is very tolerant of difference or whether it's something that is intolerant. Where you might be doing testing initially on every single tool that you manufacture, after a while and when you get a feel for if there are problems or not, you might find you only have to test one in 10 or one in 20 or one in 100 (Regulator 1).

5 Discussion and implications

The inclusion of computational design distinguishes the process of creating bespoke surgical tools from other surgical uses of 3D printing. Introducing computational design into the process expands the complex coordination and large number of necessary for surgical 3D printing [48]. The computational design system itself introduces new coordination issues, as patient scans need to be transferred to it as input, and the tool design it creates as output needs to be transferred to the 3D printer. The surgeon and radiologist may also want to check the tool design before it is fabricated, which adds further coordination problems to the process. The computational designers of the system are also stakeholders who would not appear in other surgical uses of 3D printing.

We applied the ethics risk assessment (eRA) framework to examine the findings relating to the established and unique risks of bespoke surgical tools [17]. In doing this, we also distinguish between technical and ethical risks. Technical risks are specific sources of unwanted events within a technology. Technologies correspond to the subthemes (as the subthemes relate to technologies as sources of unwanted events, such as 3D printing, medical imaging, and computational design), and technical risks correspond to the categories we identified in the participants’ responses (such as the durability of 3D printed tools, distortions in medical images, and the black-box nature of computational design). Ethical risks arise from these technical risks where beneficiaries of that risk or those exposed to the risk are not also decision-makers for that risk. This means that a technical risk may not necessarily be an ethical risk for all the stakeholders connected to it. However, stakeholders who are not beneficiaries, decision-makers, and risk exposed are at ethical risk from a technical risk.

5.1 Ethical risks of 3D printing, medical imaging, and new medical technology

Table 3 lists the established sources of risk of 3D printing, medical imaging, and new medical technology, the stakeholders who are beneficiaries, decision-makers, and risk exposed for these risks, and which stakeholders are at ethical risk from each technical risk.

Table 3 Risk roles for established risks
Full size table

Based on the eRA, fabricators are decision-makers for the technical risks of 3D printing, and both patients and surgeons are beneficiaries and exposed to these risks. Fabricators are beneficiaries of the quality control measures for 3D printing. Patients and surgeons are at ethical risk from all the technical risks associated with 3D printing, as they are not the decision-maker for any of them. Fabricators are at ethical risk from the technical risks of durability, printing accuracy, and sterilisation, as their decisions will affect patients and surgeons rather than themselves.

Radiologists are decision-makers for all technical risks relating to medical imaging, while patients are at ethical risk from each of these technical risks. Radiologists are at ethical risk from image acquisition and patient harm from performing the scan. Designers are exposed to ethical risk from image distortion as this distortion would affect the computational design system. Surgeons are at ethical risk from data error and image distortion, as these technical risks would affect the quality of the tool the surgeon would use in surgery. Existing quality assurance methods for 3D printing and medical imaging may be used to address these risks [52].

Surgeons are exposed to the technical risks of new medical technology and are also decision-makers for these risks (except for the risks from usability of the technology). Surgeons are also beneficiaries for most of the risks as they would benefit from using the technology. Surgeons are at ethical risk from conflicts of interest, cost, usability, and informed consent, as they are either risk-exposed (for conflicts of interest), not a beneficiary of the risk (cost), or not a decision-maker (usability and informed consent). The risk of conflicts of interest may be addressed by the surgeon disclosing their potentially conflicting interests, by recusing themselves from the process of gaining informed consent from the patient, or by seeking second opinions from independent third parties [53].

Patients are exposed to the risks of conflicts of interest, cost, novelty bias, status quo bias, changes in the surgical procedure, and reverting to existing surgical methods if the new medical technology is ineffective. Patients are decision-makers for novelty bias, as they may be convinced to accept a new medical technology due to the belief that new technologies are necessarily better than older alternatives. Patients are also beneficiaries of the risks of novelty bias, status quo bias, changes in surgical procedure, and reverting to existing methods. However, patients are beneficiaries and decision-makers for the risk mitigation method of giving informed consent for new medical technology to be used. As mentioned earlier, the uncertainties around the risks of new medical technologies need to be described to patients for them to give informed consent [21, 25].

The other stakeholders with risk roles related to new medical technology are hospitals and designers. Hospitals are decision-makers about the cost of medical technologies and exposed to the risks of having to fund these costs. Cost is an ethical risk for hospitals as they are not beneficiaries for this technical risk. Surgeons and patients are exposed to the risk that costs will make the technology inaccessible to them. Designers are beneficiaries of the cost of new medical technologies (as they would be paid for the system) and are decision-makers about the usability of the technology, as this is a feature of the computational design system that they control. Both cost and usability are ethical risks for designers as they are not exposed to these risks (and are not beneficiaries of usability).

5.2 Ethical risks of computational design

Table 4 lists the stakeholders with risk roles for the unique technical risks of bespoke surgical tools, and the stakeholders at ethical risk for each technical risk.

Table 4 Risk roles for the unique risks of bespoke surgical tools
Full size table

Computational design has four associated technical risks: the parameters defining the limits of potential tool designs, the representation limitations that may prevent patient scans from being accurately interpreted, the unpredictability of the tool designs it produces, and the black-box nature of the computational design system.

The stakeholders with technical risk roles relating to computational design are designers, fabricators, patients, radiologists, and surgeons. The designer is the sole decision-maker for both the technical and ethical risks connected to aspects of the computational design system as their choices in developing it affects how these risks might be expressed. The designer is also the beneficiary of the technical risk of computational design if they produce more effective tool designs, and is exposed to the technical and ethical risks that the computational design system does not produce suitable tool designs. For example, the designer benefits from the system being a black box if it produces better optimised tool designs than alternative methods of implementing computational design. As the designer implements the system, they are the decision-makers about the system’s characteristics (such as the parameters defining the design constraints for tool designs) that impose risks on others. The designer is the decision-maker for the risks that fabricators, patients, radiologists, and surgeons may benefit or suffer from [16].

Fabricators, patients, and surgeons are at ethical risk from all the technical risks of computational design. The radiologist responsible for performing the patient scans used as input for the computational design system is also exposed to the ethical risk of representation limitations in the system affecting the interpretation of the scans. The fabricator is at ethical risk from the technical risks of computation design as they will affect the success of 3D printing the tool and they do not make decisions about them. The relationship between the designer and these stakeholders might be described as paternalistic in that the designer essentially restricts or changes the level of freedom and/or responsibilities of other stakeholders through applying their own expertise to developing the computational design system.

Patients and surgeons benefit from the unpredictability of the computational design system as it may create an optimised tool design that minimises unnecessary harm that human designers would be unlikely to create. This is the same benefit patients and surgeons gain from a black-box system. Surgeons are exposed to the risk that the tool designed by the system is unsuitable for their use, and patients are exposed to the risk that the tool will be ineffective and lead to an unsuccessful surgical operation or causes greater unnecessary harm to them than a regular surgical tool.

The technical risks of computational design (parameters, representation limitations, unpredictability, and the black box) are aspects of its irreducible unpredictability. As one of the interviewed designers noted, however, this unpredictability is also the justification for using computational design. It reflects the “surprising creativity of digital evolution” [54] that sometimes produces unexpected solutions to the design problems the system is designed for. Evolutionary algorithms used in computational design may produce unexpected solutions through revealing flaws in the fitness function specification, “unintended debugging” where the produced designs exploit previously unknown flaws in the system’s hardware or software, and exceeding designer expectations of possible designs from the system [54]. These unexpected solutions represent the AI safety risks of negative effects and reward hacking [44].

Careful algorithm design may mitigate some of the risk from this unpredictability. The parameters and constraints defining the design space should impose limits on potential designs that prevent potentially dangerous or ineffective tools from being designed. These parameters and constraints may be set using the expected range of dimensions for the bespoke surgical tool to be effective. The part of the patient the tool will interact with will necessarily be within a limited range of sizes. The computational design system could be designed to check the input data to confirm that the relevant dimensions of the patient’s anatomy fall within the expected ranges. The computational system designer is the decision-maker able to implement these features to reduce the risk exposure of patients and surgeons. Radiologists or surgeons who review the patient data and medical images may also confirm that the patient’s anatomy meets the expected range of dimensions that the computational design system can design for. For these methods of mitigating risk to be effective, the designer must clearly specify the range of dimensions that the system can be trusted to design for, and the surgeon or radiologist should be aware of these limitations and how it may affect their own actions and/or decision-making.

The technical risk of the computational design system being a black box reflects broader concerns about AI systems as black boxes in healthcare [30]. However, the concerns about transparency and opacity are different in this case compared with other uses of AI in healthcare. Many AI healthcare applications deal with patient diagnosis. Opaque or black-box AI systems would prevent clinicians from understanding and verifying how the system arrived at the diagnosis or recommended a course of action [30]. The black box of computational design refers to the potential opaqueness of why the system produced a design with the characteristics that it did. The design characteristics may be due to errors in the input data, errors in the parameters and design space definitions, or in the fitness function used to evaluate possible designs within the system. The ethical risks of using computational design to create bespoke surgical tools are concerned with the safety of the designs it creates, rather than the safety of the actions performed by the system. Even through bespoke surgical tools themselves may be used as attachments for surgical robots, the safety risks of using computational design systems have more in common with the use of AI for medical diagnosis and clinical decision support rather than the safety risks of surgical robots.

5.3 Ethical risks in the tool creation process

The technical risks posed by the tool creation process are the potential for errors to compound through the process, the difficulty of the performing clinical trials of tools created using the process, and the difficulty of testing the tools created using this process. The stakeholders with risk roles for these technical risks are designers, fabricators, hospitals, patients, radiologists, regulators, and surgeons.

Unlike computational design, there are no stakeholders who are beneficiaries or decision-makers for all the related technical risks. However, surgeons are exposed to all these technical risks, and the only risk where they are not a decision-maker is that of the difficulty of performing clinical trials. Surgeons are at ethical risk from the difficulty of performing clinical trials, as they are affected by the decisions of designers and regulators. Regulators are also at ethical risk from the risk mitigation method of data collection, as their evaluation of the effectiveness of the tool creation process depends on the data collected and provided to them by designers, hospitals, and surgeons.

Patients are at ethical risk from all these technical risks, as they are not decision-makers for any of them. This ethical risk is addressed by the patient granting informed consent, as granting this consent is necessary to begin the process.

Errors may occur at each process stage. The stakeholders involved in the process (designers, fabricators, patients, radiologists, and surgeons) are both beneficiaries of and exposed to risks from the decisions made by other stakeholders. Only patients are at ethical risk from compounding errors in the process, as the other stakeholders are able to make decisions about their role in the process. Radiologists, developers, fabricators, and surgeons could each confirm that the inputs they receive from other stakeholders are as they expected, and that their outputs are suitable for the next process stage.

Many of the potential technical risks in the tool creation process may also affect medical 3D printing generally (for example, those related to medical imaging and 3D printing). These errors are qualitatively induced if they are due to human error, or quantitatively induced if they are the result of technical failures or limitations [55, 56]. Errors by surgeons or radiologists in selecting the appropriate patient image are qualitatively induced errors, while imaging artefacts in patient scans are quantitatively induced errors. Both qualitatively and quantitatively induced errors may compound throughout the process.

Including a computational design system in the process has the potential to introduce new quantitively induced errors. These errors may be compounded in the later stages of the process where the tool is fabricated and used. Qualitative and quantitative errors in the patient scan used as input for the computational design system could affect the tool design it produces. Similarly, flaws in the computational design system may lead to errors in the designs it produces, which are reflected in the 3D printed tool.

Qualitative and quantitively induced errors may be addressed by a clear distribution of role responsibilities throughout the process [56]. To address the technical risks introduced by the computational design system, there should be clearly assigned responsibilities for confirming that the input for the system is what it expects, and that the design it produces appears suitable for the tool’s intended use. Similarly, ethical responsibility for errors within the system itself and for flaws in the tools it designs should be clearly assigned. The clear assignment of responsibility would ensure that the stakeholders involved know where they are decision-makers within the process, and where decisions by other stakeholders expose them to risks.

Designers and regulators are decision-makers, beneficiaries, and exposed to the risks posed by the difficulty of performing clinical trials with bespoke surgical tools. Without performing clinical trials, designers and regulators would be unable to demonstrate the effectiveness of the tool creation process. Designers and regulators are exposed to the risk of approving and introducing bespoke surgical tools without sufficient testing. This is an ethical risk for patients and surgeons as clinical trials are important for establishing the safety and effectiveness of new medical technologies.

The ethical risk posed by the difficulty in performing clinical trials of bespoke surgical tools reflects the broader difficulty of performing randomised clinical trials (RCT) for surgical innovations. RCTs in surgery are expensive, difficult to generalise due to the strict inclusion and exclusion criteria for patients and the possible learning curve for surgeons, and blind or double-blind trials may be impossible to perform [57]. For bespoke surgical tools, there are a couple of possible means of conducting trials. A default 3D printed tool, where the tool’s dimensions are not modified to suit an individual patient, could be used to demonstrate the tool’s effectiveness compared to existing mass-produced surgical tools. Once the effectiveness of the default 3D printed surgical tool is established, it can be compared with a bespoke surgical tool. These trials would be unlikely to be blinded, as the surgeon would recognise the difference between a 3D printed tool and the regular mass-produced tool, and that the surgical team would need to know that the tool is customised as it may affect how the surgery is performed. Case studies may also serve as an alternative method of establishing the effectiveness of bespoke surgical tools [57].

The need to test the bespoke tool before surgical use is a potential disadvantage to using these tools, as it adds additional time and effort to the creation process, but this is also a particular challenge posed by single-use bespoke surgical tools. This problem might be resolved by collecting data about the input data used by the computational design system, the tool designs it creates, and the outcome of surgeries using these tools as all means of measuring the effectiveness of the system. Much of this data would already be recorded as part of regular surgical practice. Additionally, tool designs would need to be stored with patient records as the bespoke tool itself would be single-use and disposed of after surgery. There are already databases for tracking the outcomes of surgical procedures [58]. However, data may need to be shared with the computational design system designers to allow them to refine and optimise it. Patient data would also need to be recorded with the tool design to allow the designer to review any problems with the tool design. Care would be needed in data collection to protect the privacy of the patients treated with bespoke surgical tools. Existing practices for sharing anonymised medical data with software providers may mitigate this risk.

Hospitals and surgeons are beneficiaries, decision-makers, and exposed to risk by data collection. Both benefit from the insights gained from the collected data, decide what data is collected, and are exposed to the risk of infringing the privacy of patients. Designers benefit from being able to use the data gathered to refine the computational design system, and are decision-makers about how to use the collected data in revising it. Regulators benefit from having the data to better understand the effectiveness and safety of the system, and are decision-makers about how to use this data. Patients also benefit from the insights gained from the gathered data, and are exposed to the risks to their privacy from the use of their personal data.

6 Limitations and further research

The range of stakeholders interviewed is a limitation of this study. While surgeons and fabricators are well represented in the sample group, radiologists and patient advocacy organisations are both represented by only one participant each. Having more representatives of these groups would provide further insights into the risks relating to medical imaging and potential risks to patients. A greater representation of the views of actual patients in the sample may also provide deeper insights into how patients perceive the risks of new surgical technologies, and how they may consider the uncertainty around its risks and benefits (as opposed to relying on advocacy organisations that represent patient rights and interests).

A further direction for future research that was not covered by this study is to determine and quantify the likelihood and magnitude of the risks identified by stakeholders. The identified risks may differ considerably in both their likelihood and their significance. Quantitative risk assessment would therefore be useful next step for qualifying the likelihood of risks such as data errors and image distortions in medical imaging affecting the quality of the tools created based on this data.

7 Conclusion

Combining computational design and 3D printing has the potential to allow surgeons to create bespoke surgical tools with designs optimised for the individual patients they treat. In this research, we interviewed 21 representatives of stakeholders for bespoke surgical tools about what they perceived as the potential risks of using bespoke surgical tools. We identified the risks described as being either established risks of surgical innovations or as unique to bespoke surgical tools created using computational design and 3D printing. The established risks relate to the clinical use of 3D printing, medical imaging, and risks relating to new medical technology generally. The unique risks relate to the use of computational design, and to the process necessary to design these tools for surgical use. Distinguishing between unique and established risks for new medical technologies helps to identify where existing methods of risk mitigation may be applicable to a surgical innovation, and where new means of mitigating risks may need to be introduced. Similarly, identifying the unique risks highlights specific concerns that should be addressed during the development of surgical innovations.

In undertaking a closer analysis of the unique risks identified with using computational design to create designs for bespoke surgical tools, we used ethical risk analysis (eRA) to identify the risk roles (beneficiary, decision-maker, and risk-exposed) held by the stakeholders associated with these potential risks. This provided a framework that could be applied systematically to explore the perceived risks expressed by our participants about the design, use and creation of bespoke surgical tools, and it highlighted the risk roles occupied by various stakeholders at different stages of the process. In examining these risk roles, we identified how the two unique risks posed by bespoke surgical tools—computational design and the tool creation process—gave rise to a set of specific technical risks, and identified stakeholders who are at ethical risk from them. The value of distinguishing between technical and ethical risks in such a system is that allows the key responsibilities for managing these risks to be identified and any potential interdependencies between stakeholders in managing these risks to be made explicit. This research also demonstrates that an effective approach to the ethical use of new technology requires innovation in how we conceptualise such ethical considerations during the design stage of new technologies. In this paper, bespoke surgical tools provided a case study for exploring these aspects of ethical technology development and use and we believe this approach to identifying technical risks and the stakeholders at ethical risk holds promise for further refinement and application in other contexts.