1 Introduction

It is known that techniques such as User Centered Design, Human Centered Design (HCD), or User eXperience are often applied in technological research as well as development projects with the aim of creating usable and acceptable products and/or services [1]. The participation of humans or the use of personal data in research has ethical implications which must, at the very least, be supervised and approved by an ethical committee [2], and it is fair to say that the applicable legislation is often difficult to be understood by researchers, especially in the area of engineering [3]. Universities such as Harvard, UC Berkeley, or Colorado Boulder in the USA, or the University of Tübingen in Germany, for example, have specific training in ethics for engineers who involve humans in their research [4,5,6,7]. However, it is still common to find a lack of understanding regarding these issues among technical researchers, especially in Europe. In previous research, an analysis of the applicable European Directives and Regulations, and recommendations related to the experimentation with humans was drawn up [8]. In the present manuscript, the description of the tool designed following the analysis of results is included. The tool, named Ethool, was developed to help European researchers to take decisions in their projects and to determine the ethical and legal approvals needed to their research as it is often not easy to discern which applicable recommendations and regulations are experimentation dependent. This paper presents, not only the Ethool design and development, but also the iterative usability evaluation of it, including the improvements made following participants’ feedback.

2 State of the art

The present study was carried out based on the previous analysis done by the authors on the recommendations and the European directives for research with technological developments including human involvement [3]. According to that initial work, several guidelines can be found such as the Declaration of Helsinki [9] which is well accepted nowadays, with its 2009 version being mentioned in the European Regulation relating to clinical trials with medicines [10] or the Oviedo convention that was converted into an agreement by the Council of Europe as a reference point for human dignity and rights in medicine and biology [11].

In Europe, Directive 93/42/CEE regulated the use of medical devices with research purposes for many years including research studies using those devices [12]. However, a new European Regulation (2017/745) on medical devices has been in force since May 2021 [13, 14]. In both documents, it is mentioned that the competent authority of the country where the clinical study will be carried out must approve all research done with medical devices. The role of the competent authority is to, at least, evaluate the risk–benefit balance of the device for participants and determine whether the study is sufficiently relevant from a scientific point of view. Most countries of the European Union have their own competent authorities for the supervision of research with medical devices, and if this were not the case, the competent authority of another country could take on this responsibility [15].

In previously mentioned documents, it is specified that any research with medical devices must follow the recognized ethical principles, and the proposed research must be approved by the corresponding Ethical Committee. Moreover, as in any research involving humans, the relevance of the informed consent procedure of the participants is also highlighted.

The European Regulation 536/2014 concerning clinical trials [10] came into effect in May 2016 and reinforces the mandatory nature of obtaining endorsement from an Ethical Committee in order to carry out any clinical research.

In addition to the presented legislation and regarding the research with medical devices, the UNE-EN ISO 14155 standard may be considered as a reference document in any country [16]. This document includes information regarding the ethical aspects to be considered when designing a study, the need to obtain the ethical committee approval and the description of the requirements for this purpose, and the most appropriate informed consent process for the study. In the same vein, the International Conference on Harmonization is a board for the harmonization of technical requirements to be applied with humans by the pharmacists [17]. This board has developed several good clinical practice regulations that are recognized worldwide. The regulations and recommendations mentioned above were drawn up for clinical trials focused on drugs, but almost all of them should be applied in a similar way in any research with medical devices in which humans are involved. These rules also explain some guidelines for treating participants, including the informed consent process and the ethical issues to be borne in mind when designing the study protocols. They also underline the significance of compiling documentation such as a researchers’ manual, in order to provide all the details about the study and the medicines or medical devices to be evaluated.

2.1 Conclusions of the legislation review

The main outcomes of the reviewed documentation about legal aspects to be taken into account for researchers who want to evaluate their technological developments with humans are summarized as follows:

  1. 1.

    The research to be carried out is not considered to be necessarily under the supervision of an ethical committee when human beings are not directly involved in the trials, or when participating they do not use a device, or no risk (either physical, or psychological) is expected, there is no recording or treatment of personal data and the participants target group is not considered vulnerable [18]. For example, when participants are filling in an anonymous survey in which the collected data are not of a sensitive nature. However, in the case that the researchers have any doubts, ethical committee support is recommended.

  2. 2.

    The study is considered as observational research when humans are involved and the devices and/or the software to be used are for medical purposes, these must have CEFootnote 1 certification, be employed according to their intended use and following the manufacturer’s instructions. In this case, there is a need for Ethical Committee approval. This Ethical Committee must have the corresponding competences to approve that type of research.

  3. 3.

    The study is considered to be clinical research which needs Ethical Committee approval, and, in addition, the approval or the notification to the competent authority for clinical trials with medical devices, depending on the features of the system to be evaluated. In any case, the competent authority must also be informed about any relevant adverse events. The Ethical Committee involved in this case must also have the competences to approve this type of research.

  4. 4.

    Otherwise, the study is considered a research project, and therefore, Ethical Committee approval is required in the remaining cases were humans and/or their personal data are involved.

In any case when humans and/or their personal data are involved, and, moreover, when ethical approval is needed, an informed consent process for the participants would be required. When the process is anonymous, participants must be informed about the project objectives and procedure. It is also mandatory to register clinical research with medical devices in the Clinical trials in the European Union (CTIS) [19] from the 31st of January 2023[NO_PRINTED_FORM]. Other clinical trial public databases can be found, such as the Swiss National Clinical Trials Portal [20], or the platforms managed by the U.S. National Library of Medicine [21], or the World Health Organisation [22].

3 Ethool design and development

It was decided to build a flowchart according to the conclusions of the documentation review mentioned in the previous section (Fig. 1), which was used by several researchers within the same center [8]. The flowchart was designed with the aim of identifying the ethical and legal aspects that apply to any experimentation with humans when technological developments are involved including medical devices or software developments. Users were confronted with several issues and had requested an easy tool based on the proposed flowchart. Based on that feedback the Ethool tool was built according to the flowchart and thus to the documentation review.

Fig. 1
figure 1

Flowchart summarizing the European legislation for research involving humans used to build Ethool [8]

Full size image

This tool has been conceptualized based on a questionnaire format, in which the researchers are easily able to answer some simple questions and obtain a customized answer for their particular case. It was drawn up with the idea of giving a simple solution to researchers, mainly in the engineering and software computing area, in which the reviewed legislation is summarized, rather than requesting researchers to study the entire range of legislation and recommendations applicable to their developments. This solution can also give researchers the opportunity to register the features of their projects and to obtain a report including their input.

As it was decided to build Ethool based on a questionnaire format, Microsoft Forms® software was selected [23]. This technology is easy to use and, moreover, an agreement to comply with the European Regulation on Personal Data treatment is included with the company in which it has been implemented and evaluated [24]. This technology, due to its responsive interfaces, also provides the option of being accessible from any device with a browser and an Internet connection. Ethool was designed with simple questions which can mostly be answered with a “Yes” or “No” and depending on the answer the next question will only be asked when needed, avoiding unnecessary steps. The early prototype versions of the Ethool were only developed in Spanish, as the tool evaluation was carried out only in that language.

4 Ethool usability evaluation

Ethool’s usability was considered to be really important. According to Nielsen, usability comprises learnability, efficiency, memorability, error control, satisfaction and utility [25]. An evaluation of Ethool was carried out with the objective of measuring and improving usability. The evaluation was carried out in an iterative process including two trials with the same participants as proposed in any HCD methodology [1]. Between each trial, improvements were also developed on prototypes according to participants’ feedback. At the end of each trial, participants were requested to fill in the System Usability Scale (SUS) questionnaire in the Spanish version [26, 27]. The aim was to measure the usability of the tool by using this scale and to compare the results between the two prototypes including the improvements made between each trial, with the expectation of better results in the second prototype.

4.1 Participants

The target group for the tool is technological researchers who might involve humans in the design and development of their projects. For this reason, participants were recruited within the same research center in which the tool was developed. However, researchers with different profiles (with and without engineering profile) were also considered in order to identify the difficulties and usability issues that might be faced by researchers from different backgrounds.

The inclusion criteria for participating were to have worked in the aforementioned research center within the Health Unit, to be over 18 years old and to voluntarily accept to participate in the research, including signing the informed consent prepared for the study. Participants had to be fluent in Spanish, and the tool usability would be assessed for language comprehension. The foreseen distribution was for 60% of participants to be from the neuro-engineering and medical robotics area (engineering profile) and 40% to be non-researchers or researchers in other areas who were not used to doing direct research with humans (out of the target group).

Nielsen and Landauer proposed a mathematical model to calculate the number of participants needed to detect a system’s usability problems [28]. According to Nielsen’s proposal, 15 users are sufficient to identify all the usability problems [29]. However, if the study is iterative, it is proposed that testing may be made with only 5 users. An e-mail was sent to every worker within the Health Unit inviting them to participate in the evaluation. In the present study, a total of 12 people were recruited from the e-mail responders, according to the inclusion criteria, and these participated in the two organized trials. One participant dropped-out during the study reporting that she did not have enough time to complete the tasks. Therefore, no more people than needed were involved according to the minimizing ethic principle although 13 additional workers demonstrate interest in participating.

In the end, 36% of the participants were researchers who were not used to involving humans in their research. It should be noted that 45% of the participants were male, 55% female and 0% non-binary. Information was also collected regarding the number of years that participants had spent working in the research center, with the aim of identifying some differences between the people with and without experience in the field.

4.2 Methodology

As previously mentioned, the usability evaluation was designed with an iterative methodology divided into two trials in which the same participants evaluated Ethool and the feedback gathered from both trials was used to improve it.

The trials were organized as a summative test, where the participants were asked to perform several guided tasks using a script and other unguided tasks. Two different scripts were prepared to guide the participants in the tasks to be performed. 50% of the participants received Script 1 for the first trial to evaluate Prototype 1, and the other 50% were given Script 2. To evaluate Prototype 2 in the second trial, the scripts were alternated between the participants, with the aim of assessing any comprehension problems caused by the scripts that might affect the usability evaluation of the tool. Participants were requested to fill in a questionnaire with some additional open questions, with the aim of collecting qualitative information to improve the Ethool usability. Open questions were as follows:

  • Did you find any difficulty to understand the terminology used? Please, tell us which term(s) posed a problem.

  • Was it especially difficult for you to answer any of the questions? Which one(s)?

  • Did you have any difficulty to follow these guidelines?

  • Could you give us any feedback to improve this tool?

As previously explained, a SUS questionnaire was also completed by the participants at the end of each trial with the objective of measuring the usability of the tool.

As Ethool was available online, the participants were able to perform the trial from any device (computer, tablet or smartphone) with an Internet connection. Each participant was requested to use the tool at least twice for each evaluation trial, with an estimated total duration of 30 min including filling in the questionnaire. At the end of each use of the tool, a Likert scale of 5 points was also included to evaluate the process of a questionnaire-based system for the proposed evaluation, using the question “What do you think about this questionnaire?”

The study protocol and informed consent for the proposed evaluation were approved on 23 May 2019 under protocol number M10/2019/128 (document 113/2019) by the Ethical Committee of the University of the Basque Country University for Research involving Human Beings (CEISH-UPV/EHU) which has number BOPV 32, 17/2/2014 [30]. The methods used for the participants’ involvement were carried out according to the Declaration of Helsinki [9], Oviedo Convention [11] and biomedical research Spanish law [31].

5 Results

Between 22 and 33 uses of the tool were planned for each trial. Each script provided to the participants included 2 scenarios to be tested with the tool, and the user was requested to add a further scenario (2 complete uses + 1 optional use per person). In Prototype 1 evaluation, 32 uses were reported, and in Prototype 2, 27 uses, with an average of 2.9 and 2.45 uses per participant, respectively, which could be considered as the mean of the expected use and demonstrate that each participant completed at least the 2 defined scenarios of the scripts.

In the evaluation of Prototype 1, 8 of 11 participants considered that Ethool could help them in the future. It should also be noted that one of the participants reported some issues with the provided script in the evaluation of Prototype 1. Another reported some problems to answer the question about the personal data collection included in Prototype 1. Nevertheless, in the evaluation of the Prototype 2, which included some improvements over Prototype 1, all the participants considered it helpful.

The average time taken by participants using Prototype 1 for each project was 541 s, while the average time needed with Prototype 2 was 430 s, with a gain of 20.5%. Two positive conclusions could be taken from this result: (1) the improvements following the participants’ feedback in the first trial and included in Prototype 2 made Ethool more effective; (2) participants experienced a very rapid learning process between the first and the second trials.

As previously stated, the participants were requested to fill in a short Likert of 5 points regarding their subjective perception of the use of Ethool in each trial. In Fig. 2, the collected data using this Likert are shown. In this figure, the percentage of each answer for both prototypes are represented. In the evaluation of Prototype 1 the average rating of the tool was 4.59, while in the evaluation, Prototype 2 was 4.67 out of 5, with a gain of 1.71%.

Fig. 2
figure 2

Subjective perception of the tool rated in a 5-point Likert. Data is shown in percentages for each possible answer in the scale (1–5) over the total of answers for two trials (Prototype 1 evaluation results in dark gray)

Full size image

SUS questionnaire results were processed using a specific Python script which was developed according to guidelines [32] and is publicly available in GitHub [33]. Participant ID09 did not rate Question 9 in the evaluation of Prototype 2, but following Bangor, Kortum & Miller recommendation [34], it was rated with a value of 3 in order to calculate the average for that participant with the rest of the recorded data. The SUS questionnaire results from both trials are represented in Fig. 3. There, it can be seen that the mean calculation for the Prototype 2 evaluation, which included the improvements from the participants’ feedback from the first trial, achieved a higher rating than the former. Specifically, the obtained average rates were 87.5 and 93.0 for Prototype 1 and Prototype 2, respectively. According to the work published by Bangor, Kortum and Miller [34] the usability of both prototypes can be considered as “Excellent” as the averages of the two evaluations were higher than 85.0 points.

Fig. 3
figure 3

SUS results for each participant and mean for each evaluation (Prototype 1 evaluation results in dark gray). The threshold defined by Bangor et al. for an “Excellent” usability result, is represented in black

Full size image

The improvements made after the first trial and therefore included in Prototype 2 are summarized in Table 1. The interface design of the application in the different Prototypes are shown within Table 1.

Table 1 Summary of the changes implemented after each trial of the usability evaluation
Full size table

6 Discussion

From the results gathered in both trials, the low ratings from participants ID05 and ID07 are noticeable. In the evaluation of Prototype 1, Participant ID05 reported that Ethool was not sufficiently useful and requested more information in the answer to be added as improvement; a request which was implemented in Prototype 2 (see #2 in Table 1). Participant ID07 reported some problems to understand the guidelines provided for the test thereby making it more difficult for him/her to use the tool. However, both participants rated Ethool much higher in Prototype 2, although they still evaluated the tool below the average of the rest of the participants.

On the other hand, participants ID06 and ID09 rated Prototype 2 lower than Prototype 1. In the case of ID09, the final calculation was lower because this participant did not provide a rating for Question 9. The remaining answers from this participant were very positive, so it could be considered that she would have rated it with a better value than 3, which was used for the average calculation. In the case of ID06, there is no noticeable reason to explain this difference. Moreover, this participant commented that the tool had improved in Prototype 2 compared to the previous version.

Regarding the profile of the participants, the division between neuro-engineering and medical robotics and other areas, as well as the number of years spent in the company were considered relevant, as the language used in some descriptions was fairly technical. However, there was no evidence of any connection between the results and the previous experience or background of the participants.

It needs to be pointed out that some additional improvements were suggested by participants, but they could not be implemented for various reasons. Those proposals are listed below:

  • In the question “Is it necessary to collect or save any of the following information?” with options such as informed consent, participants’ name or personal data, it was reported that more than one answer was possible. It was not feasible to give this choice because Microsoft Forms® tools does not give the option of programming different branches depending on the answers with different combinations. With the aim of avoiding any misunderstanding, some instructions were added to the question, as has been previously explained (see #4 in Table 1).

  • Some examples of medical devices were requested to make it easier to understand the objective of the product and to help decide whether it was a medical device or not. Looking for correct examples for every case is a complex task, so this suggestion was postponed to be potentially added in the future if requested by more users.

7 Conclusions and future work

In this manuscript, a tool called Ethool intended to facilitate decision-making on the European legal issues applicable in research involving humans has been presented. Ethool also carries out the function of collecting information from the experiments in the case that a report is needed.

Moreover, the usability validation of the developed tool, in which 11 people participated in two iterative trials, has been presented. These volunteers reported subjective opinions regarding the user experience and ease of use of the tool. Participant feedback was objectively and subjectively processed according to the information gathered. Between each trial various improvements in the tool were implemented which had some impact on the participants evaluation. The SUS questionnaire was used to evaluate the usability of Ethool as other researchers do in similar usability evaluations [35]. Ethool was highly rated with an improvement of 5.5 points in the SUS questionnaire between each trial, with a relative gain of 6%.

In the DIH-HERO European project, in which the authors are participating, several semi-structured interviews were carried out about awareness and the use of standards in robotic development, including any ethical and legal issues that might apply. One of the conclusions from those interviews was that a tool like Ethool could be useful for small and medium-sized enterprises (SMEs) or researchers, with the aim of summarizing the applicable best practices, standards, and legislation. The inclusion of the information regarding specific details from other European countries (ethical committees contact points) and the translation of Ethool into English were carried out in the context of this European project. For this reason, the tool was designed offering a general answer to the questionnaire and specific contact points information for each country after selecting one from the list. Visual aspect of this new design is shown in Fig 4.

Fig. 4
figure 4

Visual aspect for the answer in the online version

Full size image

Ethool was presented in various events organized by the DIH-HERO consortium. The objective for the tool is to be used by SMEs or researchers interested in carrying out experiments with technological developments involving humans. For this reason, Ethool has been made available online through the DIH-HERO portal [36]. Authors expect that wider use of the tool will generate additional feedback on its usefulness or usability, which would be extremely valuable with regard to improving Ethool in the mid-term. At the same time, the Spanish version of Ethool is available only for researchers working at the same centers as the authors. With this approach in, which the tool could be tested with real projects, the authors expect to receive more feedback from real users about needs or possible improvements. The additional improvements suggested by participants but not implemented during this evaluation will also be considered in the future if more users agree on them. Information from additional countries will also be added in the mid-term thanks to the participation of DIH-HERO partners.

The authors expect that Ethool will be able to help SMEs and researchers, mainly those from the field of engineering, to take the appropriate decisions in the future.