Introduction

Abatacept, a fully human fusion protein, is the only biologic for the treatment of rheumatoid arthritis (RA) that selectively modulates the CD80/CD86:CD28 co-stimulatory signal required for full T cell activation and is available in both intravenous and subcutaneous (SC) formulations. The intravenous formulation of abatacept has demonstrated efficacy in several patient populations, including methotrexate-naive patients with early RA [1] and patients with an inadequate response to methotrexate [2–5] or to anti-tumor necrosis factor therapy [6, 7]. SC abatacept has been shown to be non-inferior to intravenous abatacept [8, 9]. The SC formulation, available as a pre-filled syringe, was first approved in the US in July 2011 for the treatment of moderate-to-severe RA in adults. Since then, SC abatacept has received marketing approval for the treatment of adult RA in numerous regions, including the EU, Japan, Canada, and Australia.

Although the SC delivery method affords users the benefit of self-injection at home, the pre-filled syringe requires several hand manipulations. This can be difficult for individuals with RA, as the disease often affects the small joints of the hand and impairs dexterity. To address this limitation and increase options for patients, a pre-filled, single-use autoinjector for abatacept (ClickJect®; Bristol-Myers Squibb, Princeton, NJ, USA) has been developed, with the aims of increasing the ease of the injection process and minimizing use error risk.

The abatacept autoinjector was designed and developed using human factors engineering (HFE). The autoinjector has a large rubberized grip and a hidden needle, and uses an automated injection process including automated delivery of the entire syringe contents. It also incorporates visual feedback to indicate the end of the injection, and shields the needle after injection. These features were designed to facilitate the injection process by improving handling ergonomics and reducing the number of hand manipulations, as well as to help patients who are new to self-injection and to overcome barriers to self-injection such as needle phobia.

This report describes the results of two studies that were conducted to assess the usability and acceptability of the abatacept autoinjector. A formative (pre-validation) study was performed to identify aspects of the product design and instructions for use (IFU) that could be further optimized to reduce the risk of user errors, while a summative study was performed as a final validation of the usability of the product and its labeling.

Methods

Two usability studies—formative and summative—were performed to measure the success of the HFE approach for device development (for details of the HFE, see Supplementary File 1). The goals of these studies were to determine: (1) whether the abatacept autoinjector, when provided with IFU, can be used without patterns of preventable use errors that would cause user or patient harm; and (2) if the autoinjector is acceptable for real-world use based on an assessment of key user needs (i.e., comfort, control, ease of use, confidence of dose) and overall acceptability.

The summative and formative studies were conducted by an independent company, Ipsos-Insight, LLC, separate from the study sponsor (Bristol-Myers Squibb). Participants were informed that the studies were testing a new autoinjector for an RA therapy, but no other specifics were shared. Throughout the studies, participants were blinded to the drug for which the autoinjector was intended, as well as to any sponsor involvement. Informed consent was obtained from all patients for being included in the study; study risks were included in the informed consent.

Study Population

To replicate actual use, the study population included individuals from three defined user groups who met the following criteria: (1) patients with RA: both injection naive and experienced, with a minimum disease duration of 6 months and a diagnosis of moderate-to-severe RA (as evidenced by current treatment with a biologic or non-biologic disease-modifying anti-rheumatic drug); (2) caregivers: both injection naive and experienced, with a family member who has been diagnosed with RA; and (3) healthcare professionals (HCPs): injection experienced, who routinely work with and train patients with RA, and with a minimum of 2 years of experience. For the summative study, a minimum of 15 respondents per user group were recruited in accordance with the US Food and Drug Administration human factors guidance [10].

In each study, users were divided into two groups: trained and untrained (Fig. 1). To reduce the potential for use error, users should be trained, by an HCP, on the correct technique for preparing and performing injections when a therapy is prescribed, prior to the first injection. However, because there is no guarantee that all users will be adequately trained before their first use, the device was also validated with untrained participants. For the formative study (Fig. 1a), the trained arm comprised only patients; for the summative study (Fig. 1b), the trained arm included both patients and caregivers. Because most HCPs are not expected to receive any training on how to use the autoinjector, this group was not represented in the trained arm of either study. Prior to the studies, representatives from Bristol-Myers Squibb trained a nurse educator on the correct use of the autoinjector. The nurse educator in turn trained patients and caregivers using only the IFU and a device, to ensure the training was representative of that available in a routine clinical setting.

Fig. 1
figure 1

Trained and untrained user groups by study type. HCPs healthcare professionals

Full size image

Use Environment

As these were simulated-use studies, efforts were made to simulate an end user’s actual use environment, and the study materials were designed to replicate the ergonomics and steps of actual administration. Injections are typically performed in low-traffic, low-distraction, low-noise settings, with normal office environment lighting conditions, and this environment was therefore replicated at the market research facilities where the studies were conducted. Study materials included an injection pad, which was strapped to the desired injection location (abdomen or thigh) in place of the injection site, a refrigerator (unplugged) for device storage, a hand sanitizer to mimic hand washing, and production-grade devices filled with medical-grade silicone oil to simulate the viscosity of abatacept and, hence, the injection time. The devices did not contain active therapy.

Study Procedures and Assessments

During the formative and summative studies, each participant was provided with the IFU and a device, and was asked to perform the necessary steps per the IFU to simulate an injection. The investigators evaluated each step (based on a risk analysis conducted prior to the study; see Supplementary File 1) using an assessment checklist, based on the perceptual, cognitive, and physical requirements of each step. High-priority tasks were defined as those that required correct completion for successful dose delivery.

The investigators then rated the observed performance as ‘correct performance’ (A), ‘performed with difficulty’ (D), or ‘use error’ (UE). A ‘performed with difficulty’ (D) classification included all close calls and near misses in which the user experienced confusion or difficulty, misinterpreted the IFU, or made an error that would result in mistreatment or harm, but then recovered and was able to continue so that no actual performance failure occurred. If a performance error occurred, but the user self-identified that the error had been made without prompting, it was also considered a ‘performed with difficulty’ (D) error, since the individual acknowledged the incorrect step and would self-correct with subsequent usage. A ‘use error’ (UE) classification included instances where the participant did not complete the step as appropriate. Following the usability assessment, the investigator and participant discussed the underlying causes of any difficulties (D) or use errors (UE) encountered during the simulated-use assessments, to ascertain if modifications could be incorporated into the product design or IFU to further optimize ease of use.

Participants were also asked to rate the acceptability of the autoinjector against a series of key attributes (i.e., comfort, control, ease of use, confidence of dose) and overall acceptability using a 7-point scale (1 = very unacceptable, 4 = neutral, and 7 = very acceptable). If time allowed, those participants who were experienced with an RA autoinjector were provided with their current or most recent autoinjector, allowed to re-familiarize themselves with it, and asked to provide competitive ratings. The participants did not perform a simulated injection with the competitor autoinjector. Participants experienced with an RA autoinjector were also asked for their preference of autoinjector based on their experiences during the simulated-use assessment.

A paired t test was used to identify statistical significance (P < 0.05) for the competitive rating analysis, while an exact binomial test was used for the user preference analysis. Finally, respondents were asked to name key positive features of the abatacept autoinjector, based on their simulated-use experience.

Results

The formative (pre-validation) study was conducted from February to March 2014 at market research centers in Fort Lee, NJ, USA; Fort Lauderdale, FL, USA; and San Francisco, CA, USA. A total of 34 patients with RA, 10 caregivers, and 10 HCPs were recruited (Fig. 1a). The summative (validation) study was conducted from August to September 2014 at market research centers in Fort Lee, NJ, USA; Baltimore, MD, USA; Stamford, CT, USA; and Dallas, TX, USA. A total of 51 patients with RA, 33 caregivers, and 15 HCPs were recruited (Fig. 1b). Respondents from Fort Lee, NJ, USA who had participated in the formative study were not eligible to participate in the summative study. The participant demographics are shown in Table 1.

Table 1 Participant demographics of (A) patients with RA, (B) caregivers, and (C) healthcare professionals
Full size table

Simulated-Use Usability Assessment

Most participants completed each individual step of the simulation task without use errors (Table 2). In the formative study, two use errors were observed with the high-priority injection steps, both related to the respondent [one untrained patient and one HCP (i.e., untrained)] not holding the device in place long enough to complete the injection. Although these errors were observed, post-study actions were not taken to modify the device design or labeling, since the root-cause analysis identified the outcome to be a result of the respondents’ non-compliance to the protocol and study methodology, rather than a result of mental confusion or physical limitations. However, minor layout changes were made to the IFU to improve compliance with two (low-priority risk) use steps: ‘pinch skin’ and ‘check the autoinjector for damage’.

Table 2 Frequency of completion of individual use steps during the formative and summative studies
Full size table

In the summative study, all participants except one (an untrained patient) completed all of the high-priority injection steps (Table 2). The exception related to the ‘hold for count and/or wait until blue indicator stops moving’ step. The use error was deemed an artifact of the study, as the patient was currently using a competitor autoinjector and was applying their expectations of the ‘feel’ in the skin of an injection to indicate the appropriate hold time. Since this tactile end-of-dose indication does not apply to a simulated-use setting, no mitigation steps were deemed necessary.

Simulated-Use Acceptability Assessment

Average acceptability scores for the device were consistently high for each user group for all measures (comfort, control, ease of use, confidence of dose, overall acceptability) (Table 3). In the formative study, all 54 participants responded (34 patients, 10 caregivers, and 10 HCPs); mean overall acceptability scores (out of 7) for patients, caregivers, and HCPs were 6.6, 7.0, and 6.8, respectively. In the summative study, 94 out of 99 participants responded (48 patients, 31 caregivers, and 15 HCPs); mean overall acceptability scores for patients, caregivers, and HCPs were 6.6, 6.8, and 6.8, respectively.

Table 3 Participant user experience data
Full size table

Current Autoinjector User Subgroup Analysis

Among patients who had previously administered injections with a competitor autoinjector, overall mean acceptability ratings were significantly greater for the abatacept autoinjector versus competitor autoinjectors across all user groups in both studies [formative study: patients 6.7 vs 5.2 (P = 0.0001), caregivers 7.0 vs 4.6 (P = 0.0093), HCPs 6.8 vs 5.1 (P = 0.0020); summative study: patients 6.5 vs 5.9 (P = 0.0404), caregivers 6.8 vs 5.8 (P = 0.0047), HCPs 6.8 vs 5.1 (P = 0.0002); Table 4]. In the individual categories of comfort, control, ease of use, and confidence of dose, all groups rated the abatacept autoinjector at least the same or superior to competitor autoinjectors; this difference was significant for almost all parameters, across all user groups, and in both studies (Table 4).

Table 4 Comparative user experience data following abatacept autoinjector use in (A) the formative study or (B) the summative study, from participants who had previously administered injections with an alternative autoinjector
Full size table

Participants demonstrated a strong preference for the abatacept autoinjector over competitor autoinjectors (Table 5). When preference data from the two studies were combined, a significantly greater number of participants in each user group preferred the abatacept autoinjector over their current or most recent autoinjector [patients 85.7% vs 14.3% (P = 0.00002), caregivers 84.2% vs 15.8% (P = 0.00443), HCPs 95.0% vs 5.0% (P = 0.00004)]. For the individual studies, this preference remained significant for patients in the formative (P = 0.00049) and summative (P = 0.01062) studies, and for HCPs in the summative study (P = 0.00049). Although the other comparisons were not significant, all preference data showed at least a trend in favor of the abatacept autoinjector in both studies.

Table 5 Autoinjector preferences from participants who had previously administered injections with an alternative autoinjector
Full size table

Key Positive Features

In total, 31 formative study participants and 87 summative study participants reported on the positive features of the abatacept autoinjector. The most frequently noted positive features were: rubberized grip (formative study, noted by 58% of respondents; summative study, 56%), device size (formative, 55%; summative, 52%), visualization of dose progression (window size and location, colored plunger rod; formative, 45%; summative, 57%), button ergonomics (shape, reach, activation force; formative, 19%; summative, 37%), and ease of use (number of steps, simplicity; formative, 29%; summative, 30%) (Fig. 2).

Fig. 2
figure 2

Key positive factors of the abatacept autoinjector reported by respondents

Full size image

Discussion

The independently conducted studies reported here found that the newly designed autoinjector for the SC delivery of abatacept in patients with RA was easy to use, with low residual risk to users in a real-world setting. Most participants, whether patients, caregivers, or HCPs, trained or untrained, performed all high-priority use steps correctly. Participants rated the abatacept autoinjector highly for user experience in terms of comfort, control, ease of use, confidence of dose, and overall acceptability, with scores nearing the maximum of the 7-point scale used. All user groups across both studies rated the abatacept autoinjector significantly higher than competitor devices for overall acceptability, and when preference data from both studies were combined, significantly more participants in each user group preferred the abatacept autoinjector over their current or most recent device. The drivers for the positive ratings and preference could be grouped into three main categories: device ergonomics (size, rubberized grip, button reach, force to activate), which provides comfort and security during injection; visualization of dose progression (size and location of window, colored plunger rod), which provides confidence of dose delivery; and simplicity of the process, which contributes to the overall ease of use of the device.

Because RA can affect the small joints of the hands, many patients with RA suffer from compromised dexterity. In addition to interfering with activities of daily living, poor dexterity can affect the ability to perform the steps required for self-injection in patients who would otherwise be eligible for an SC therapy. Autoinjectors are available for the administration of many SC therapies for chronic conditions, including those in which patients have impaired dexterity [11, 12]. The single-use, pre-filled autoinjector described here was developed following the principles of HFE, to help users to inject one dose of SC abatacept in a home environment. As evidenced by the results of the current studies, the new autoinjector is an important advance for patients receiving, or eligible to receive, SC abatacept, as it increases the ease of the injection process without introducing usability challenges.

In both the formative and summative studies, the vast majority of respondents completed each individual step of the simulated injection without use errors. These results demonstrate that, overall, both trained and untrained users were able to operate the autoinjector safely and effectively to deliver the SC injection by following the IFU.

Furthermore, the overall preference in favor of the abatacept autoinjector over other available injection devices across a range of attributes indicates that the device’s features were favored by users with experience of such devices. The fact that similar results were obtained across the two studies provides further credence to these findings. These data also indicate that the device features designed in early development (i.e., shape, rubberized grip, placement/size of window, colored plunger rod, button ergonomics) contributed to the favorable comfort, control, confidence of dose, ease of use, and acceptability ratings for the device, indicating value by the end users. Furthermore, as no patterns of error were observed in the studies, use risk was mitigated.

A limitation of these evaluations is the simulated-use design of the studies, rather than autoinjector use in a clinical setting. However, according to US Food and Drug Administration guidance, simulation testing, as conducted here, is an acceptable method for assessing the safe and effective use of an autoinjector device. Greater support for the ease of use of the autoinjector may have been obtained by restricting the study population to patients with more severe hand deformity, for whom any improvement would be most beneficial. In addition, respondent numbers were low in the subgroup analyses. A further limitation is the lack of a simulated injection using the competitor autoinjector device; permitting a simulated injection with a competitor device may have allowed for a more accurate comparison with the abatacept autoinjector versus relying on participant memory. The time span between using and rating the competitor device may have influenced the comparative data; therefore, not capturing and adjusting for this time difference may affect the interpretation of these results. Lastly, the autoinjector was branded in order for participants to evaluate its colored design features. As such, HCPs may have ascertained which product the autoinjector was designed to deliver.

Conclusions

The abatacept autoinjector was found to be highly acceptable against key measures of comfort, control, ease of use, confidence of dose, and overall acceptability in two independently conducted simulated-use studies. High overall acceptability ratings were achieved, and these ratings were significantly greater than those for competitor devices. In addition, a significantly greater number of participants in each user group preferred the abatacept autoinjector over their current or most recent autoinjector.

Participants’ positive experiences with the abatacept autoinjector can be attributed to the following key features: device ergonomics (size, shape, rubberized grip, button reach, and activation force), which provides comfort and security during injection; visualization of dose progression (size and location of window, colored plunger rod), which provides confidence of dose delivery; and simplicity of the process, which contributes to the overall ease of use of the device. These results show that HFE optimized the device design and IFU of the abatacept autoinjector to ensure its effective use by patients, caregivers, and HCPs without patterns of preventable use errors.