Introduction

Over the last decade, opioid-related deaths have increased at an alarming rate resulting in a growing public health concern in the United States. There were more than four times as many opioid-related fatalities in 2010 as there were in 1999 [1] and opioid-related fatalities have remained at these levels through 2013 [2–4]. This increase coincided with a nearly fourfold increase in the use of prescribed opioids for the treatment of pain [5]. Naloxone hydrochloride injection, used for four decades by trained emergency response personnel, is the first-line treatment to reverse life-threatening opioid-induced respiratory depression (OIRD) and overdose [6, 7].

In the 1990s researchers began to investigate off-label administration of parenteral formulations of naloxone via the intranasal route of delivery and emergency medical technician units initiated pre-hospital intervention protocols using the intranasal route of naloxone administration in the early 2000s [8]. To further combat the growing opioid overdose crisis, members of the Harm Reduction Community began distributing naloxone nasal atomization kits (NXN) [9]. These organizations provide instructional sessions attended typically by the at-risk opioid users who are then expected to subsequently train their life partners, caregivers, or family members on the correct assembly and use of the kit.

The NXN consists of a glass cartridge containing a parenteral formulation of naloxone, a nasal atomizer (e.g., LMA® MAD Nasal™ [10]; Teleflex Inc., Morrisville, NC, USA), personal protective equipment, and instructions-for-use. The instructions specify that the naloxone should be administered into each nostril of an individual suffering from OIRD. The intranasal route of naloxone administration is not approved by the United States Food and Drug Administration (FDA). Therefore, use of this intranasal naloxone kit is considered an off-label use of naloxone. Additionally, the limited availability of naloxone combined with the level of training required to ensure correct assembly and use of the kit has been a barrier for wider community access to this potentially life-saving drug [11].

A novel naloxone auto-injector (NAI), EVZIO® (naloxone hydrochloride injection; kaleo, Inc., Richmond, VA, USA), was specifically developed using state-of-the-art human factors engineering to address this unmet medical need. NAI was developed for immediate administration, with little to no training, by family members or caregivers of opioid overdose victims during an opioid emergency outside of a medically supervised setting. NAI provides voice instructions that assist in guiding the user through administration of a 0.4 mg dose of naloxone hydrochloride injection and contains a retractable needle system that conceals the needle before, during and after administration. NAI was granted fast track designation and a priority review by the FDA. NAI is indicated for the emergency treatment of known or suspected opioid overdose as manifested by central nervous system or respiratory depression and is intended for immediate administration as emergency therapy in settings where opioids may be present [12].

Various organizations have suggested that the NXN kits are easy to use and may not require training [13]; however, no usability studies have been reported that evaluate the use of NXN kits without training or the ability of individuals to retain what they learned during training to successfully administer a dose of naloxone during an opioid emergency. A recent study reported an evaluation of the use of NXN in trained and untrained users in a community setting. However, this evaluation was not specifically designed to evaluate correct use of NXN, usability of NXN, or responder’s adherence to instructions [11].

This is the first known study conducted to evaluate the usability of NXN as compared to NAI during a simulated opioid overdose emergency by participants without training and following one-on-one training sessions by a healthcare professional. This study was conducted by a third-party contract research organization whose specialty is human factors investigation and residual risk analysis.

Methods

Participants

Healthy adult participants who were not severely visually impaired or hearing impaired were eligible to enroll. All participants were English language proficient. The participants had no prior experience with using, or being trained on, any naloxone delivery products and no relevant experience with similar nasal or injection delivery systems.

Study Materials

NXN were purchased and consisted of one carton of naloxone hydrochloride for injection, USP (International Medical Systems Ltd., South El Monte, CA, USA; NDC 76329-3369-1) and LMA MAD Nasal Device contained within a plastic bag. Evzio (NDC 60842-030-01) were obtained from kaleo, Inc. NXN and NAI are shown in Figs. 1 and 2, respectively. A full size adult mannequin was used to simulate a patient experiencing an opioid emergency.

Fig. 1
figure 1

Naloxone nasal atomization kit (LMA® MAD Nasal™; Teleflex Inc., Morrisville, NC, USA) components and packaging. In descending order: Outer packaging for naloxone and delivery syringe (top); plastic delivery syringe; glass vial of naloxone hydrochloride; nasal atomization device; packaging for nasal atomization device (bottom left)

Full size image
Fig. 2
figure 2

Naloxone 0.4 mg auto-injector (naloxone hydrochloride injection, USP; EVZIO®; kaleo, Inc., Richmond, VA, USA) outer case (left) and device (right)

Full size image

Study Objectives

The primary objective of the study was to compare the usability of NAI and NXN in the administration of naloxone during a simulated opioid overdose emergency. The primary endpoint was successful administration of a simulated dose of naloxone into a simulated patient (mannequin) before and after receiving training without committing a Critical Task Error. Critical Task Error was defined as an error in essential task steps required to administer a clinically meaningful dose of naloxone successfully. The secondary endpoints were using the NAI or NXN in accordance with their instructions-for-use and the completion time of administration for NAI or NXN. Critical Task Errors, errors in adherence to instructions, and time to completion were measured. Subjective feedback was obtained for NXN and NAI after each use phase to facilitate evaluation of these errors and user preferences for each product.

Study Design

This was a single-site, randomized, open-label study. The study included three different phases. In phase 1, study participants were presented with brief information on the use scenario, but were not provided any training on the use of the two products. The use environment simulated a home setting including chairs, coffee table, and the mannequin (simulated patient) located on a couch randomly assigned to face-up or face-down orientation. To add distraction and induce stress, a movie played on a television screen in the background. The participant was observed with the use of several cameras and two investigators located within the simulated environment. The investigators documented the participant’s administration using each product (randomly assigned) into a simulated patient experiencing an opioid overdose. No training was provided to participants prior to or during phase 1 and participants could only rely on the instructions-for-use found on the products themselves and no additional instructional materials (e.g., instruction-for-use leaflets, etc.) were present.

In phase 2 of the study, each participant received training from a healthcare professional (nurse) on the use of each product. Training included having each participant demonstrate correct use of NAI and NXN.

For phase 3, at 7 or 8 days after the training session, the participants returned to attempt to administer naloxone to a simulated patient mannequin using NAI and NXN during a simulated opioid overdose emergency. No additional training was provided during phase 3. Participants were expected to rely on their recall of training to demonstrate correct use of the products. Participants were monitored by two investigators for compliance with critical tasks and adherence to instructions-for-use. Participants were videotaped to archive their performances during both simulated use phases of the study. Either investigator was permitted to terminate a simulation if the participant posed a safety risk to himself or herself.

Study Oversight

All procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1964, as revised in 2013. Informed consent was obtained from all participants included in the study. The study was performed by UL-Wiklund Research & Design and was conducted under the review and approval of the Allendale Investigational Review Board. Study participants were blinded to study funding which was provided by kaleo, Inc.

Assessments of Usability

The participants were assessed for their ability to complete a successful administration of naloxone into a mannequin using both NAI and NXN. There were two simulation sessions during the study; during phase 1, participants were presented with an opioid overdose emergency scenario of a simulated living room environment with distractions (e.g., observers, television playing in the background), with the simulated overdose victim mannequin lying on a couch. With no prior training, participants were instructed to administer a simulated dose of naloxone into the mannequin after being presented with each naloxone product, NAI and NXN. Participants could only rely on the instructions-for-use found on the products themselves and no additional materials (e.g., instructions-for-use, leaflets, etc.) were present.

Immediately following phase 1, each participant received one-on-one training for each product from a nurse and had to demonstrate correct use in accordance with instructions (phase 2). Seven to eight days post-training, the participants returned for phase 3 and repeated the identical opioid emergency scenario simulation with both naloxone delivery products. For each session, participants were given a maximum of 15 min to complete each administration.

Completion time was assessed from the time each product was provided to the participant and the investigator indicated ‘start’ to the time the participant completed administration of product, verbally stated ‘finished’ or after 15 min (maximum time allowed). In cases where the tasks could not be completed (i.e., the user gave up or the simulation was terminated due to time), the participant’s time was imputed with a value of 15 min.

Evaluation of Usability and Administration During Phase 1 and Phase 3

Correct use of medical products, including drug/device combination products can be defined as compliance with critical tasks or adherence to instructions. Adherence to instructions is a measure of using the product as intended, including adherence to steps that may not strictly be required to deliver a clinically meaningful dose of naloxone. Critical tasks for this study were those steps that are required to ensure successful naloxone administration resulting in a clinically meaningful dose of naloxone available for systemic absorption. For example, for NAI, the drug is indicated for subcutaneous or intramuscular use and the instructions-for-use states to inject into the outer thigh. However, an injection into commonly accepted subcutaneous or intramuscular locations of the abdomen, thigh, deltoid, or buttocks would be expected to deliver an effective dose. Therefore, if a participant injected NAI into the deltoid of the mannequin, it would be coded as a use error for adherence to the instructions-for-use, but not as a critical use error. Similarly, for NXN, the instructions-for-use indicates administering exactly half the dose (1 mL) into each nostril; however, this may not be required to achieve a clinically meaningful effective dose. Therefore, the critical task for NXN was defined as administering any quantity of naloxone into both nostrils, assuming that some exposure to both nostrils would permit absorption of a sufficient quantity of naloxone from the nasal mucosa.

Statistical Analyses

Descriptive statistics were calculated to summarize continuous measures. Frequency count/percentages were used for summarizing categorical measures. For each product, the proportions of participants who would have successfully administered a clinically meaningful dose were reported as counts and percentages. Success was defined as not committing any use errors on each product’s critical tasks as presented in Table 1 as well as the ability to use each product according to the product’s instructions-for-use as presented in Table 2.

Table 1 Critical tasks for NAI and NXN
Full size table
Table 2 Adherence to instructions-for-use for NAI and NXN
Full size table

The ability to administer NAI and NXN in accordance with each product’s instructions-for-use was also assessed and reported as counts and percentages.

McNemar’s test was performed for comparison of critical task success as well as adherence to instructions-for-use success for NAI and NXN between phase 1 and phase 3. Wilcoxon rank-sum test performed as sensitivity analysis for time to task completion for comparison between NAI and NXN within each phase (phase 1 and phase 3), and NAI or NXN between phase 1 and phase 3.

Results

Participants

A total of 42 healthy participants between 18 and 65 years of age were enrolled in this study. The demographics and baseline characteristics of the participants are shown in Table 3. The mean age was 41 ± 13.4 years for the participants. There were 45.2% male and 54.8% female participants. All 42 participants completed the study.

Table 3 Participant demographics
Full size table

Usability and Administration of NAI and NXN

During phase 1, 90.5% of participants completed a successful injection using NAI on the first attempt with no prior training or instruction leaflets versus 0.0% using NXN for the simulated opioid overdose scenario (P < 0.0001).

Phase 2 resulted in all participants successfully completing a one-on-one training session with a nurse, which included each participant demonstrating successful use of each naloxone product. Seven to eight days after the training session, participants returned for phase 3 and repeated the simulation.

During phase 3, 100% of participants successfully administered a simulated naloxone dose using NAI versus only 57.1% using NXN (P < 0.0001).

Task completion time for NAI was 0.9 ± 0.25 min for phase 1 (untrained) and 0.5 ± 0.15 min for phase 3 (post-training). NXN task completion time for phase 1 was 6.0 ± 4.76 min and 2.0 ± 2.15 min for phase 3.

Participant Completion of Critical Tasks

Each product’s critical tasks are detailed in Table 1. During the phase 1 opioid emergency simulation (i.e., no training or instructional reference materials provided), 38 out of 42 participants (90.5%) successfully administered a simulated dose of naloxone with NAI compared to 0 out of 42 participants (0.0%) with NXN (P < 0.0001).

During the phase 3 opioid overdose emergency simulation (i.e., after training), 42 out of 42 participants (100%) successfully administered a simulated dose of naloxone with NAI compared to 24 out of 42 participants (57.1%) with NXN (P < 0.0001). The proportion of participants who successfully administered a simulated dose of naloxone with NXN was significantly (P < 0.0001) greater in phase 3 (i.e., post-training) than in phase 1 (i.e., no training). The results are summarized in Table 4.

Table 4 Critical task success for NAI and NXN
Full size table

The Critical Task Errors for NAI and NXN are shown in Table 5. During phase 1 there were four Critical Task Errors for NAI. One error involved injection into an incorrect muscle, the calf, and three individuals failed to hold the auto-injector in place after the injection for at least one second. There were no Critical Task Errors for NAI during the phase 3 simulations.

Table 5 Critical use errors for NAI and NXN
Full size table

For NXN, there were a total of 110 Critical Task Errors during Phase 1 simulations. None of the participants attempted to administer naloxone into both nostrils and 35 failed to administer naloxone into a single nostril. In addition, 32 participants leaked drug during assembly, 7 did not assemble the naloxone cartridge into the syringe correctly, 27 failed to attach the atomizer to the syringe, and 2 did not remove the yellow protective caps from the syringe.

For phase 3 simulations with NXN, there were 31 Critical Task Errors, which included 15 failures to administer naloxone to the second nostril, 8 failures to administer naloxone into either nostril, 9 incidences of drug leakage during assembly, 4 errors attaching the naloxone cartridge to the syringe correctly, and 3 errors attaching the atomizer to the syringe correctly. Because of the sequential nature of the assembly and administration process, some errors precipitated additional use errors for NXN.

Participant Adherence to Instructions-for-Use

The instructions-for-use for each product are shown in Table 2. In phase 1 (i.e., before training), 36 of 42 (85.7%) participants successfully administered a simulated dose of naloxone according to its instructions-for-use with NAI versus 0 of 42 (0.0%) with NXN (P < 0.0001).

In phase 3 (i.e., post-training), 42 of 42 (100%) participants successfully administered a simulated dose of naloxone according to its instructions-for-use with NAI versus 0 of 42 (0.0%) with NXN (P < 0.0001). The proportion of participants who successfully administered a simulated dose of naloxone with NAI according to its instructions-for-use was significantly (P < 0.0313) greater in phase 3 (i.e., post-training) than in phase 1 (i.e., no training). The results are summarized in Table 6.

Table 6 Adherence to instructions-for-use success for NAI and NXN
Full size table

The use errors associated with failure to adhere to the instructions-for-use are shown in Table 7. There were six errors in adherence to instructions for using NAI during phase 1; in addition to the NAI Critical Task errors, two individuals held the NAI longer than one second, the minimum, but less than the 5-s countdown. There were no errors in the adherence to NAI’s instructions-for-use for during phase 3. There were 127 errors in adherence to NXN’s instructions-for-use during phase 1 and 67 errors during phase 3. The main errors in adherence to NXN’s instructions-for-use during phase 1 and phase 3 were the failure to hold the atomizer by the wings while attaching it to the syringe and the inability to divide the dose at approximately 1 mL between the two nostrils.

Table 7 Adherence to instructions-for-use errors for NAI and NXN
Full size table

Time to Complete Naloxone Administration During Phase 1 and Phase 3

The time differences for the study participants to administer a dose of naloxone to the simulated patient (Table 8) were significant when comparing NAI and NXN, with NAI task time during phase 1 being 0.9 ± 0.25 min and NXN 6.0 ± 4.76 min (P < 0.0001). A large number of participants (15 of 42) did not complete the administration tasks using NXN and by study protocol were assigned the maximum time of 15 min. This maximum assignment time was based on the likelihood that most emergency response personnel would arrive prior to 15 min since the national emergency medical service response time is approximately 9.4 min [14]. Phase 3 (approximately 1 week post-training) participants’ time differences with the simulated opioid emergency, showed both product completion times significantly improved, with NAI being 0.5 ± 0.15 min and NXN 2.0 ± 2.15 min (NAI P < 0.0001; NXN P < 0.0001).

Table 8 Time to task completion for NAI and NXN
Full size table

Discussion

The primary objective of this study was to compare the usability of NAI and NXN in the administration of naloxone during a simulated opioid overdose emergency. The currently available NXN is primarily distributed and used by first responders, police, and members of some harm reduction clinics, but is increasingly being made available by pharmacists. NAI has recently been approved by the FDA for use in the home or non-clinical, out-of-hospital setting, by caregiver and family member administration. Nasal atomization kits are an important tool to help combat the public health crisis of opioid overdose morbidity and mortality. Numerous studies have demonstrated that these kits have saved lives and that there is a place for a nasally administered naloxone product in the emergency treatment of OIRD [8, 15–20]. This includes populations who are well educated in manipulating glass syringes and who encounter frequent opioid overdoses, such as in the harm reduction community, and trained professionals such as emergency medical technicians, law enforcement, and other trained first responders who are adept at responding to stressful, emergency situations.

Nasal atomization kits and their accompanying instructions-for-use have not been subject to rigorous Human Factors Engineering development and testing to evaluate the usability of these kits by the general population, especially by family members or caregivers of chronic opioid users. In addition, the safety and efficacy of the intranasal route of naloxone administration has not been subject to FDA review and approval, including in special populations such as pediatric patients suffering from unintentional opioid poisoning. Naloxone is effective in the treatment of opioid overdose only if it is: (1) available for administration, (2) administered correctly by the user, and (3) administered in a timely fashion as early intervention is often the determinant outcome when faced with a life-threatening OIRD event [21].

Data presenting success of the nasal naloxone kits have been primarily based upon qualitative, observational surveys of needle exchange clinics and other harm reduction centers that distribute naloxone kits [22]. These surveys collect data based on responses to questions including the number of overdose reversals reported over a specific period of time, but do not ask questions regarding the number of patients requiring alternative treatment due to incorrect usage of the naloxone nasal kit or the number of overdoses that occurred despite the naloxone nasal kit being dispensed, but not readily available during the overdose. Despite initial enthusiasm for the naloxone kit products and evidence of their life-saving potential, unanswered questions remain regarding their availability, usability and effectiveness in treatment of suspected opioid overdose.

One challenge associated with generic naloxone products, including vials and syringes as well as the NXN kit is the lack of unified instructions-for-use or patient counseling information [23]. In the case of this study, the NXN component product labeling [23] may have confused users, contributing to Critical Task Errors. For example, the carton in which the naloxone is provided for NXN describes use by “intravenous, intramuscular, or subcutaneous” parenteral routes and the pre-filled syringe looks like a delivery system intended for parenteral administration (Fig. 1) [23]. This is an inherent flaw associated with the NXN kits in which the user is provided written instructions and visual cues that are contrary to the intended nasal use of the product. Therefore, without the benefit of training, many participants attempted to mimic parenteral administration, confusing the atomizer as a needle. Moreover, the naloxone carton has a small surface that provides limited instructions for assembly, but since the atomization device is included separately, the instructions have no guidance for its further assembly onto the pre-filled syringe to create the finished NXN [23]. Consequently, some participants either failed to attach the atomizer, or assembled it incorrectly including cases where either the atomizer or the glass cartridge of naloxone was damaged (e.g., broken glass) rendering administration impossible. In addition, in many cases participants declared that they had completed the task, believing they had performed a successful administration of naloxone with NXN when in fact they had not been successful. Even after one-on-one, personalized training and demonstration of correct administration technique, a substantial proportion of users were still unable to correctly administer NXN.

By comparison, NAI contains labeling and voice instructions that facilitated correct administration. NAI is self-contained, with instructions-for-use, including text and symbols on the flat surface of the product (Fig. 2) [12]. The product includes a voice instruction system that assists in guiding users through the correct administration process. Therefore, although the supplemental instructions-for-use documents were not provided to the participants, the participants were able to rely on instructions built into the product. In addition, the administration of NXN required eight steps compared to five steps for NAI (Table 2). For NXN, some of the defined steps required multiple actions such as removing two caps from the syringe or the inherent need to remove the different components from different packages. Therefore, participants had more opportunities to fail in adhering to instructions or completing critical tasks.

The results of this study should be viewed in light of the limitations of the study and its specific design to test a “worst-case” scenario. First, the assignment of Critical Task Errors for NXN required the determination of a “clinically meaningful dose.” Because nasally administered naloxone is not an FDA approved route of administration, the selection of the dose was based on published literature in which a full dose of 1 mL is administered into each nostril [8, 15–18]. Several studies describe the need for additional doses of naloxone, therefore, the study team viewed the full dose of NXN as the minimally effective amount [24, 25]. Moreover, administration into both nostrils is listed as a required step in the instructions-for-use and in the published clinical studies. Therefore, administration into both nostrils was deemed a critical task; however, the study team permitted administration of a partial dose to both nostrils to meet the standard for a critical task, while administration of approximately 1 mL into each nostril was required for adherence to instructions. It is unknown if administration of some amount of naloxone into a single nostril is efficacious in some or all suspected overdose cases, and it is unknown how much of the full dose (2 mL) is required to be administered into the two nostrils to result in a clinically meaningful benefit. Second, because this study was designed to evaluate a “worst-case” scenario, both simulation phases of this study, before and after training, were conducted without the benefit of instructions-for-use leaflets for the participants. There were numerous errors associated with the assembly and administration of NXN which may have been mitigated if the users had the benefit of clear visual and written instructions, especially instructions that replaced those that were on the outer packaging of the naloxone glass syringe. However, the likelihood of users for NXN and NAI to both have access to instructional leaflets as well as refer to instructional leaflets during an opioid overdose emergency is low given the nature of an opioid overdose emergency.

Despite these study limitations, the data demonstrates that in a setting where a layperson is required to administer naloxone, they have a significantly higher success rate at administering a clinically meaningful dose using NAI versus NXN, and they complete the task utilizing NAI in less time than is required using NXN. The improved completion time from phase 1 to phase 3 for NAI and NXN underscores the effect training has on a user’s ability to quickly administer each product for life-threatening OIRD. The success rate and time-to-administration favor the use of NAI especially in cases where participants are likely to have received limited or no training. Even after training on the proper use of the two products, results were also statistically significant in favor of NAI over NXN in terms of both success rates of administering a clinically meaningful dose and time of completion.

Finally, all study participants reported that they would have confidence in using NAI in an actual emergency whereas only one individual, post-training, indicated he would be confident in using NXN during a suspected opioid overdose emergency. Participants attributed their confidence in using NAI to the limited number of steps required to complete an injection, the helpful voice instructions, and the countdown that indicates when the injection is complete.

Conclusion

This study demonstrated that during a simulated opioid overdose emergency there was a statistically significant difference in a participant’s ability to successfully administer a simulated dose of naloxone with or without training or instructional leaflets using NAI as compared to the NXN. Additionally, this study demonstrated that no participant could correctly administer the NXN without training. Although both products resulted in a statistically significant reduction in time to administer a dose following one-on-one training by a healthcare professional, this study found that less than 60% of participants were able to use the NXN as compared to 100% with the NAI. Residual use errors after training for NXN were attributed to difficulties assembling or handling the device, or remembering to administer the naloxone into both nostrils of the simulated overdose victim.

This is the first study that has evaluated the usability of naloxone atomization kits currently distributed and used in the United States, with or without comparison to other naloxone delivery methods. Assembly and use of these kits is subject to numerous Critical Task Errors in which the person administering the naloxone would not provide benefit to the opioid-related overdose victim. This study indicates that delivery systems, when adapted for alternate uses without FDA scrutiny and approval, may be subject to use-related errors that could result in a patient not receiving a potentially life-saving dose of medication. Additionally, this study demonstrates that laypersons are at risk of failure to properly use the intranasal naloxone delivery system under a simulated emergency environment.

Early administration of naloxone can be lifesaving during a suspected opioid emergency. However, a naloxone formulation can only be effective during a stressful suspected opioid emergency if it is readily available and administered correctly by the user in a timely manner. Naloxone distribution may become more widespread in the community due to the continued increase in opioid-related morbidity and mortality. This study demonstrates it is critical that careful design and Human Factors Engineering be incorporated into the development of potentially life-saving naloxone products intended for use by laypersons in community settings such as the home. Identification of Critical Task Errors early in development permit a risk evaluation and the opportunity to re-design elements that are subject to the most substantial use-related errors, or associated with the greatest risk.

This study validated the ability for both trained and untrained individuals to safely and effectively administer NAI which fulfills the medical need for a naloxone product that allows laypersons, including family members and caregivers, to administer naloxone hydrochloride in the home or non-clinical setting during a potentially life-threatening opioid emergency.