1 Background

Studying pain in a clinical setting can be difficult due to its subjectivity and the lack of meaningful and reliable assessment tools. Currently, patient-reported pain is typically captured using tools such as verbal or paper questionnaires. Although these can be relatively easy to implement, the data collected relies heavily on patient-recall, which introduces reporting inaccuracies. In addition, paper diaries provide no fool-proof mechanism to verify when reports were completed relative to a pain episode, making it impossible to determine if a patient is compliant with reporting requirements and if their data is accurately capturing the episode [1, 2]. The relative timing of reporting becomes especially important when studying the effect of interventions for treating pain. The more rapid migration from paper diaries to eDiaries has been driven by this need for greater data accuracy and compliance, data integrity, data security and more effective patient monitoring especially as clinical trials become more decentralized [3,4,5]. Regulatory agencies such as the U.S. Food and Drug Administration are also placing greater value on patient-reported outcomes (PRO), and especially, ePRO for data collection to support labelling claims such as pain [6, 7]

In addition, chronic pain studies typically rely on pain reported by patient recall at baseline and at some pre-defined future timepoint(s) (e.g. 90 days post-baseline), with responder rates being calculated based on the change in pain across those discrete intervals [8,9,10]. At times, these intervals may be more frequent (e.g. weekly) [11] but may also only focus on one pain score (e.g. average daily pain) [12]. Pain reporting methodology such as these can work with devices like Spinal Cord Stimulation (SCS) or Peripheral Nerve Stimulation (PNS) because the devices are in the “on” state continuously. For devices that are not “on” continuously (e.g. on-demand use), this paradigm of pain data collection would not only exclude sessions between visits, but can also exclude capturing other pain scores throughout the day. Additionally, this design would omit valuable information about each session’s impact on a patient’s pain profile as well as a more detailed understanding of daily pain changes over time for devices used on-demand.

Prior to the QUEST Study, the Altius® High Frequency Nerve Block (HFNB) System was studied in a 10 subject pilot study for chronic PAP. Subjects used a paper diary to report pain intensities before and after each on-demand treatment session [13]. Although validation was performed to compare timing of the diary pain levels with a corresponding session, paper diaries made it difficult to accurately cross check timing of reports with device uses. This made it impossible to determine if entries were reported in “real-time” (subject reports pain episode as the event occurs), thereby determining the direct impact of individual sessions on the subject’s pain profile.

Therefore, the QUEST Study – a multicenter, double-blinded, randomized, active-sham controlled clinical trial designed to evaluate the safety and effectiveness of the Altius® System and it’s on-demand treatment sessions for severe chronic PAP – required a unique electronic data capture system to effectively track and validate individual session impact on subject’s pain. The data capture system also allowed for collection of multiple pain scores at the end of the day (e.g. daily worst, least, and average pain) for a more complete understanding of a patient’s daily pain profile over the course of the entire 12-month study period. To our knowledge, QUEST is the first longitudinal trial to capture repeated measurements in “real-time” alongside multiple end-of-day pain measurements using ePRO to study chronic PAP (patented system) [14, 15]. This manuscript discusses the use of a mobile phone with a custom ePRO application within QUEST, along with data capture processes and results emphasized through the 3-Month Primary Endpoint period. It discusses system design and implementation, study milestones around the eDiary, data analysis using validated sessions, key learnings, along with potential suggestions for improvement in future work.

2 Methods

The QUEST study was conducted in compliance with the U.S. Code of Federal Regulations. The study protocol and informed consent forms were approved by the Western Institutional Review Board (IRB) along with each site IRB in accordance with the Declaration of Helsinki [14].

2.1 Study population and screening

Adults (≥ 21 years-old) with unilateral, lower-limb amputation suffering from severe, chronic (persistent > 6 months) PAP refractory to previous pain management treatments were enrolled in the study. Additional details on the screening process outside of the eDiary portion were previously published [14].

Once enrolled, subjects were assessed on their ability to report their pain levels, pain medication use and prosthetic use during the eDiary Eligibility phase. Subjects were given an eDiary for four weeks and were required to report pain scores while using their medication for their PAP, along with their worst, least, average, and current pain at the end of the day (End-of-Day report). Subjects were also asked to report their intake of medication for breakthrough pain (rescue) and scheduled (routine) pain medication. Subjects needed to demonstrate compliance via two consecutive weeks of reporting (≥ 12 days out of 14 days of daily End-of-Day Reports completed) and frequent episodes of significant pain levels (≥ 8 days out of 14 days with End-of-Day Worst Pain ≥ 5) as shown in Fig. 1. Subjects would progress to the next phase of the screening process if they met both criteria. Oversight from site coordinators, Neuros, and the ePRO vendor (24/7 helpdesk) were in place if subjects had technical challenges or needed additional clarification on reporting.

Fig. 1
figure 1

eDiary Eligibility phase. Subjects proceeding on with initial screening phase would need to demonstrate compliant reporting and adequate pain frequency to move to the nerve block injection phase of screening ”Compliant Reporting” passing criteria was ≥ 12 out of 14 days of daily End-of-Day Reports completed. “Enough Pain Days” passing criteria was ≥ 8 days out of 14 days with End-of-Day Worst Pain ≥ 5

Full size image

2.2 Site coordinator training prior to eDiary issuance

Neuros and the ePRO vendor (Axiom Real-Time Metrics, Mississauga, ON, Canada) were responsible for training site personnel prior to participating in QUEST. The training focused on the eDiary system, subject account creation and log-in process, subject eDiary issuance, and how to report on the ePRO application. The phones used by site coordinators during training had a version of the ePRO application that was linked to a test site account (trainer eDiary); otherwise, this was an exact replica of the subject eDiary. Coordinators were required to complete each type of report as if they were a subject, gaining firsthand experience entering the data and receiving notifications for entering reports. Coordinators were also trained on best reporting practices, including reporting in “real-time” per notifications, reporting in areas with poor/no connectivity, and contacting the vendor helpdesk.

2.3 eDiary design overview

QUEST subjects reported their use of pain treatment daily. If participants were prescribed rescue medications for breakthrough pain, they were instructed to report each time they used the medication. Each report would include the medication name, dose, time of administration, along with their pain level at the time they took their pain medication using the 0–10 Numerical Rating Scale (NRS). Subjects were prompted by audio and visual text notifications to report their NRS scores at 30 min and at 120 min after initial use of their rescue pain medication. Once participants were implanted and randomized, they reported their daily uses of the Altius® system [14]. Device Treatment Report functionality was provided to subjects on the day of randomization so they could start reporting their Altius sessions that same day (Day 14 visit, Fig. 2).

Fig. 2
figure 2

QUEST study schedule with eDiary reporting periods notated, along with primary and secondary endpoints captured during Randomized Testing

Full size image

Subjects reported their NRS pain level upon use, along with a confirmation that the device was turned on. Participants were prompted by audio and visual text notifications to report their NRS scores at 30 min and 120 min after use of the Altius® System. At the end of each day, subjects reported their daily pain scores (worst, least, average, “pain right now”), along with their use of routine pain medication (scheduled) and prosthetic use. Reporting from all subjects was completed in “real-time” because the ePRO application did not permit retrospective recollection of treatment report pain levels. The subject treatment workflow is described and depicted in Fig. 3.

Fig. 3
figure 3

Device Treatment Report and Rescue Medication Treatment Report subject workflow along with notification sequence to alert subjects to complete follow-ups. Report components are seen by subject in order from 1 to 6 above. Notifications for each report could extend for 15 min, where subjects could receive them at the follow-up time, 5 min post, 10 min post and 15 min post follow-up time, depending on how long after the initial notifications that the subject reported

Full size image

2.4 eDiary design workflow specifics

The ePRO application outlined in Fig. 3 was designed to capture rescue pain medication uses in a Rescue Medication Treatment Report and Altius device uses in a Device Treatment Report. Subjects would initiate each report by selecting “I Need Treatment for Pain” and proceed to select which treatment they were using, as well as report their initial pain score. Subsequently, 30 min after the initial portion of the report was initiated and transmitted, the eDiary would alert the subject with audio and visual text notifications to enter their pain score. If the subject did not immediately enter the pain scores, reminder alerts were sent every 5 min for 15 total minutes, after which the subject could not complete the entry. This 30-min pain score was the primary endpoint collection mechanism of QUEST. Subjects would receive similar alerts at 120 min only if a 30-min report was previously completed. This design was intended to focus on primary endpoint data collection and not reporting this pain score would mitigate the importance of the 120-min report. Subjects could also “interrupt” an existing workflow with another if they needed additional treatment. For example, if subjects reported their rescue pain medication use, and they decided to use the Altius Device within 120 min of initiating the Rescue Medication Treatment Report, they could then start a new Device Treatment Report, which would become the active workflow. They could not enter data into the previous workflow once it was interrupted. The interruption workflow was also designed to mirror the Altius device design of one 30-min treatment session followed by a 30-min lockout period, allowing for one use per hour. Site coordinators adjusted Device Treatment Report access on the subject account in the ePRO application.

At the end of each day, subjects would receive a notification to complete their End-of-Day Report, where they would enter their Worst, Least, Average and “Pain Right Now” pain scores, along with their use of routine pain medication and duration of prosthetic use. Once this report was completed and sent successfully to the database, it would become unavailable until the same time the following day. If the subjects did not report after the first notification, they would receive alarms at 5 min, 15 min, 30 min, 60 min, and 120 min post their selected End-of-Day alarm time until they completed the report. The subjects would have until 11:59 pm local time to complete and submit their reports. The eDiary system allowed subjects to select their preferred time to complete this report between 6 and 9pm per their local time zone. Supplement 1 (eDiary Daily Use Guide) contains step-by-step reporting instructions provided to subjects and corresponding ePRO application screenshots.

2.5 Data acquisition and compliance

The Full Analysis Set (FAS) cohort was comprised of the 170 subjects that were implanted, randomized successfully at their Day-14 visit (85 subjects in both groups, Treatment and Control) and reported Altius Device uses on their eDiary during the Randomized Testing phase. Data acquisition and flow for these subjects is shown in Fig. 4.

Fig. 4
figure 4

Diagram for QUEST data flows. Subjects report on their eDiary (At Home Use) where reports were submitted to the Fusion database. Fusion also contained electronic copies of IPG logs gathered at programming visits (Study Visits). Within Fusion, IPG sessions were paired with eDiary sessions. Data from these paired session reports were then used for the primary and key secondary endpoint analyses. End-of-Day Reports provided additional medication and prosthetic use information for secondary endpoint and supplemental analyses. Additional data were captured at study visits via case report forms and entered into Clindex EDC. These data were then used to analyze all primary safety endpoints and other event data

Full size image

After each Treatment Report and End-of-Day Report were completed, they were sent to a (US FDA Title 21 Code of Federal Regulations) Part 11-compliant database (Fusion) in “real-time”. FDA regulations compliance for data integrity, data security, data confidentiality, audit trails and electronic signatures. Once subjects came in for study visits, which included programming adjustments, a Neuros Field Clinical Specialist (FCS) would download the usage logs from the Altius Implantable Pulse Generator (IPG). The logs on the device would automatically maintain the information for each device use, such therapy initiation start time and the session duration. The FCS would then upload the IPG logs electronically to the Fusion database. The logs were then paired algorithmically (designed by Neuros and Axiom) with the Device Treatment Report data during the same time window. The pairings were categorized based on the device initiation time and the initial Device Treatment Report time. A report of all pairings for a subject at a programming visit was automatically generated from the Fusion database and then used by the Neuros FCS to review subject compliance based on the timings of eDiary and device pairings. These reports would maintain prior pairings and add subsequent pairings with each programming visit. The goal was to ensure a close oversight of reporting of all device uses in “real-time”.

Additionally, the End-of-Day Reports, along with Rescue Medication Treatment Reports were used for secondary endpoint medication analyses performed by Avania Clinical LLC (Marlboro, MA) and Neuros. End-of-Day Reports also contained prosthetic use information which was utilized for subsequent supplemental analysis. Information around each subject visit was captured on case report forms and entered into the Clindex Electronic Data Capture (EDC) system (Fortress Medical, Hopkins, MN). All safety and event analyses data were used from the Clindex EDC and analyzed by Neuros and Avania, with analysis plans and study design for The QUEST Study published previously (clinicaltrials.gov Identifier NCT 02221934).

2.6 Derived datasets and analysis

All comparison reports were imported and filtered for all pairings that occurred during the Randomized Testing Phase for each subject (Fig. 4, endpoint analysis). Visit dates captured in Clindex EDC (Day-28 to Day-91, Fig. 4) served as the time window for which the comparison reports were selected. For a session to be counted as a qualifying session and included in the 3-month primary effectiveness endpoint analysis, a Device Treatment Report needed to be initiated within 15 min of Altius device use. For this pair to be included in the responder rate analysis, the Altius session also needed to last for 30 min for the Treatment group, or 7–9 min for the Control group. Additionally, Device Treatment Report initial pain needed to be ≥ 4 on 0–10 NRS and a 30 min follow-up report had to be able to calculate a pain reduction score for that session. If the 30-Minute follow-up was missing, the session was counted as a failed session not meeting the ≥ 50% pain reduction criteria. Duplicate sessions were removed from the analysis. All pairings outside of the 15-min pairing window were used for additional sensitivity analyses.

The 3-month primary effectiveness endpoint was the proportion of responders in the Treatment therapy program group vs the Control active-sham program group at 30 min. Specifically, a responder needed to report ≥ 50% pain reduction in ≥ 50% of the treatment sessions. This algorithm ensured that the pain data were associated with a device session to directly quantify the session’s impact on the subject’s pain profile and responder rate.

3 Results

3.1 eDiary eligibility subject accountability

Table 1 contains the results of the eDiary Eligibility screening period of QUEST.

Table 1 eDiary Eligibility subject accountability for the 497 subjects that met the criteria for assessment
Full size table

In total, 497 subjects reported for at least two weeks allowing a determination to be made. Of those subjects, 82% (410) passed and moved onto the next phase of the study, whereas 18% failed (87) and exited. Most subjects that failed were due to non-compliant reporting (61) rather than not meeting the pain level requirements (26). The average number of days to pass starting on the day that informed consent was signed was 18.2 days with a standard deviation of 11.4 days, and a median of 14 days.

3.2 Randomized testing dataset size and compliance

In total, 14,337 Device Treatment Reports (170 subjects) were captured by the FAS cohort during Randomized Testing per Table 2.

Table 2 Randomized testing phase sessions reported by the FAS cohort (170 subjects)
Full size table

Of those Device Treatment Reports, 66.4% of sessions were used for the primary effectiveness endpoint analysis (9542). Additionally, 56% of all sessions had a 30 min pain score and were evaluated as a responder or non-responder. The 10.4% difference was sessions that were categorized as failures that did not have a 30 Minute follow-up completed.

Table 3 explains subject compliance levels achieved during the Randomized Testing phase. A compliant report was defined as a Device Treatment Report with its initial portion completed within 15 min of an Altius Device use. Subjects were grouped based on the percentage of reports that met the criteria.

Table 3 FAS eligible subject reporting compliance during Randomized Testing Phase
Full size table

Of the FAS cohort, 52.9% of subjects (90/170) met the criterion in at least 85% of their reported initial Device Treatment Report sessions, and 59.9% (100/167) of subjects completed at least 85% of their End-of-Day Reports. The cohort completed 77% of all initial Device Treatment Reports and completed nearly 83% of all End-of-Day Reports during this period.

3.3 QUEST eDiary data capture

eDiary data were captured during the entire duration of the study (2014–2022). Each subject reported for up to approximately 12-months or until study exit, whichever came first. The QUEST eDiary system captured data from 548 subjects across 30 clinical sites shown in Table 4. The dataset included all subjects that reported any data while enrolled in QUEST.

Table 4 QUEST dataset stratified by number of data points and report type
Full size table

The QUEST eDiary dataset consisted of 1,773,356 data points from 197,952 reports. For example, each self-initiated treatment report for pain was comprised of 3 sections (initial, 30-minute follow-up, 120-minute follow-up), with each section potentially resulting in 3 to 5 data points at minimum. Treatment Reports comprised the largest segment of data at 53% of total data points (947,519) and 67% of total reports (131,924). Device Treatment Reports accounted for 62% of all Treatment Report data points (583,727) and 69% of all reports (90,831). End-of-Day Reports comprised 47% of data points (825,837) and 33% of reports (66,028). The QUEST Study results (clinicaltrials.gov Identifier NCT 02221934) were published previously. The results showed significantly more responders in the Test Group compared to the Control group based on the pain data captured with the ePRO system, achieving the primary effectiveness endpoint.

4 Discussion

4.1 System design

The QUEST Study, results (clinicaltrials.gov Identifier NCT 02221934) and design published previously, warranted a unique and robust means of pain data capture and analysis [14]. The ability to collect this data over time was demonstrated initially in a 10 subject Pilot Study; however, collecting data with a paper diary was suboptimal given the need for accurate, “real-time” data (data sent to the database and able to be viewed within minutes of subject report completion), across a larger study population size and number of clinical centers [13]. Subjects in the QUEST FAS cohort reported their pain continuously from their initial enrollment through 12-months, including daily pain scores (worst, least, average, “pain right now”), where most pain studies typically take fewer measurements at discrete time points and comparing these against each other to calculate responder rates. The unique, patented system validated individual device uses with “real-time” data collection to a subject’s pain profile within 2 h, along with daily tracking of a subject’s worst, least and average pain resulting from the device uses [15]. We believe that this level of detail allowed us to uncover more definitively over time that subjects can see improved pain reduction on a daily and a per session basis that amounts to long-term overall pain reduction. To our knowledge, this is the first study to follow subjects PAP longitudinally using multiple, repeated measurements and provide as comprehensive of a view of subject pain profile over time [14, 16].

4.2 Key compliance learnings

The QUEST Study also provided several key learnings for subject engagement and compliance. Subjects averaged approximately 361 reports (197,952 Reports per 548 subjects) during their time in QUEST (minimum 1 report, maximum 4,977 reports). Subjects that reported during Randomized Testing averaged approximately 85 Altius Device use sessions (see Table 2, 14,377 sessions for 170 subjects) over the 2 months (minimum 7 sessions, maximum 444 sessions). Subjects also averaged approximately 122 End-of-Day Reports (66,028 reports per 541 subjects) during their time in QUEST (minimum 1 report, maximum 852 reports). The study population stayed engaged overall, even with concerns of reporting fatigue Sponsor field staff also helped to maximize reporting compliance by using comparison reports to discuss proper data entry habits up to the subject’s most recent device use.

Neuros and the vendors also found certain design considerations could impact compliance. The timing of available reports with the notification firings to be a significant challenge in the design process, and something that could negatively impact compliance if subjects did not receive them as expected. Subject co-morbidities and technical acumen could also impact compliance. The eDiary Eligibility phase was also critical for ensuring a level of compliance as it was the first touch point the subjects had with the mobile application and the reporting expectations. Considering all these factors, compliance rates were relatively high and led to robust data capture. From our review, we did not find any truly comparable studies that measured eDiary compliance rates. These values however do align with values in the literature for Randomized Control Trials for other treatment modalities [17]. As a potential improvement, Bring Your Own Device (BYOD) could be implemented in future work and provide additional advantages with subjects including using their own smartphone, reducing the burden of carrying an additional phone and a potential increase in compliance [18, 19].

5 Conclusions

The QUEST Study is the first large multicenter, double-blinded, randomized, active-sham-controlled trial to capture repeated, longitudinal measurements for subjects with chronic PAP with HFNB. As such, a unique data capture and analysis system was created that showed not only that the primary outcomes of the study were met but that it could provide comprehensive “real-time” pain data over time. These data, combined with daily End-of-Day pain reporting, provided a comprehensive view of subject pain data longitudinally that we have not yet seen in the literature, despite subject health challenges and ePRO design consideration challenges. We see process improvements in future work such as optimization of ePRO notifications and the potential use of BYOD for patients that prefer to use their own mobile devices.