Abstract
Introduction
The Reporting recommendations Intended for pharmaceutical risk Minimization Evaluation Studies (RIMES) was developed to improve the quality of reporting of risk minimization program evaluations. In light of continued inadequacies in study reporting, and high-profile program implementation failures, we updated the RIMES Checklist to incorporate additional concepts from the Standards for Reporting of Implementation studies (StaRI).
Methods
The development of the updated checklist, the RIMES-StaRI Extension (RIMES-SE), entailed developing a study protocol and drafting an initial pool of items based on a mapping of the RIMES against the StaRI checklist. A modified e-Delphi exercise was then conducted to determine the importance and understandability of items for checklist inclusion. An expert workshop and an online commentary period for additional feedback followed.
Results
The RIMES-SE contains 27 items. It includes two signature features of the StaRI Checklist: 1) a dual strand of items (represented in two columns) describing the risk minimization program (the ‘intervention’) and the corresponding implementation strategy; and 2) applicable to an array of different research methodologies.
Conclusions
The RIMES-SE Statement and Checklist extends the reporting guidelines set forth in the original RIMES Checklist via inclusion of key implementation science concepts. It is intended to improve the quality and transparency of reporting of risk minimization evaluation studies so as to advance drug safety science.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
The Reporting recommendations Intended for pharmaceutical risk Minimization Evaluation Studies (RIMES) was published in 2018 to improve the quality of reporting of risk minimization evaluation studies. |
Evidence that the reporting of these studies continued to be inadequate, and that risk minimization programs were being poorly implemented, prompted us to update and harmonize the RIMES with constructs from the Standards for Reporting of Implementation studies (StaRI) Checklist, a best practice standard for reporting implementation science studies in public health. |
The 27-item RIMES-StaRI Extension (RIMES-SE) was derived by the use of rigorous methodology. It has a dual strand of items that focus on both the risk minimization program (‘the intervention’) and the strategy used to implement it and it is applicable for studies using a heterogeneous range of designs. |
Adoption of the RIMES-SE Checklist by researchers and endorsement by journal editors should improve the reporting of risk minimization studies, enhance transparency, facilitate researchers’ ability to identify these studies, and promote synthesis of the available evidence base. |
1 Introduction
The effectiveness of pharmaceutical risk minimization programs in protecting public health has been the subject of intense regulatory scrutiny over the past decade. These programs, which represent a population-based type of drug safety measure, are mandated by regulators for products with serious risks, either as a condition of marketing authorization or to investigate safety information newly identified in the post-market period [1, 2]. Under the terms of the regulatory commitment, these programs must be evaluated to determine whether they have been implemented as intended and are effective and sustainable in the context of real-world clinical care.
Risk minimization programs can be defined as ‘complex interventions’ [3, 4]. Characteristic features of complex interventions are that they have several interacting components, require numerous behaviors by those delivering or receiving the intervention (e.g., healthcare professionals [HCPs], patients), target different groups or organizational levels (e.g., individual, healthcare setting, healthcare system), involve a range of outcomes, and may require some flexibility or tailoring of the intervention to optimally fit within different healthcare settings [4].
When designing and evaluating complex interventions, it is essential to specify the theoretical (or empirical) basis for how the program is expected to achieve the desired effects. Similarly, it is important to conduct both a formative evaluation (to determine intervention feasibility and acceptability) and a comprehensive process evaluation (to determine the quality and fidelity of program implementation, and whether any adaptations occurred). A process evaluation is especially valuable when program evaluation results show lack of effect as it can clarify whether this was due to problems in program implementation or to the ineffectiveness of the program itself [4]. Lastly, a range of measures are needed to fully understand program effects, including measures of reach, adoption, impact, and ongoing maintenance, as well as whether the program was successful in reducing the incidence of the targeted risk(s), imposed undue burden on the healthcare system, impeded patient access to the drug, or had other unintended consequences [2, 4].
In recent years, three high-profile assessments of risk minimization programs—one in Europe for valproate products, and two in the United States (US) (one generally, and a second for opioid analgesics specifically)—revealed that program implementation had been incomplete and evaluation data insufficient [5,6,7]. Similarly, published reviews have shown that both the quality and comprehensiveness of risk minimization evaluation studies are highly variable, assessment of program implementation and context often inadequate, and findings regarding program effects limited or uncertain [8,9,10,11].
In an effort to build the evidence base in this area, a group of researchers under the auspices of the International Society for Pharmacoepidemiology (ISPE) developed a quality reporting checklist called the Reporting recommendations Intended for pharmaceutical risk Minimization Evaluation Studies (RIMES) [12]. The RIMES Checklist consists of 43 items and was designed to guide standardized, comprehensive, and transparent reporting of risk minimization evaluation study results. The RIMES was intended to be reviewed and updated periodically to remain abreast of the evolving science and regulatory guidance in this area.
Since its publication in 2018, the RIMES Checklist has had a significant impact on the field of drug safety. First, it engendered two comprehensive reviews of risk minimization evaluation studies, one focusing on studies published in the peer-reviewed literature, and a second focusing on risk minimization evaluation study reports submitted to the European Medicines Agency’s (EMA’s) Pharmacovigilance Risk Assessment Committee (PRAC) [13, 14]. These reviews highlighted numerous shortcomings in risk minimization reporting, including that evaluations rarely referenced the use of theories, models, or frameworks to guide program design and evaluation, and descriptions of program implementation, adaptations, and context were inadequate or missing.
Second, prominent pharmacovigilance organizations in Europe, including the EMA and the European Network Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) [1, 15], cited the RIMES Checklist in relevant guidance documents and recommended that it be used when reporting the results of risk minimization evaluation studies. A translated version of the RIMES Checklist along with guidance on how it should be applied were also published in a Chinese pharmacovigilance journal [16].
Globally, healthcare delivery organizations, including pharmacovigilance bodies, have been facing mounting pressures to transform into learning healthcare systems [17, 18]. Implementation science, defined as “the scientific study of methods to promote the systematic uptake of research findings and other evidence-based practices into routine practice,” is instrumental to such a transformation [19]. Consistent with this, and contemporaneously with the development of the RIMES, a new checklist, the Standards for Reporting of Implementation studies (StaRI), was published. The StaRI Statement and Checklist were intended to guide standardized, transparent, and complete reporting of IS research [20].
The application of implementation science for drug safety and risk minimization has been gaining increasing recognition [3, 19, 21,22,23,24,25,26]. Recently, pharmacovigilance regulatory guidance documents have begun incorporating concepts, constructs, and terminology from implementation science that are consistent with recommendations for evaluating complex interventions [1, 2, 4].
The purpose of the current work was to further refine and harmonize the RIMES with updated frameworks from implementation science, namely, by reviewing and including relevant items from StaRI to create a RIMES StaRI extension (RIMES-SE).
1.1 Scope and Relationship with Other Reporting Standards
The RIMES-SE fills a niche in the growing array of quality checklists developed for the reporting of healthcare research [27]. The field of therapeutic risk minimization seeks to implement, scale up, and maintain effective interventions (e.g., behavioral, educational, healthcare process improvements) in the context of real-world clinical care to minimize the harmful effects of exposure to product-related risks. The programs are mandated to be implemented in full according to the terms of the marketing authorization commitment at the time of product launch. As a result, randomized experimental designs are not feasible and evaluators must employ non-experimental study designs (e.g., observational, time series, and/or mixed methods approaches) for program evaluation purposes. Given the variety of drug-related risks that may be targeted for minimization and the diversity of HCPs, healthcare settings, and patients involved, a heterogeneous array of data sources and data collection methods are appropriate and relevant to use.
Existing checklists for observational research in healthcare (i.e., STROBE [28]), or for research using routinely collected health data (RECORD [29]; RECORD-PE [30]) focus on a limited set of study designs, methods, and data sources. Other checklists developed for reporting quality improvements in healthcare (SQUIRE [31]) or for reporting behavioral and public health evaluations using non-randomized designs (TREND [32]) are not fully applicable due to their lack of emphasis on program implementation and maintenance.
2 Methods
Extending the original RIMES Checklist involved a five-stage process (Fig. 1) guided by the methodology described in the Development Health Research Reporting Guidelines [33].
RIMES-SE: consensus process diagram
2.1 Initial Steps
In 2023, a team of experts in risk minimization and/or implementation science (MYS, EHM, HP, and AW) convened to review the RIMES Checklist and guide the e-Delphi review process. Two experts (MYS and EHM) were members of the original RIMES authoring team; a third member (HP) was the lead author of the StaRI Statement; and a fourth member (AW) had chaired the FDA’s Drug Safety and Risk Management Advisory Committee. We developed a protocol to describe the development process and publicly posted it on the EQUATOR website [27].
2.2 Pre-e-Delphi Activities
We mapped the items from the RIMES (n = 47) and StaRI (n = 27) against each other to converge on similarities and determine gaps in the RIMES (Table 1). We then developed a pool of items for potential inclusion in the RIMES-SE . In doing so, we eliminated the StaRI item (#20) on resource costs and economic outcomes as not being applicable as financial measures are not regulatory considerations and drug manufacturers view this information as proprietary and hence do not make it publicly available.
2.3 Defining Concepts
The RIMES-SE shares two signature features with the StaRI Checklist. The first feature is the dual strand of items (represented in two columns) describing the intervention (i.e., the risk minimization program) and the corresponding implementation strategy [20]. In contrast to the StaRI Checklist, however, which places primary emphasis on the implementation strategy, the first strand of the RIMES-SE refers to the risk minimization program and the second strand refers to the supporting implementation strategy [20]. Items in both columns should be populated; however, if any information is unknown to the evaluator (as can often be the case for third-party evaluators such as academically based research groups), there is an option to report ‘information unknown.’
In the RIMES-SE Checklist, the term ‘implementation strategy’ is meant to encompass the collective set of activities that the product manufacturer and local HCPs, healthcare site staff, and third-party program administrators might use to implement the risk minimization program across different geographic regions and different levels (e.g., individual, healthcare setting, healthcare system, national regulatory policy). For example, common risk minimization program implementation activities include training of company staff at local country affiliate offices, establishing centralized hubs for verifying HCP program certification status, and requiring local affiliates to develop a detailed implementation plan outlining the steps to be completed as well as the process metrics to capture.
The second signature feature is that RIMES-SE is designed to be applicable to the heterogeneous array of research methodologies used in risk minimization program evaluation [1, 2]. It is recommended that researchers consult other reporting checklists for guidance as to how to report specific types of evaluation.
2.4 E-Delphi Panel
An e-Delphi process [34] was used to select items for inclusion in the RIMES-SE and reach scientific consensus with experts engaging asynchronously in an anonymous manner.
2.4.1 Selection of Experts
We identified an international, cross-disciplinary group of scientific experts (n = 29) knowledgeable in risk minimization or implementation science with a focus in the healthcare setting.
First, we directly contacted eligible individuals within our collective professional networks. We then used a snowball sampling approach in which we asked invited participants to identify others. As risk minimization program evaluation is a specialized area within drug safety science, the pool of eligible scientific experts is relatively limited.
2.4.2 Survey Administration
Invitation to participate in the e-Delphi panel and links to the online surveys were emailed to the experts (n = 28, 96.6% intention to participate) with a deadline of 2 weeks for responding. Reminders emails were sent out 1 week before the deadline.
The experts were asked to rate the importance and understandability of each checklist item using an 11-point rating scale that ranged from 0 (not at all important/understandable) to 10 (extremely important/understandable). Open-ended, free-text responses were invited to allow for further input regarding item meaning and rationale. See Supplementary Files 1 and 2 for the e-Delphi survey instruments (Round 1 and Round 2, respectively) in the electronic supplementary material (ESM).
Surveys were administered using Qualtrics (Qualtrics.com; Provo, UT, USA) and ethical review was conducted by the Advarra Institutional Review Board (IRB), Columbia, MD, USA. For the US and Western Europe, the project was deemed to be exempt from IRB review (May 18, 2023; Pro00071189); for Canada, ethics approval was granted on May 24, 2023 (Pro00071646).
2.4.3 Analysis
Descriptive statistics were calculated for each item (median, inter-quartile range). A pre-specified threshold level of 80% agreement was used as defined by importance and understandability scores of 7, 8, 9, or 10 [35]. In round one, the experts were asked to rate all items (n = 34). In round two, the experts rated only the items that did not achieve the threshold level of agreement in round one and that had been subsequently revised to improve meaning and/or clarity. Items from both survey rounds were analyzed using Stata, version 12 (stata.com; College Station, TX, USA). Qualitative comments and suggestions were imported into a Word document for review by authoring team members (MYS, VN, and EHM).
2.5 Post-e-Delphi Panel Activities
To further evaluate understandability of the consensus items among risk minimization practitioners who will apply the standards for their evaluations, we convened a face-to-face, 1-hour workshop at the International Conference for Pharmacoepidemiology in Halifax, Nova Scotia, Canada, on August 25, 2023. Attendees were members of ISPE’s Benefit–Risk Assessment, Communication, and Evaluation Special Interest Group (BRACE SIG) [36], and possessed training and professional experience in drug safety and/or pharmacoepidemiology (n = 8), were drug regulators (n = 2), or had combined expertise in implementation science, public health, drug safety, and pharmacoepidemiology (n = 2). The group reviewed the results of the e-Delphi exercise. Workshop attendees recommended some additional, minor editorial changes to item wording and provided suggestions regarding examples to include in an accompanying explanation and elaboration document to the RIMES-SE Checklist.
Following the workshop, the proposed RIMES-SE Checklist was posted on the BRACE SIG message board hosted by ISPE. The full BRACE membership, consisting of 250 professionals working in the field of drug safety in the pharmaceutical industry, academia, or in regulatory authorities globally, was invited to review and comment for a 2-week period. The final version of the RIMES-SE Checklist was developed based on a distillation of the feedback received and iterative discussions among the authors (MYS and EHM).
2.6 Post-Publication Activities
While this is outside the scope of the present paper, the intention is to upload the reporting standard and accompanying explanation and elaboration document (see ESM File 4) onto the EQUATOR Network website to facilitate guideline translation to practice.
3 Results
Table 2 presents summary descriptive statistics for the e-Delphi process. Of the 29 experts initially approached, all but one accepted the invitation to participate. During the first e-Delphi round, 15 of the 28 experts (53.6% response rate) provided ratings on the importance of each item. Of the 34 items listed, 29 (85.3%) reached the a priori level of consensus for inclusion. In terms of understandability, 18 items (52.9%) reached the a priori level of consensus for inclusion.
Based on the results of Round 1, items with either importance or understandability scores <80% agreement were reviewed, and their wording was revised in response to comments from the e-Delphi panelists. Qualitative comments from Round 1 are summarized by theme and domain in Table 3. Exemplar quotes included “More tangible examples may be helpful.”; and “I had to read this a few times. There are a lot of concepts (all important) rolled into one” (Table 3).
In summarizing the received feedback, seven items were deemed to be redundant or irrelevant to risk minimization programs and removed altogether, leaving an item pool of 27 items.
Among the 27 retained items, only those which had importance or understandability ratings of < 80% from Round 1 were included in Round 2 (n = 19 items). The response rate in Round 2 was 53.6% (15/28). Fifteen of the invited experts also rated the items in Round 2; however, because responses were anonymous, we are unable to determine the round-to-round response rate.
Of the 19 items assessed, 18 reached the a priori level of consensus for importance (94.7% consensus). Fifteen of the 19 items reached the a priori level of consensus for inclusion in terms of understandability (78.9% consensus). The single item (item 7) which failed to reach 80% consensus on importance referred to the context in which the program was delivered, including healthcare regulations, other relevant policies, social, economic or political factors that might influence implementation. Given that context can strongly affect program implementation and external generalizability, a decision was made to retain that item.
The other four items were reworded to improve their understandability, including item 4b (the scientific rationale for the implementation strategy/ies); item 6 (the goals/objectives of the risk minimization evaluation study, design and key features of the evaluation, date of implementation of risk minimization program, and any changes to evaluation plan, with reasons); item 17a (primary and other outcome(s) of the Intervention, including precision of assessment and whether the primary outcome met a pre-specified success threshold); and item 20a (degree of fidelity involved in delivering the risk minimization program intervention elements). Qualitative comments from Round 2 are summarized by theme and domain in Table 4. Exemplar quotes regarding understandability included “It may be beneficial to distinguish RM objectives and RMP implementation objectives in the example using (A) and (B) in the first sentence and then categorize the example accordingly;” and, “As an implementation scientist, the fidelity Q[uestions]s are clear and understandable, but I think others will need more explanation.” (items 6 and 20a, respectively; see Table 4).
3.1 The RIMES-SE
The updated, 27-item RIMES-SE Checklist is presented in Supplementary File 3 (see ESM); Table 5 describes key terms used in the Checklist. Items fall into nine sections consistent with the organization of scientific manuscripts and with the reporting conventions set forth in the StaRI [20]. A detailed description of all RIMES-SE items along with a supporting rationale and illustrative examples are provided in the Explanation and Elaboration document in Supplementary File 4 (see ESM), and in the StaRI Statement’s Explanation and Elaboration document [20].
Twelve of the RIMES-SE Checklist items entail consideration of the risk minimization program and its implementation strategy together. In particular, authors are asked to specify (a) the hypothesized mechanism(s) by which both the intervention and the implementation strategy are expected to work, (b) the characteristics of the intervention recipients (e.g., HCPs, patients) as well as the implementing sites and site staff, (c) a description of the intervention components (including any stakeholder involvement in their development) as well as the implementation strategies deployed, and d) the intervention and implementation outcome measures and corresponding results.
Areas where the RIMES-SE Checklist differs significantly from its predecessor include its clear demarcation between the intervention and the implementation strategy, its focus on implementation planning, execution, and assessment as a cohesive longitudinal process, and its emphasis on describing program implementation context, including barriers and facilitators that may have affected implementation. These approaches are consistent with the emerging emphasis on designing for dissemination and sustainability from the outset [37], and on understanding contextual factors when comparing effectiveness of interventions between programs and within programs over time [38, 39].
4 Discussion
Reporting guidelines are intended to facilitate study replication and synthesis of the evidence base by stipulating a common set of items that should be reported in all study manuscripts [40]. In the context of risk minimization, there is a public health imperative to improve the transparency of reporting of evaluation studies so as to build the evidence base regarding what types of programs and implementation strategies work to reduce drug-related risks, under what types of conditions and for what types of patient and healthcare provider populations.
The RIMES-SE Checklist was developed specifically to address this gap in the reporting of risk minimization evaluation studies. It represents an advance over the original RIMES Checklist in several ways. First, it enhances ease of use by reducing the number of Checklist items from 43 to 27. In doing so, several items were deleted altogether from the original RIMES Checklist (e.g., details regarding risk minimization tool content and distribution modalities, training of implementers, and specific discussion of internal and external validity). The remainder were reworded and incorporated into new, more broadly defined items in the Methods, Results, and Discussion sections.
Second, the RIMES-SE updates the content of the original RIMES Checklist to reflect evolving best practices in the design and evaluation of risk minimization programs as set forth in regulatory guidance. Third, it emphasizes the essential inter-connection between the development of a risk minimization program and its implementation by the use of a dual strand reporting format. Fourth, it explicitly links pharmaceutical risk minimization evaluation to the field of implementation science, a positioning which may foster greater inter-disciplinary collaboration in this area, encourage increased scientific rigor in the design and reporting of risk minimization program implementation processes, strategies and outcomes, and facilitate interpretation of evaluation study results. Lastly, by virtue of the extensive consensus process we undertook, uptake of the RIMES-SE among key stakeholders (e.g., pharmacovigilance professionals, regulators, and pharmacoepidemiologist researchers) should be strong. Whether this proves to be the case, however, should be assessed in a future study.
We recognize that some RIMES-SE items may reflect approaches not yet in wide use among risk minimization program evaluators (e.g., application of theories, models, and frameworks to guide design and/or piloting of the intervention; the assessment of barriers and facilitators to program uptake; description of context of program implementation and contextual changes). In particular, in instances where the evaluators were not involved in the program’s design or implementation planning, they may not know or have access to such information. In such cases, researchers can report ‘not known.’ In other instances, where it has been determined that the approach was not used or not examined, ‘none used’ or ‘not assessed’, respectively, are the appropriate response options. Moving forward, however, we hope that risk minimization program planners will collaborate with program evaluators during the design phase (as has been recommended by regulators [2]), and that the RIMES-SE will be consulted throughout the process to promote adoption of the good practices which this checklist embodies.
Our study had several notable strengths. First, we used an e-Delphi method to obtain input on the RIMES-SE items from a group of leading experts in drug safety, risk minimization, and implementation science. E-Delphi is a technique for achieving consensus among subject matter experts in an iterative, anonymous, and asynchronous manner, and it is a recommended approach for developing reporting guidelines [29, 33]. A key advantage of an e-Delphi approach, as compared with group discussions or other in-person consensus techniques, is that it minimizes known biases associated with face-to-face group processes [41]. It is also a more feasible, convenient, and cost-effective way to bring together a group of experts, thus helping to maximize representativeness. An additional strength of our study is that we pre-specified the e-Delphi level of consensus, a documented shortcoming of some e-Delphi research to date [42]. Lastly, we succeeded in recruiting a heterogeneous panel of experts both in terms of content expertise (i.e., drug safety, regulatory science, implementation science, pharmacoepidemiology) and sector affiliation (i.e., regulatory agency, academia, pharmaceutical industry, pharmacovigilance consulting).
Two study limitations are worth noting. First, the size of our e-Delphi panel was relatively small. However, given that pharmaceutical risk minimization is a highly specialized topic, this was not unexpected. Individuals with expertise at the intersection of public health, implementation science, pharmacovigilance, and pharmacoepidemiology constitute a singular group.
Second, e-Delphi participants were drawn from Western Europe, the United Kingdom, and North America exclusively. Lack of input from other regions, including countries in the Asia-Pacific area, may potentially have caused us to overlook some cultural and/or healthcare system factors germane to risk minimization programs and their evaluation. The authors used their collective (and extensive) professional networks to identify eDelphi panelists who had deep subject matter expertise in their respective disciplines. In recruiting known professional connections, we hoped to increase the panelists’ commitment to completing the eDelphi questionnaires.
Risk minimization program design, implementation, and evaluation is a dynamic area of pharmacovigilance and regulatory science. Innovative approaches to incorporating patient and provider perspectives, the use of digital risk minimization tools, and the use of mixed methods designs are all examples where the science is evolving. There is also a need for greater inter-disciplinary collaboration among scientists (e.g., in the fields of public health, implementation science, data science, behavioral medicine) to advance methods, improve the effectiveness of risk minimization program design, and to develop new, more effective risk minimization tools and strategies to enhance program adoption, impact, and sustained use.
5 Conclusions
The RIMES-SE Statement and Checklist seeks to extend the risk minimization evaluation reporting guidelines set forth in the original RIMES Checklist via inclusion of key implementation science concepts incorporated in the StaRI Statement and Checklist. Our goal in doing so has been threefold. First, by specifying a standard, minimum set of items that should be reported in every risk minimization evaluation study, we aim to assist regulators, researchers, and other readers in assessing the quality of the research. Specifically, standardized reporting on the goals and rationale of the program, how the program was implemented, and what implementation and intervention outcomes were achieved, makes it easier to interpret the risk minimization evaluation results, and to understand the study’s strengths and limitations. Second, we seek to help researchers identify risk minimization evaluation studies in the published literature, thus facilitating their ability to critically appraise and synthesize this body of evaluation research. Third, we aim to increase awareness among regulators, researchers, and product manufacturers about good practices in risk minimization program design and evaluation. Such good practices include comprehensive implementation planning, the deployment of multiple, multi-level strategies to address identified barriers and facilitators to program uptake and maintenance, the use (and documentation) of program adaptations across different healthcare settings and geographies, and the assessment of program implementation outcomes as well as those relating to intervention effectiveness.
We recognize that these guidelines will need to be updated periodically in response to new developments in the field. We welcome input from pharmacovigilance practitioners, patients, regulators, researchers, and other interested stakeholders in the continuous updating of the RIMES-SE. Through the ongoing use of the RIMES-SE by researchers, regulators, and journal editors, we anticipate seeing improvements in the reporting of risk minimization evaluation studies in terms of both comprehensiveness and transparency. This, in turn, should lead to safer, more appropriate use of medicines and, ultimately, better outcomes for patients and public health more broadly.
References
European Medicines Agency. Guideline on good pharmacovigilance practices (GVP): module XVI—risk minimization measures: selection of tools and 5 effectiveness indicators (Rev 3). 2021. [cited 2023 Nov 20]. https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/guideline-good-pharmacovigilance-practices-gvp-module-xvi-risk-minimization-measures-selection-tools_en.pdf.
Food and Drug Administration. Risk evaluation and mitigation strategies assessment: planning and reporting; draft guidance for industry; availability 2019 [1153–5]. 2019. [cited 2023 Nov 20]. https://www.govinfo.gov/content/pkg/FR-2019-02-01/pdf/2019-00676.pdf.
Smith MY, Morrato E. Advancing the field of pharmaceutical risk minimization through application of implementation science best practices. Drug Saf. 2014;37(8):569–80. https://doi.org/10.1007/s40264-014-0197-0.
Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374: n2061. https://doi.org/10.1136/bmj.n2061.
PRAC recommends new measures to avoid valproate exposure in pregnancy: new restrictions on use; pregnancy prevention programme to be put in place. 2018. [cited 2023 Dec 4]. https://www.ema.europa.eu/en/documents/referral/valproate-article-31-referral-prac-recommends-new-measures-avoid-valproate-exposure-pregnancy_en.pdf.
FDA's risk evaluation and mitigation strategies: uncertain effectiveness in addressing the opioid crisis. 2020. [cited 2023 Dec 4]. https://oig.hhs.gov/oei/reports/OEI-01-17-00510.asp.
Department of Health and Human Services. FDA lacks comprehensive data to determine whether risk evaluation and mitigation strategies improve drug safety. 2013. [cited 4 Dec 2023]. https://oig.hhs.gov/oei/reports/OEI-04-11-00510.asp.
Artime E, Qizilbash N, Garrido-Estepa M, Vora P, Soriano-Gabarro M, Asiimwe A, Pocock S. Are risk minimization measures for approved drugs in Europe effective? A systematic review. Expert Opin Drug Saf. 2019;18(5):443–54. https://doi.org/10.1080/14740338.2019.1612875.
Gridchyna I, Cloutier AM, Nkeng L, Craig C, Frise S, Moride Y. Methodological gaps in the assessment of risk minimization interventions: a systematic review. Pharmacoepidemiol Drug Saf. 2014;23(6):572–9. https://doi.org/10.1002/pds.3596.
Levinson DR. FDA lacks comprehensive data to determine whether risk evaluation and mitigation strategies improve drug safety. 2013. [cited 2023 Dec 4]. https://oig.hhs.gov/oei/reports/oei-04-11-00510.pdf.
Mazzaglia G, Straus SMJ, Arlett P, da Silva D, Janssen H, Raine J, Alteri E. Study design and evaluation of risk minimization measures: a review of studies submitted to the European Medicines Agency for Cardiovascular, Endocrinology, and Metabolic Drugs. Drug Saf. 2018;41(2):191–202. https://doi.org/10.1007/s40264-017-0604-4.
Smith MY, Russell A, Bahri P, Mol PGM, Frise S, Freeman E, Morrato EH. The RIMES statement: a checklist to assess the quality of studies evaluating risk minimization programs for medicinal products. Drug Saf. 2018;41(4):389–401. https://doi.org/10.1007/s40264-017-0619-x.
Okai J, Bahri P, Sarinic VM. Reporting quality of evaluations of additional risk minimization measures by MAH: an assessment of studies submitted in the European Union. Drug Saf. 2021;44(12):1402–3.
Russell AM, Morrato EH, Lovett RM, Smith MY. Quality of reporting on the evaluation of risk minimization programs: a systematic review. Drug Saf. 2020;43(5):427–46. https://doi.org/10.1007/s40264-020-00905-8.
ENCePP guide on methodological standards in pharmacoepidemiology. 2022. [cited 2023 Dec 4]. https://www.encepp.eu/standards_and_guidances/methodologicalGuide7_1.shtml.
Xiaolu N, Shaowen T, Jinyang Y, Haibo S, Zhu F, Feng S. Interpretations of reporting recommendations intended for pharmaceutical risk minimization evaluation studies (RIMES) statement. Chin J Pharmacovigil. 2023;20(5):524–9. https://doi.org/10.19803/j.1672-8629.20220527.
Smith M, Saunders R, Stuckhardt L, McGinnis JM. Best care at lower cost: the path to continuously learning health care in America. National Academies Press; 2013.
Reporting and learning systems for medication errors: the role of pharmacovigilance centres. 2014. [cited 2023 Dec 4]. https://iris.who.int/bitstream/handle/10665/137036/9789241507943_eng.pdf.
Jackson GL, Cutrona SL, Kilbourne A, White BS, Everett C, Damschroder LJ. Implementation science: helping healthcare systems improve. JAAPA. 2020;33(1):51–3. https://doi.org/10.1097/01.JAA.0000615508.92677.66.
Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for reporting implementation studies (StaRI) statement. BMJ. 2017. https://doi.org/10.1136/bmj.i6795.
Cox J, Gutner C, Kronfli N, Lawson A, Robbins M, Nientker L, et al. A need for implementation science to optimise the use of evidence-based interventions in HIV care: a systematic literature review. PLoS ONE. 2019;14(8): e0220060. https://doi.org/10.1371/journal.pone.0220060.
De Geest S, Zúñiga F, Brunkert T, Deschodt M, Zullig LL, Wyss K, Utzinger J. Powering Swiss health care for the future: implementation science to bridge “the valley of death.” Swiss Med Wkly. 2020;150(3738): w20323. https://doi.org/10.4414/smw.2020.20323.
Olson MS, Rootkin L. The triple win—implementation science benefits patients, healthcare systems and industry alike. J Comp Eff Res. 2022;11(9):639–42. https://doi.org/10.2217/cer-2022-0058.
New heart disease drug to be made available for NHS patients. 2020. [cited 2023 Dec 4]. www.gov.uk/government/news/new-heart-disease-drug-to-be-made-available-for-nhs-patients#:∼:text=The%20yet%20to%20be%20approved,to%20start%20later%20this%20year.
Huynh L, Toyserkani GA, Morrato EH. Pragmatic applications of implementation science frameworks to regulatory science: an assessment of FDA Risk Evaluation and Mitigation Strategies (REMS) (2014–2018). BMC Health Serv Res. 2021;21(1):779. https://doi.org/10.1186/s12913-021-06808-3.
Toyserkani GA, Huynh L, Morrato EH. Adaptation for regulatory application: a content analysis of FDA risk evaluation and mitigation strategies assessment plans (2014–2018) using RE-AIM. Front Public Health. 2020;8:43. https://doi.org/10.3389/fpubh.2020.00043.
Enhancing the quality and transparency of health research. 2023. [cited 2023 Dec 4]. https://www.equator-network.org/.
von Elm E, Altman DG, Egger M, Pocock SJ, Gotzsche PC, Vandenbroucke JP. The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med. 2007;147(8):573–7. https://doi.org/10.7326/0003-4819-147-8-200710160-00010.
Nicholls SG, Quach P, von Elm E, Guttmann A, Moher D, Petersen I, et al. The reporting of studies conducted using observational routinely-collected health data (RECORD) statement: methods for arriving at consensus and developing reporting guidelines. PLoS ONE. 2015;10(5): e0125620. https://doi.org/10.1371/journal.pone.0125620.
Langan SM, Schmidt SA, Wing K, Ehrenstein V, Nicholls SG, Filion KB, et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ. 2018;363: k3532. https://doi.org/10.1136/bmj.k3532.
Ogrinc G, Davies L, Goodman D, Batalden P, Davidoff F, Stevens D. SQUIRE 2.0 (standards for quality improvement reporting excellence): revised publication guidelines from a detailed consensus process. J Contin Educ Nurs. 2015;46(11):501–7. https://doi.org/10.3928/00220124-20151020-02.
Des Jarlais DC, Lyles C, Crepaz N, Trend Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement. Am J Public Health. 2004;94(3):361–6. https://doi.org/10.2105/ajph.94.3.361.
Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2): e1000217. https://doi.org/10.1371/journal.pmed.1000217.
Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Health. 2020;8:457. https://doi.org/10.3389/fpubh.2020.00457.
RIMES-SE—an extension of the standards for reporting implementation studies—StaRI checklist extension (SE) for reporting recommendations intended for pharmaceutical risk minimization evaluation studies (RIMES) statement. 2023. [cited 2023 Dec 3]. https://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#RIMES.
Radawski C, Morrato E, Hornbuckle K, Bahri P, Smith M, Juhaeri J, et al. Benefit-risk assessment, communication, and evaluation (BRACE) throughout the life cycle of therapeutic products: overall perspective and role of the pharmacoepidemiologist. Pharmacoepidemiol Drug Saf. 2015;24(12):1233–40. https://doi.org/10.1002/pds.3859.
Kwan BM, Brownson RC, Glasgow RE, Morrato EH, Luke DA. Designing for dissemination and sustainability to promote equitable impacts on health. Annu Rev Public Health. 2022;43:331–53. https://doi.org/10.1146/annurev-publhealth-052220-112457.
Shelton RC, Chambers DA, Glasgow RE. An extension of RE-AIM to enhance sustainability: addressing dynamic context and promoting health equity over time. Front Public Health. 2020;8:134. https://doi.org/10.3389/fpubh.2020.00134.
Feldstein AC, Glasgow RE. A practical, robust implementation and sustainability model (PRISM) for integrating research findings into practice. Jt Comm J Qual Patient Saf. 2008;34(4):228–43. https://doi.org/10.1016/s1553-7250(08)34030-6.
Altman DG, Simera I, Hoey J, Moher D, Schulz K. EQUATOR: reporting guidelines for health research. Lancet. 2008;371(9619):1149–50. https://doi.org/10.1016/S0140-6736(08)60505-X.
Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, Sheps DM. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol. 2016;16:56. https://doi.org/10.1186/s12874-016-0165-8.
Foth T, Efstathiou N, Vanderspank-Wright B, Ufholz LA, Dutthorn N, Zimansky M, Humphrey-Murto S. The use of Delphi and Nominal Group Technique in nursing education: a review. Int J Nurs Stud. 2016;60:112–20. https://doi.org/10.1016/j.ijnurstu.2016.04.015.
Rogers EM. Diffusion of innovation. 5th ed. Free Press; 2003.
Movsisyan A, Arnold L, Evans R, et al. Adapting evidence-informed complex population health interventions for new contexts: a systematic review of guidance. Implement Sci. 2019;14:105. https://doi.org/10.1186/s13012-019-0956-5.
Lomas J. Diffusion, dissemination, and implementation: who should do what? Ann NY Acad Sci. 1993;703:226–35. https://doi.org/10.1111/j.1749-6632.1993.tb26351. (discussion 235–237).
Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009;36(1):24–34. https://doi.org/10.1007/s10488-008-0197-4.
Wolfenden L, Sutherland R, Shelton RC, Yoong SL, Nathan N. Scale-up and sustainment of effective interventions. In: Brownson RC, Colditz GA, Proctor EK, editors. Dissemination and implementation research in health. 3rd ed. Oxford: Oxford University Press; 2023. p. 27–65.
Acknowledgements
We would like to thank the following colleagues for their participation in the e-Delphi panel: Rinad Beidas, Priya Bahri, Ross Brownson, Jay Dave, Rachel Davis, Alicia Gilsenan, Giampiero Mazzaglia, Rania Mouchantaf, Monika Pietrek, Borsika Rabin, Amir Sarayani, Jodi Segal, Rachel Sobel, Gita Toyserkani, and Katy Trinkley. We thank Ms Madeline Mehall for programing the e-Delphi questionnaires.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Funding
EHM’s effort was supported in part by the National Center for Advancing Translational Science of the National Institutes of Health under award number 2UL1TR002389. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. No other financial support was received.
Conflict of interest
MYS is a fulltime employee of Evidera, a consultancy that performs healthcare research for a wide range of biopharmaceutical and biotechnology companies, and is a shareholder in Thermo Fisher Scientific, which owns PPD. She is also an adjunct faculty member in the Department of Regulatory and Quality Sciences, School of Pharmacy, University of Southern California, Los Angeles, CA, USA. VN and NM have no conflicts of interest to report. EHM has received funding over the past three years for consulting services on issues of drug safety from Eli Lilly, Health Care Service Corporation, Inc., i2o, Humana Inc., Ipsen, Molina Healthcare, Inc., Reata Pharmaceuticals, Syneos Health, and United Healthcare Services. HP receives grant monies paid to her institution for applied and implementation research unrelated to risk minimization program evaluations. In the past three years she has received speaker fees from Teva and Sandoz for non-promotional talks unrelated to topics covered in this paper. AGW has received consulting fees from Bayer KG, Genentech Inc, Arbor Pharmaceuticals, and Ipsen, and has received research funding from Merck, Sharp and Dohme.
Ethics approval
Ethical review was conducted by the Advarra Institutional Review Board (IRB), Columbia, MD, USA. For the United States (US) and Western Europe, the project was deemed to be exempt from IRB review (May 18, 2023; Pro00071189); for Canada, ethics approval was granted on May 24, 2023 (Pro00071646).
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Availability of data and materials
Available upon request to the corresponding author.
Code availability
Not applicable.
Author contributions
MYS designed the study, analysis, wrote the first draft of the manuscript, and prepared the final draft for submission. NM conducted statistical analysis of the e-Delphi data, prepared the e-Delphi results tables, and reviewed the final draft of the manuscript. VN prepared the tables and assisted in reviewing and revising the manuscript. HP, AGW, and EHM assisted in the study design, the conduct of the data analysis, and reviewed the final draft of the manuscript. All authors read and approved the final version. This manuscript has been endorsed by the International Society for Pharmacoepidemiology.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc/4.0/.
About this article
Cite this article
Smith, M.Y., Morrato, E.H., Mora, N. et al. The Reporting Recommendations Intended for Pharmaceutical Risk Minimization Evaluation Studies: Standards for Reporting of Implementation Studies Extension (RIMES-SE). Drug Saf (2024). https://doi.org/10.1007/s40264-024-01417-5
Accepted:
Published:
DOI: https://doi.org/10.1007/s40264-024-01417-5