FormalPara Key Points

Probability of completing evaluations of the effectiveness of additional risk minimisation measures (aRMMs) for medicinal products within 60 months after marketing authorisation (MA) was one in five.

Estimates of the duration of aRMM effectiveness evaluation studies around MA are too optimistic, with the majority of studies being delayed when compared to their planned end date.

Duration of aRMM effectiveness evaluation studies varies according to study design.

1 Introduction

While medicinal products hold major benefits for patients, adverse drug reactions (ADRs) cause significant morbidity and mortality. It was estimated that 3.5% of hospital admissions are due to ADRs [1]. In the European Union (EU), the European Medicines Agency (EMA) is responsible for evaluating medicinal products’ centralised marketing authorisation (MA) applications and monitoring their safety in the post-authorisation period. Within the EMA, the Pharmacovigilance Risk Assessment Committee (PRAC) is responsible for all aspects of risk management of medicinal products. The European Union Risk Management Plan (EU-RMP) is important in risk management and has been part of the authorisation procedure for medicinal products since 2005. This EU-RMP aids in identifying, characterising, monitoring, and minimising risks throughout the life-cycle of medicinal products [2]. Specifically, it describes measures that can be introduced to minimise important risks (potentially) associated with the use of the medicinal product (i.e., risks that might have an impact on the benefit-risk balance of the medicinal product) [3]. Besides presenting routine risk minimisation measures applicable to all medicinal products (i.e., Summary of Product Characteristics [SmPC], Package Leaflet [PL], package design, pack size, and legal status), the EU-RMP may also include additional risk minimisation measures (aRMMs). These measures are introduced when the risks are considered to be insufficiently managed by routine risk minimisation measures [3, 4]. They include educational materials for health care professionals (HCPs) and/or patients/caregivers, controlled access programmes, controlled distribution programmes, pregnancy prevention programmes (PPPs), and Direct Healthcare Professional Communications (DHPCs) [4].

A medicinal product’s need for aRMMs is assessed both at the time of MA and continuously in the post-authorisation period [5,6,7]. Previous research has indicated that approximately 30% of all medicinal products registered between 2012 and 2015 had aRMMs at MA [5]. Furthermore, new information regarding risks may require post-authorisation introduction, change, or discontinuation of aRMMs [3, 8]. Thus, formal evaluation of the effectiveness of aRMMs in the post-marketing phase is crucial for continuous re-evaluation of the medicinal product’s benefit-risk balance as this examines whether the objectives of aRMMs are fulfilled or amendments are needed to protect patients’ health. If aRMMs are ineffective at minimising the important risks of medicinal products, preventable adverse events leading to morbidity, hospitalization, and mortality can occur [9]. Therefore, timely evaluation of aRMMs must be implemented to address potential safety issues and facilitate safe use of medicinal products. Besides, the additional burden of aRMMs on HCPs, patients, and marketing authorisation holders (MAHs) should remain proportionate to the risks. Thus, evaluations should also consider whether aRMMs are still necessary and whether those measures have been sufficiently integrated into routine clinical practice [10]. Amendments to the EU’s pharmaceutical legislation in July 2012 rendered evaluation of the effectiveness of aRMMs mandatory [6]. As a result, a regulatory framework for evaluating aRMMs was introduced in the Good PharmacoVigilance Practice (GVP) Module XVI—Risk Minimisation Measures: Selection of Tools and Effectiveness Indicators [4]. In general, aRMMs can be evaluated using routine (e.g., Periodic Safety Update Reports [PSURs] to regulatory authorities) and/or additional pharmacovigilance activities (i.e., using post-authorisation safety studies). Regarding these additional pharmacovigilance activities, the number of studies that evaluated the effectiveness of aRMMs is growing [11,12,13]. Among products with aRMMs authorised between 2006 and 2015, one-third of EU-RMPs included an aRMM effectiveness evaluation study and this proportion has increased over time [11]. Furthermore, 78.3% of medicinal products with aRMMs for medication errors have a study in place to evaluate the effectiveness of these measures [14].

In aRMM effectiveness evaluation studies, the need for speed must be properly balanced with the quality of the study [10]. On the one hand, collecting sufficient data and thorough evaluation of the effects is time-consuming. On the other hand, if adjustments to the aRMMs are needed, they should be implemented as quickly as possible to protect patients’ health and/or reduce unnecessary burden on stakeholders [15]. Good PharmacoVigilance Practice Module XVI Rev 2 (implemented in 2017) specifies that timelines for measuring the effectiveness of aRMMs should be determined on a case-by-case basis for each medicinal product. However, it also provides guidance on time points of particular relevance for evaluation, namely after initial implementation of aRMMs (e.g. within 12–18 months) and in time of renewal of the MA (i.e. five years after initial MA) [4].

Despite these guidelines, previous research had not reported on the duration of aRMM effectiveness evaluations. Consequently, knowledge regarding the actual time needed to complete aRMM effectiveness evaluations is limited. In this study, we aim to assess the duration of aRMM effectiveness evaluations, restricted to additional pharmacovigilance activities, in medicinal products licensed via the central authorisation procedure in the EU between July 2012 and December 2021.

2 Methods

2.1 Study Design

We performed a retrospective cohort study within a source population that included all medicinal products authorised by the EU’s centralised procedure between 1 July 2012 and 31 December 2021. As evaluation of the effectiveness of aRMMs was introduced to pharmaceutical legislation in July 2012 [6], we chose this date as the starting point for our study. We excluded generic, biosimilar, and informed consent applications since the EU-RMP of those medicinal products is expected to replicate their reference products, especially regarding aRMMs. Medicinal products subject to duplicate or multiple marketing applications were included only once. From this source population, we established a study cohort of medicinal products with aRMMs at the time of MA. The data lock point (DLP) for data collection was 1 April 2022 (end of follow-up).

2.2 Data Collection

For the source population, we identified medicinal products to be included in our study using the European Public Assessment Report (EPAR) database from the EMA (www.ema.europa.eu). For each medicinal product, we retrieved the EPAR—Public Assessment Report related to the initial MA procedure. This report includes a summary of the initial MA assessment, including the EU-RMP. We reviewed this document to establish whether aRMMs were in place at the time of MA.

For the study cohort of medicinal products with aRMMs at the time of MA, we retrieved the approved full EU-RMPs at the time of MA from the Medicines Evaluation Board (MEB) internal database. This was necessary because the publicly available EPAR reports often lacked information to identify aRMM effectiveness evaluation studies accurately, especially for products authorised in 2012 and 2013. For evaluation studies identified from the EU-RMPs, we extracted the first and most recent study protocols approved by the EMA via the MEB internal database. Moreover, we identified the final study reports of the MAHs and corresponding final assessment reports of the PRAC using this database. Last, we collected the minutes of the PRAC meetings in which the final study reports were discussed.

2.3 Inclusion and Exclusion of Effectiveness Studies

We included studies only if a stated objective was to study the effectiveness of aRMMs as described within the EU-RMP at MA. Evaluation studies that were requested post-authorisation were not included in this study. Before the start of the evaluation study, PRAC might conclude that a study requested at MA was no longer necessary in order to evaluate the effectiveness of aRMMs for medicinal products. We excluded these studies (n = 2).

2.4 Study Outcomes

Our primary outcome was the duration of aRMM effectiveness evaluations for medicinal products. For the purpose of our study, we defined duration as the time from the date of MA to the date of the PRAC’s final recommendation based on assessment of the final study report (i.e., the PRAC outcome). We chose the date of MA approval as a starting point for analyses since the aRMM implementation date was difficult to retrieve, varied by country, and certain medicinal products were not marketed in specific countries. The date of the PRAC outcome was extracted from the PRAC minutes. For medicinal products with more than one aRMM effectiveness evaluation study, we defined the date of the PRAC outcome for the last completed study as the endpoint of interest. Besides, we assessed the duration of the aRMM effectiveness evaluation studies. We subdivided the duration into three periods for each evaluation study: from MA to the start of the evaluation study (defined as the start of data collection), from the start of the evaluation study to the final study report, and from the final study report to the PRAC outcome (Fig. 1).

Fig. 1
figure 1

Overview of the timeline of an aRMM effectiveness evaluation, from marketing authorisation to PRAC outcome. Three distinct periods can be identified for aRMM effectiveness evaluation: from marketing authorisation to the start of the evaluation study, from the start of the evaluation study to the final study report, and from the final study report to the PRAC outcome. *Start of evaluation study was defined as start of data collection in the evaluation study. ±A medicinal product with aRMM at MA might have multiple studies with the objective to evaluate the effectiveness of aRMMs. The multiple studies often contribute individually to a combined indication of the effectiveness of aRMMs aRMM additional risk minimisation measure, PRAC Pharmacovigilance Risk Assessment Committee

Full size image

As a secondary outcome, we assessed the delay in completion of the aRMM effectiveness evaluation studies. Therefore, we collected data on the planned date of submission of the final study report. This date was retrieved from the EU-RMP at MA or the first study protocol if it was not specified in the EU-RMP. The actual date of the final study report was used to identify the actual duration of each evaluation study. A study was considered delayed when the actual date was > 3 months later than the planned date of the final report.

2.5 Cohort and Study Characteristics

Using the EPAR—Public Assessment Reports, we collected data to describe the medicinal products included in our source population and study cohort including active substance, Anatomical Therapeutic Chemical (ATC) classification, year of MA, authorisation status at DLP (authorised; withdrawn), aRMMs at MA (yes; no), type of aRMMs at MA, and target population of aRMMs (HCPs; patients/caregivers; both). Classification of the aRMMs was based on the GVP Module XVI, as presented in Table 1 [4]. Currently, various definitions are given for PPP elements. According to our definition based on the GVP Module XVI rev 2, a PPP should comprise the following: (1) contraindication for pregnancy in the SmPC and PL; (2) educational material targeted at HCPs and patients/caregivers to inform about the teratogenic risk and actions to minimise this risk; (3) controlled access at the prescribing or dispensing level to ensure that a pregnancy test is conducted and negative results are verified before prescription, as well as at an appropriate frequency during and after treatment; and (4) counselling on teratogenic risks, on effective contraceptive measures, and in the event of inadvertent pregnancy [4]. Based on our definition, we classified PPPs for medicinal products for which a PPP was not explicitly noted in the EPAR (n = 3).

Table 1 Types of aRMMs
Full size table

We used the final study report or, if it was not available, the latest approved study protocol or EU-RMP to gather information on the individual aRMM effectiveness evaluation studies. Data were recorded on the study design (cross-sectional study; cohort study), data source (primary data collection [e.g., survey]; secondary data collection [e.g., electronic health records]; both), study population (HCPs; patients/caregivers; both), and type of indicator (process indicator; outcome indicator; both). Process indicators evaluate the implementation steps of aRMMs, whereas outcome indicators provide an overall measure of the level of risk control achieved by aRMMs [4, 13]. Based on the GVP Module XVI, we defined process indicators as those that measure the receipt of aRMMs by the target population, clinical knowledge (e.g., attitude, understanding, knowledge of HCPs or patients/caregivers), or clinical action (e.g., compliance with monitoring recommendations, prescribing behaviour). We defined indicators as outcome indicators only if they represent safety outcomes (frequency/severity of adverse events and medication errors). Therefore, all other non-safety related indicators that the MAHs classified as outcome indicators were reclassified as process indicators [4, 12].

Data collection and categorisation were performed by one researcher and discussed with a second researcher in case of doubt. For quality control, 17% (n = 13) of the data extracted for aRMM evaluation studies was checked by comparing it with data independently collected by a third researcher.

2.6 Data Analyses

We used descriptive statistics to describe the characteristics of the medicinal products included in our source population and study cohort and of the evaluation studies. The categorical variables were assessed using frequencies and percentages.

To account for the time needed to complete effectiveness evaluations, we assessed the duration of aRMM evaluations using time-to-event analyses. We performed these separately for medicinal products and aRMM evaluation studies as units of analysis. If the evaluations were not completed at DLP (i.e., the PRAC outcome), we censored the medicinal products and studies at DLP. If the evaluations were not completed due to feasibility issues or market withdrawal, we censored medicinal products and studies at the date of discontinuation of the evaluation/study. We calculated the probability of finalisation for aRMM effectiveness evaluations (i.e., the PRAC outcome) within 60 months after MA using Kaplan-Meier survival analyses. The cut-off of 60 months was chosen in line with the time for evaluation of the renewal of the marketing authorisation mentioned as time point of particular relevance in the GVP Module XVI Rev 2 [4]. We compared the duration of the evaluation studies between study designs using Kaplan-Meier analysis and a log-rank test (significant if p < 0.05). For the individual studies, a timelines plot was created to provide an overview of the distribution of the duration across the three distinct periods as outlined in “2.4. Study Outcomes”.

We calculated the median planned duration of aRMM evaluation studies with the interquartile range (IQR) for all studies based on the planned date for submitting the final study report. For each study, we determined whether the planned date of submission was before our DLP. For studies with a planned date before DLP, we used a Kaplan-Meier analysis to calculate the probability of finalising an aRMM study within one and two year(s) after the planned submission of the final study report.

Data collection and data analyses were performed in Microsoft Excel, SPSS version 28.0.1.1, and R version 4.1.2.

3 Results

3.1 Characteristics of Source Population and Study Cohort

Our source population consisted of 526 medicinal products that were centrally authorised in the EU between 1 July 2012 and 31 December 2021. The characteristics of the medicinal products are presented in Table 2. Of these medicinal products, 134 (25.5%) were authorised with aRMMs. The majority of these medicinal products was granted standard MA (n = 113, 84.3%) and was still registered at DLP (n = 121, 90.3%). Medicinal products within the ATC groups “Antineoplastic and immunomodulatory agents” (n = 56, 41.8%) and “Alimentary tract metabolism” (n = 18, 13.4%) were most prevalent.

Table 2 Characteristics of medicinal products authorised via the centralised authorisation procedure between 1 July 2012 and 31 December 2021
Full size table

In our study cohort of 134 medicinal products with aRMMs, the most common type of aRMM was educational material; this was in place for 131 (97.8%) medicinal products (Table 2). In addition to an educational programme, at least one other aRMM was required for 25 (18.7%) medicinal products: a controlled distribution programme (n = 13, 9.7%), controlled access programme (n = 7, 5.2%), PPP (n = 5, 3.7%), and/or DHPC (n = 3, 2.2%). Two (1.5%) medicinal products only required a DHPC, and one (0.7%) had a PPP and DHPC. More than half (n = 79, 59.0%) of aRMMs targeted both HCPs and patients.

The frequency distribution of medicinal products with aRMMs varied between 2012 and 2021 (Fig. 2). The proportion of medicinal products with aRMMs at MA fluctuated over time with no visual upward or downward trend.

Fig. 2
figure 2

Medicinal products (n = 526) authorised per year with and without aRMMs at MA from July 2012 to December 2021. The bars represent the number of medicinal products without aRMMs at MA (light orange) and with aRMMs at MA (dark orange). The line represents the percentage of medicinal products with aRMMs at MA (right Y axis). *Medicinal products authorised from 1 July 2012 onwards were included. aRMM additional risk minimisation measure, MA marketing authorisation

Full size image

3.2 Characteristics of aRMM Effectiveness Evaluation Studies

For almost half (n = 63, 47.0%) of medicinal products, the EU-RMP at MA described a study evaluating the effectiveness of aRMMs. We identified 52 (38.8%) medicinal products with one aRMM effectiveness evaluation study and 11 (8.2%) medicinal products with multiple aRMM effectiveness evaluation studies planned at MA. As presented in Figure 3, the proportion of medicinal products with at least one aRMM effectiveness study planned varied between 2012 and 2021, ranging from 33.3% in 2018 to 85.7% in 2016, but without a clear upward or downward trend. All 63 medicinal products with a planned evaluation study had educational material for HCPs and/or patients in place. Additionally, five (7.9%) medicinal products had a controlled access programme, four (6.3%) had a controlled distribution system, two (3.2%) had a PPP, and one (1.6%) had a controlled distribution system, PPP, and DHPC.

Fig. 3
figure 3

Medicinal products with aRMMs (n = 134) authorised per year with and without an aRMM effectiveness evaluation study planned at MA from July 2012 to December 2021. The bars represent the number of medicinal products without a study planned at MA (light orange), with one study planned at MA (dark orange), and with multiple studies planned at MA (blue) (left Y axis). The line represents the percentage of medicinal products with at least one study planned at MA (right Y axis). *Medicinal products authorised from 1 July 2012 onwards were included. aRMM additional risk minimisation measure, MA marketing authorisation

Full size image

We retrieved 77 individual aRMM effectiveness evaluation studies, of which 7 were imposed studies (9.1%). All studies involved observational studies with a cross-sectional design in 45 (58.4%) studies and a cohort design in 32 (41.6%) studies (Table 3). The majority of studies (n = 53, 68.8%) used primary data collection solely or in combination with secondary data collection. A total of 37 (48.1%) studies were targeted at patients/caregivers only, 29 (37.7%) studies were targeted at HCPs only, and 11 (14.3%) studies were targeted at both patients/caregivers and HCPs. As shown in Table 3, most studies evaluated process indicators (n = 68, 88.3%). Of the 68 studies that included process indicators, 39 (57.4%) evaluated receipt of the aRMMs, 41 (60.3%) evaluated clinical knowledge, and 45 (66.2%) evaluated clinical action/behaviour.

Table 3 Characteristics of studies evaluating the effectiveness of aRMMs planned at marketing authorisation
Full size table

3.3 Duration of aRMM Effectiveness Evaluation—Studies as Unit of Analysis

Of the 77 aRMM effectiveness evaluation studies, 27 (35.1%) were finalised and assessed by PRAC during the study period. For 43 (55.8%) studies, the study was being set-up, ongoing, or under assessment by PRAC at DLP. The remaining seven (9.1%) studies were terminated because the medicinal products were withdrawn from the market (n = 4), the studies were not feasible (n = 2), or the aRMMs were removed (n = 1). Figure 4 shows the duration of the individual aRMM effectiveness evaluation studies divided into the three main periods: time from MA to the start of the evaluation study, from the start of the evaluation study to the final report, and from the final report to the PRAC outcome. The first two periods were the most time consuming.

Fig. 4
figure 4

Timelines plot for duration of aRMM effectiveness evaluation studies. The duration is divided among the three distinct periods: from MA to the start of the evaluation study (light blue), from the start of the evaluation study to the final study report (light orange), and from the final study report to the PRAC outcome (dark orange). The planned date of final study report is depicted (black bars) together with the planned duration until submission of the final study report if the planned date of final study report was after the DLP (grey bars; 1 April 2022). Each line in the plot represents an individual effectiveness evaluation study (n = 77). Studies at the top of the figure were censored at DLP. *Effectiveness evaluation studies were assumed to be started if the start date of data collection as specified in the latest study protocol was reached. In case the start date of the evaluation study was not documented, the time between MA and the final study report was depicted (dark blue). aRMM additional risk minimisation measure, DLP data lock point, MA marketing authorisation, PRAC Pharmacovigilance Risk Assessment Committee

Full size image

Of the evaluation studies with at least 60 months follow-up (n = 37), the probability that an evaluation study was finalised and assessed by PRAC was 25.2% (95% CI 12.3–36.2). This probability was 37.0% (95% CI 16.9–52.2) for cross-sectional studies and 9.6% (95% CI 0.0–21.4) for cohort studies as shown in Fig. 5. Overall, the probability of completing an evaluation study with assessment by PRAC was higher for cross-sectional studies than for cohort studies (p = 0.002).

Fig. 5
figure 5

Kaplan-Meier estimates for finalisation of aRMM effectiveness evaluation studies by study design: cohort studies (orange) and cross-sectional studies (blue). The dashes in the solid line represent censored studies, and the dotted lines represent the 95% confidence interval for the Kaplan-Meier estimates. The number at risk is provided at different timepoints for cohort studies and cross-sectional studies, indicating the number of studies that were still accounted for at that timepoint. aRMM additional risk minimisation measure, MA marketing authorisation PRAC Pharmacovigilance Risk Assessment Committee

Full size image

3.4 Delay of aRMM Effectiveness Evaluation Studies

For 68 (88.3%) aRMM effectiveness evaluation studies, the planned date for submitting the final study report was provided. The median duration from MA until the planned date of submission was 45 months (IQR: 36–60 months). Based on the planned date of submission, 42 (54.5%) studies should have been completed at DLP. The majority (n = 34/42, 81.0%) of these studies were delayed for more than three months compared to their planned date of submission (Fig. 4). Kaplan-Meier survival analysis showed that the probability of finalising an evaluation study within one year after the planned date of submission of the final study report was 29.9% (95% CI 14.1–42.7), including studies that were finished on time (Fig. 6). Within two years after the planned final report date, the probability of completing an aRMM evaluation study increased to 55.1% (95% CI 35.6–68.7).

Fig. 6
figure 6

Kaplan-Meier estimate for duration of delay of aRMM effectiveness evaluation studies. The dashes in the solid line represent censored studies, and the dotted lines represent the 95% confidence interval for the Kaplan-Meier estimate. The number at risk is provided at different timepoints, indicating the number of studies that were still accounted for at that timepoint. aRMM additional risk minimisation measure

Full size image

3.5 Duration of aRMM Effectiveness Evaluation—Medicinal Products as Unit of Analysis

As 11 out of 63 medicinal products had more than one evaluation study, we also studied the time to complete the entire programme of studies to gain insights into the duration for evaluating the effectiveness of aRMMs at the medicinal product level. During the study period, the evaluation was completed for 17 (27.0%) out of 63 medicinal products. For three (4.8%) medicinal products, one evaluation study was finalised, but evaluation of the effectiveness of aRMMs was not yet completed since multiple evaluation studies were in place. The evaluation was ongoing for 37 (58.7%) other medicinal products at DLP. For six (9.5%) medicinal products, the evaluation could not be completed because of market withdrawal (n = 4), removal of the aRMMs (n = 1), or study infeasibility (n = 1).

Figure 7 depicts the Kaplan-Meier curve for the duration of aRMM effectiveness evaluations for medicinal products. The probability of completing an aRMM effectiveness evaluation for medicinal products within 60 months after MA was 20.7% (95% CI 6.8–32.6; n at risk = 29).

Fig. 7
figure 7

Kaplan-Meier estimates for finalisation of aRMM evaluations for medicinal products. The dashes in the solid line represent censored medicinal products, and the dotted lines represent the 95% confidence interval for the Kaplan-Meier estimate. The number at risk is provided at different timepoints, indicating the number of medicinal products that were still accounted for at that timepoint. aRMM additional risk minimisation measure, MA marketing authorisation

Full size image

4 Discussion

To our knowledge, this is the first study to assess the duration of aRMM effectiveness evaluations using additional pharmacovigilance activities for centrally authorised medicinal products in the EU. We identified 63 medicinal products with aRMMs and at least one planned aRMM effectiveness evaluation study at MA authorised between July 2012 and December 2021. To account for the time required to complete an effectiveness evaluation and differences in follow-up time for medicinal products, we assessed the probability of finalising evaluations using time-to-event analyses. The probability of finalising an evaluation for medicinal products within 60 months was one in five (20.7%; 95% CI 6.8–32.6).

Evaluating the effectiveness of aRMMs is an important element of risk management for medicinal products. However, evaluation studies were planned for only half of the medicinal products with aRMMs. A previous study indicated that one-third of the medicinal products with aRMMs authorised between 2006 and 2015 described an evaluation study within their EU-RMP. This proportion increased to 43% when analysis was restricted to the period from 2012 onwards (i.e., from the year of the amendments to the pharmaceutical legislation) [11]. These findings agreed with those of our study. Another study showed that 78.3% of medicinal products with aRMMs specifically for medication errors had a study in place to evaluate the effectiveness of aRMMs [14]. This higher proportion of medicinal products with an evaluation study may have been due to differences in the inclusion criteria for an aRMM effectiveness study. We defined a study as an evaluation study only if it had the objective to study the effectiveness of aRMMs as described in the EU-RMP. In contrast, Hoeve et al. used the EPAR and EU-PAS registry as data sources and also included “analyses of spontaneous reports” as an evaluation study. Furthermore, the higher proportion of medicinal products with an evaluation study might also be specific to the safety concern of medication errors.

For the other half of medicinal products authorised with aRMMs included in our study, evaluation of the effectiveness of aRMMs using only routine pharmacovigilance activities was planned. This included for example analyses of spontaneous reports in the PSURs. We did not investigate how medicinal products with evaluation studies in place differed from medicinal products for which the effectiveness of aRMMs was assessed using routine pharmacovigilance only. While studies could be perceived as the preferred way to conduct thorough evaluation of the aRMM effectiveness, these studies often have limitations and/or are poorly designed. Furthermore, conducting aRMM effectiveness evaluation studies might not be feasible for all medicinal products, for example, due to low use of the medicinal product in practice. Therefore, the different approaches for evaluating aRMM effectiveness should be carefully considered for each medicinal product.

The timing of aRMM effectiveness evaluations is challenging because the need for the timely data to protect patients’ health must be balanced with accurate performance of the evaluation [10, 15]. The GVP Module XVI Rev 2 provides guidance on time points of particular relevance for evaluating the effectiveness of aRMMs, namely after initial implementation of the aRMMs (e.g., within 12–18 months) and in time for the renewal of a MA (i.e., five years after MA) [4]. It should be noted that this module was published in 2017, and an update of the module (Rev 3) was drafted in 2021. This draft still suggests that evaluation of aRMMs should be considered after initial implementation of aRMMs and in time for renewal of a MA, but it also identifies an additional timepoint of interest at three years after implementation of aRMMs [16]. The results of our study showed that none of the aRMM effectiveness evaluations using additional pharmacovigilance activities as planned at MA could be finalised within 18 months or 36 months after MA. The probability of completing an aRMM effectiveness evaluation within five years after MA was still low. However, it should be noted that the GVP Module XVI Rev 2 and 3 do not explicitly state that evaluations should be completed at the timepoints of interest, and the guidelines suggest that timelines for measuring the effectiveness of aRMMs should be determined on a case-by-case basis for each medicinal product [4, 16]. The obligatory renewal procedure at 60 months after MA allows regulators to have a logical timepoint for assessment of the effectiveness. This is also an important timepoint from a public health perspective as the authorities perform a re-evaluation of a medicinal product, assessing whether the benefit-risk balance remains favourable [17]. Knowledge on the effectiveness of aRMMs is key for a comprehensive assessment of risks of a medicinal product.

The relatively long duration of aRMM effectiveness evaluations, as indicated by our results, may be partly explained by the time needed to design the study. This includes setting up the study, finalisation of the study protocol, and assessment of the study protocol by PRAC, which can even take several months. Besides, there may have been logistical challenges in conducting the study. This includes, for example, limited interest of HCPs and/or patients to participate in studies of medicinal products that are already marketed (e.g., time-consuming to participate). In studies using databases, the lag time of these databases can be restrictive, ranging from several months to years [18, 19]. Furthermore, MAHs might have limited incentive to complete post-marketing safety studies, including aRMM evaluation studies, in a timely manner as these could provide negative results about their product and offer no to little financial benefit [20]. Last, experience with the use of the medicinal products in practice should be gained to obtain meaningful and representative results—a step that also takes time. As stated before, the time needed to obtain representative results should be balanced with the need for timely data in interest of the MAHs and regulators to facilitate optimal pharmacovigilance.

Although the duration of the distinct time periods varied between individual studies, the time between MA and the start of the study represented a substantial portion of the total duration, even stretching to several years in some cases. Medicinal products with corresponding aRMMs should be used in practice for an appropriate period to study the aRMM effectiveness. Thus, factors that might prolong the duration of this period are delayed market launch (e.g., reimbursement approval) and/or difficulties in setting up a study. Furthermore, the duration of the evaluation study varied between studies, which may have been at least partially due to their study design. In general, assessment by PRAC seemed to vary from a few months to one year, potentially depending on the requests for supplementary information by PRAC to the MAHs. A further study with more focus on the distribution of duration among the distinct time periods is suggested to provide a more detailed examination.

We found that the probability of completing an evaluation study with assessment by PRAC was significantly higher for cross-sectional studies than for cohort studies. This finding is aligned with our expectations as we assumed that preparing and conducting cross-sectional studies (e.g., surveys) would take less time compared to preparing and conducting cohort studies (e.g., drug utilisation studies) [21]. Although cross-sectional studies are completed sooner, cross-sectional studies often include surveys, which present significant limitations, such as non-response bias [13, 15, 22, 23]. Furthermore, cross-sectional studies only allow assessment of process indicators regarding receipt, knowledge, and self-reported behaviour [13]. Whilst cohort studies need more time to be completed, these studies are subject to less bias and often allow behavioural and/or safety outcomes to be assessed.

As was shown by previous research, we found that only a minority of the aRMM effectiveness evaluation studies addressed outcome indicators alone or in combination with process indicators [12, 21, 24]. This is despite the fact that outcome indicators are considered the ultimate measures of success for aRMMs [4, 25]. According to GVP Module XVI Rev 2, assessment of process indicators should not replace but rather complement assessment of outcome indicators [4]. Thus, optimal aRMM evaluation should include a dual evidence approach (i.e., evaluating a combination of process and outcome indicators) [4, 15, 25].

This study also indicated that estimates of the duration of aRMM effectiveness evaluation studies around MA are too optimistic with the majority (81.0%) of studies being delayed when compared to their planned final report date. In contrast, previous research has indicated that 21 out of 24 (87.5%) studies complied with the agreed deadline, although the initial deadline was postponed in eight out of 24 (33.3%) studies [21]. These findings were based on a different cohort of studies, namely completed aRMM and routine risk minimisation measure evaluation studies, which were issued at MA or in the post-marketing phase. Furthermore, our study suggested that the duration of delay is substantial, with only a 55.1% probability of completing an aRMM effectiveness study within two years after the planned final report date. The reasons for delay could not be systematically retrieved from the data sources for each individual study. A previous study identified reasons for postponement of the final report date, which included low recruitment of participants and limited drug uptake [21]. As stated before, MAHs might face logistical challenges in conducting a study that might not be foreseen at MA, including delayed market launch. Regulators should be aware of and agree with the justified adjustments of milestones for and delay of aRMM effectiveness evaluation studies, given the submission of updated study protocols and PSURs. Although milestones often seem to be updated on reasonable grounds, delays are undesirable and efforts should be made to provide representative results as quickly as possible to preserve patient safety. Thus, regulators and MAHs should attend more to feasibility and justified planning of aRMM effectiveness evaluation studies to avoid unnecessary delay.

This study also has some limitations to address. First, we used the date of MA approval as a starting point to assess the duration of the evaluations because the aRMM implementation date varied per country and was difficult to retrieve. Consequently, we could not account for the time between MA and implementation of the aRMM (e.g., market launch). This potentially led to an overestimation of the effective duration of the aRMM evaluations. It should be noted that market launch is beyond the power of regulatory authorities. Second, our study was limited to aRMM effectiveness evaluations as planned in the EU-RMP at MA. Whilst risk management is an iterative process continuously applied throughout the product life cycle, we did not include post-marketing introduced aRMMs and/or effectiveness studies. For example, an additional evaluation was requested after completion of the initially planned study for some medicinal products, but we did not include these evaluations in our analyses. Third, we restricted our analyses to the submission of final study reports, so we did not take into account possible interim reports. This is despite the fact that these might give indications of and be perceived as a periodic review of the effectiveness of aRMMs. Fourth, the initial selection of aRMM evaluation studies was based on the EU-RMP at MA, often only on the reported titles and study objectives. Although cases of doubt were discussed with a second researcher, an incomplete selection of the eligible evaluation studies could not be ruled out. Last, the majority of studies were ongoing at DLP. Therefore, calculating the median time-to-event to describe the duration of the evaluations and studies was not possible.

5 Conclusion

Estimates of the duration of aRMM effectiveness studies around MA are too optimistic, with the majority of studies being delayed and a probability of submission of the final report within one year after the planned date of less than one in three. As the median planned duration until submission of the final report was 45 months after MA, information on the effectiveness of aRMMs in time of the renewal of a MA was expected to be substantial. However, the probability of completing a study within 60 months after MA was one in four and of completing a full evaluation programme within 60 months after MA for a medicinal product was only one in five (e.g., both including PRAC outcome). A consequence of the relatively long duration until aRMM effectiveness evaluations are finalised might be that the reaction time to resolve possible safety issues is slow and revised aRMMs implemented in clinical practice are protracted. In addition, time to completion seems to be dependent on study design factors that are time-dependent (such as data collection method). Further analyses with more focus on the reasons for these delays as well as in-depth analyses on factors associated with duration are suggested.