FormalPara Key Points

This post-authorisation safety study (PASS) evaluated the effectiveness of additional risk minimisation measures for the direct oral anticoagulant apixaban in Europe using healthcare professional and patient surveys.

Healthcare professional and patient respondents had satisfactory levels of knowledge on bleeding risk. Distribution of the Prescriber Guide and Patient Alert Card was not optimal, although a high proportion of recipients read and used the materials.

No changes were made to the content of the educational materials as a result of this study, but the Prescriber Guide has been distributed as an additional digital platform and the Patient Alert Card was added into the medicine packaging.

1 Introduction

Apixaban (ELIQUIS®) is an oral, direct factor Xa inhibitor that is marketed in the European Economic Area (EEA) by the Bristol Myers Squibb/Pfizer Alliance. It received its first authorisation there in May 2011 and is approved for the following indications [1]:

  • Prevention of venous thromboembolic events (VTE) in adult patients who have undergone elective hip or knee replacement surgery (hereafter referred to as ‘VTEp’)

  • Prevention of stroke and systemic embolism in adult patients with non-valvular atrial fibrillation (NVAF), with one or more risk factors

  • Treatment of deep vein thrombosis (DVT) and pulmonary embolism (PE), and prevention of recurrent DVT and PE in adults (hereafter referred to as ‘VTEt’)

The most important identified risk of apixaban, as with other anticoagulants, is bleeding.

Medicinal products in the European Union (EU) require a Risk Management Plan (RMP), which describes safety concerns that need to be proactively managed using risk minimisation measures (RMMs) and/or further studied in the pharmacovigilance plan [2]. Routine RMMs, such as the Summary of Product Characteristics (SmPC) for healthcare professionals (HCPs) and the Package Leaflet (PL) for patients, are usually sufficient to ensure a positive benefit-risk balance for most products. However, some medicines also need additional risk minimisation measures (aRMMs) that may include educational programmes or more stringent controlled access programmes/distribution systems, as described in the Guideline on good pharmacovigilance practices (GVP) Module XVI [3].

The EU-RMP for apixaban includes additional risk minimisation tools to address the risk of bleeding. These educational materials comprise a Prescriber Guide and Patient Alert Card, which provide additional communication of key safety messages beyond routine RMMs.

The goal of these tools is to further minimise the risk of bleeding associated with the use of apixaban. The educational materials were targeted at all physicians who were expected to prescribe/use apixaban for approved indications. In some countries, pharmacists and/or nurses were considered to be relevant additional target audiences.

The Prescriber Guide describes subpopulations at higher risk of bleeding, dosing and dose adjustment, managing overdose and bleeding, switching treatment, surgery/invasive procedures, temporary discontinuation, use of coagulation tests and key information about the Patient Alert Card. The Patient Alert Card covers signs and symptoms of bleeding, when to seek attention from an HCP, treatment compliance and informing HCPs that the patient is taking apixaban if they need to have any surgery or invasive procedure.

The aRMMs were introduced across EEA countries starting from December 2012 and continuing through 2013, with timings dependent on local factors such as national health authority approval. Copies of the Patient Alert Card were originally made available to HCPs together with the Prescriber Guide, so that HCPs would provide a card to their patients. Starting from April 2015, however, an updated Patient Alert Card was supplied within the apixaban product packaging as the main distribution channel. This aimed to improve compliance and tracking of the Patient Alert Card and align with the approach used for similar medications.

Evaluating the effectiveness of aRMMs is a requirement in the EU. This is an important step to ensure the effectiveness of the aRMMs and to establish whether modifications are needed to the risk minimisation tools or the way in which they are implemented [3].

Studies to evaluate risk minimisation effectiveness in the EU are usually conducted as a post-authorisation safety study (PASS), particularly when they include behavioural or safety outcome indicators, and follow detailed guidance described in GVP Module VIII [3, 4]. Cross-sectional HCP and/or patient surveys are the most common methodology used to evaluate the effectiveness of aRMMs. Despite limitations, surveys are able to provide information on metrics at multiple levels, such as the process indicators of reach and usage of the risk minimisation tools and risk knowledge of the target audience [3,4,5,6].

This study included surveys of HCPs and patients from selected EEA countries (including the UK, which at the time of the study was a member of the EEA). The primary objective was to evaluate the effectiveness of the apixaban Prescriber Guide and Patient Alert Card in terms of knowledge and comprehension of the bleeding risk associated with apixaban treatment, as communicated by the risk minimisation tools. Secondary objectives were to evaluate the effectiveness of the risk minimisation tools in terms of their distribution and utilisation, as well as behaviour of HCPs and patients. This study illustrates the experience of conducting evaluation of aRMM effectiveness in the EEA for a product with multiple indications.

2 Methods

2.1 Study Design

This was a non-interventional, cross-sectional study in the EEA involving samples of HCPs who treated patients with apixaban, as well as patients who had taken apixaban, for any of the currently approved indications. The online, multiple-choice response surveys were conducted in ten countries that represented over 90% of apixaban unit sales in the EEA between January 2013 and March 2014. These countries were Austria, Belgium, Denmark, France, Germany, Italy, Norway, Spain, Sweden, and the UK.

A total sample size of 384 HCP respondents was planned to allow for estimation of the proportion of HCPs with knowledge of the important identified risk of bleeding associated with apixaban treatment, with a 95% confidence level, a marginal error of 5% and a conservative 50% estimate for proportions of correct responses. Patient recruitment occurred via HCPs using a ratio for HCP:patient recruitment of 2:1, to avoid creating a significant burden for HCPs. A total sample of 192 patient respondents was planned with a marginal error of 7%, a 95% confidence level and 50% estimate for proportions of correct responses.

A study inclusion criterion was that an HCP was required to have been involved in the treatment of at least one patient with apixaban within the previous 9 months. Patient inclusion criteria included: aged ≥18 years at recruitment, had taken apixaban for any approved indication, and were able to understand and complete the consent form and patient questionnaire (supported by a caregiver, if needed).

2.2 Survey Populations

An initial HCP sampling frame was constructed based on relevant databases available to the Marketing Authorisation Holder (MAH) in each participating country. These comprised HCP lists that had been developed for each country to appropriately represent target HCP populations for distributing communications about the risk minimisation tools accounting for local characteristics, which were typically agreed with the national health authority. They included contact details of HCPs potentially involved in apixaban treatment for approved indications. The overall sampling frame included primary care practitioners and a range of relevant secondary care specialties including internal medicine, cardiology, neurology, intensive care, orthopaedics, surgery, anaesthesia, geriatrics and haematology.

Initial HCP lists were randomised using simple random sampling to provide a target list of HCPs per country from which participants would be invited. Recruitment was performed with an invitation mailing, including a response form, to HCPs on the initial target list and up to two follow-up mailings, if needed. If the target HCP recruitment numbers were not reached through the first randomisation round, the initial lists of HCPs were re-randomised, without replacement, to provide a further target list. Participant HCPs were also invited to refer other HCPs (e.g., physicians, specialist nurses, and pharmacists). Patient recruitment took place via HCPs, and caregivers were permitted to complete the patient survey on behalf of patients, if necessary. Patient recruitment was not mandatory for HCP participation. An honorarium could be paid as compensation for HCP time spent participating in the survey and/or recruiting patients.

The study HCP and patient samples were intended to be distributed evenly across the participant countries to be representative of these markets. In countries where recruitment of the targeted number of HCPs or patients was not expected to be achievable, over-sampling from other surveyed countries was attempted. The targeted proportion of the sample sizes for each indication were 45% NVAF, 45% VTEt and 10% VTEp; since apixaban had low market usage for the VTEp indication that meant fewer HCPs and patients were available for this indication who could potentially participate in the survey.

2.3 Survey Implementation

The surveys consisted predominantly of multiple-choice response questions, as well as a limited number of free-text fields for descriptive comments. Study participants were sent a web URL link and a unique identifier to access the online survey. Limited pre-testing was performed of both HCP and patient draft structured questionnaires, which resulted in minor changes being made to finalise their content.

The primary endpoint was proportion of HCPs using apixaban and patients treated with apixaban with knowledge of the important identified risk of bleeding associated with treatment. The secondary endpoints were proportion of HCPs using apixaban and patients treated with apixaban who have received the risk minimisation tools, proportion of HCPs and patients who have utilised the risk minimisation tools and the extent of tool usage, and distribution of behaviour questionnaire results for HCPs and patients. Data collection started in August 2015 and ended in February 2017.

Descriptive statistics, appropriate to the nature of the variable (continuous/discrete), were used and the analysis was performed using Stata and Microsoft Excel. For all variables, estimates were calculated using the total number of responses per question as the denominator. For HCP results, point estimates and standard errors were calculated with a correction factor to account for the survey design and data collection characteristics, i.e., a single stage survey design with stratified sampling across countries, where the primary sample units were sampled randomly without replacement. Moreover, strata-specific sampling weights were calculated as \({N}_{i}/{n}_{i}\), where \({N}_{i}\) was the number of HCPs in the country target list and \({n}_{i}\) was the number of HCPs in the corresponding country sample. Since there was no sampling frame for patients/caregivers, survey results for this population were not weighted.

The patient question on Recognition of potential signs and symptoms of bleeding had a mix of 12 correct and incorrect response options. A scoring system assigned one point for each right answer, with knowledge levels classified as High (9–12 points), Moderate (4–8 points) or Low (0–3 points).

3 Results

3.1 Survey Participant Information

There were 66,997 invitations issued to HCPs to participate in the survey across 9 of the 10 countries involved (excluding Italy, where the market situation resulted in local recruitment being carried out by a different provider and the number of invitations sent not being available).

A total of 1483 HCPs returned the response form expressing interest to participate. The total who responded excluding Italy was 1307 HCPs (2.0% of those invited). Of these, 280 HCPs either ultimately declined taking part or were ineligible, which left 1027 eligible HCPs willing to participate (1.5% of those invited). There were 407 HCPs (370 HCPs excluding Italy) and also 129 patients who completed their surveys.

The final number of HCP participants analysed was 385 and the final number of patient/caregiver participants analysed was 125, following the exclusion of 22 HCPs and four patients. Most of these (20 HCPs and one patient) were excluded due to an issue with institutional contracting in Spain and were replaced through recruitment of further Spanish HCPs. The HCP target sample size was met, after which no further HCPs were invited to take the survey, whilst the patient target sample size was not reached. Table 1 provides characteristics of HCP and patient/caregiver participants included in the analysis.

Table 1 HCP and patient respondent characteristics
Full size table

3.1.1 HCP Characteristics

Numbers of participating HCPs across the 10 countries were generally similar, as intended, and target proportions were met for indications. Among the 385 analysed HCP participants, high proportions were physicians (340; 88.3%) and apixaban prescribers (348, 90.4%). The highest study participation was found among HCPs working in general practice (33.8%), cardiology (26.8%), internal medicine (12.7%) and neurology (7.0%). By comparison, the most common specialties of HCPs invited to participate (data available for 9 countries, excluding Italy) were: general practice 40.7%, internal medicine 13.2%, cardiology 5.8% and neurology 4.1%.

3.1.2 Patient Characteristics

The number of patient survey respondents varied significantly between countries. The highest number came from Sweden (28 of 125 analysed; 22.4%), due in part to oversampling per study protocol to compensate for low recruitment from other countries. Most patients were prescribed apixaban for the NVAF indication (83, 66.4%). There were 10 (8.0%) caregivers who completed the survey on behalf of patients.

3.2 Assessment of Knowledge on Managing the Risk of Bleeding

3.2.1 HCP Knowledge Results

The effectiveness of the risk minimisation measures in terms of knowledge of the important identified risk of bleeding associated with apixaban treatment was assessed using several survey questions and key results are presented in Tables 2 and 3.

Table 2 HCP responses to key knowledge questions impacting risk of bleeding
Full size table
Table 3 HCP knowledge of patient groups at increased risk of bleeding complications
Full size table

Early recognition of symptoms that require immediate contact with an HCP: 370 HCPs (96.1%; weighted: 0.94) recognised the correct response, i.e., bleeding which does not stop on its own. The weighted proportion of HCPs who answered correctly was slightly higher for HCPs who stated they used the risk minimisation tools compared to those who did not (0.96 vs 0.90, respectively).

Knowledge of contraindications: 293 HCPs (76.1%; weighted: 0.78) correctly responded which one of the patient subgroup options presented was not an apixaban contraindication. The weighted proportion of correct responses was higher for HCPs who used rather than did not use the tools (0.82 vs 0.69).

Knowledge of correct dosage: 291 prescribers (83.6%; weighted: 0.80) responded with the correct standard apixaban dose for the indication specified. The weighted proportion of correct responses was slightly higher for HCPs who used rather than did not use the tools (0.82 vs 0.75).

Knowledge of patient subpopulations at increased risk of bleeding complications: prescriber responses on whether or not five patient subgroups were at increased risk of bleeding complications when treated with apixaban are shown in Table 3. The number of correct responses ranged from 221 HCPs (63.5%) to 299 HCPs (85.9%) across the five specified patient subgroups, which corresponded to weighted proportions of correct HCP responses varying from 0.604 to 0.902.

3.2.2 Patient/caregiver Knowledge Results

The effectiveness of the risk minimisation measures in terms of patient knowledge was assessed using several survey questions, and key results are provided in Tables 4 and 5.

Table 4 Patient/caregiver knowledge of bleeding risk and communicating risk factors to HCPs
Full size table
Table 5 Patient knowledge of signs and symptoms of bleeding while taking apixaban
Full size table

Knowledge of bleeding as an important adverse reaction: 89 patients (71.2%) responded correctly that abnormal bleeding is an important side effect of apixaban treatment. The proportion who answered correctly was higher among respondents who indicated they had used the Patient Alert Card compared to non-users (79.4% vs 62.9%, respectively).

Communicating risk factors to HCPs: 96 patients (76.8%) responded correctly on communicating to HCPs, conditions that cause abnormal bleeding. The proportion who answered correctly was similar between respondents who indicated they had used or had not used the Patient Alert Card (76.2% vs 77.4%).

Recognition of potential signs and symptoms of bleeding: 28 patients (22.4%) had high, 62 (49.6%) had moderate and 35 (28.0%) had low knowledge level scores (Table 5). For respondents who used the Patient Alert Card, 30.2% exhibited high knowledge levels compared to only 14.5% of those who had not used it. Furthermore, the proportion with low knowledge levels was less among those who had used the risk minimisation tool compared to those who had not (19.1% vs 37.1%).

3.3 Assessment of Distribution and Usage of the Risk Minimisation Tools

Of the 385 HCP participants, 226 (58.7%) responded that they had received or obtained the apixaban Prescriber Guide. Furthermore, the majority of patients/caregivers (74; 59.2%) indicated that they were aware of the Patient Alert Card. Of those, 66 (89.2%) indicated that they had received or obtained a copy (Table 6). An overwhelming majority of HCP respondents (221, 97.8%) who had received the Prescriber Guide had read at least part of it and patient respondents (60, 90.9%) who had received the Patient Alert Card had read it at least once.

Table 6 Receipt of apixaban risk minimisation tools
Full size table

Most HCP respondents (169, 74.8%) had used the Prescriber Guide to discuss apixaban with patients. When asked about usage of the Patient Alert Card, 41 (62.1%) patients who had received it responded that they kept the card with them at all times and 19 (28.8%) responded most of the time (Table 7).

Table 7 Use of apixaban risk minimisation tools
Full size table

3.4 Assessment of Participant Behaviour

Key results from questions assessing behaviour showed that most HCPs (306, 79.5%) indicated that they had discussed with their apixaban patients the need to seek immediate medical attention for a bleeding event which did not stop on its own. The majority of patients (91, 72.8%) responded that an HCP had discussed the risk of bleeding with them, whilst 24 (19.2%) replied the risk of bleeding had never been discussed, and 10 (8.0%) did not know or could not recall. HCPs had discussed the risk of bleeding for a higher proportion of patient respondents who used the Patient Alert Card compared with those who did not (77.8% vs 67.7%, respectively).

When asked what they would do about a bleeding event that did not stop on its own, a high proportion of patient respondents (108, 86.4%) answered correctly that they would seek medical attention immediately, 11 (8.8%) answered incorrectly, and 6 (4.8%) did not know. The proportion of correct responses was slightly higher for patients/caregivers who had used the Patient Alert Card than in those who had not (88.9% vs 83.9%).

3.5 Utility of the Risk Minimisation Tools

An overwhelming majority of HCP respondents who received the tools found the Prescriber Guide (216, 95.6%) to be either very or quite useful and judged the Patient Alert Card (207, 97.6%) to be either very or quite useful for patients (see Table 8).

Table 8 Utility of apixaban risk minimisation tools
Full size table

A high proportion of patient/caregiver respondents who had read the Patient Alert Card (55, 91.7%) found it either very or quite useful and around half (31, 51.7%) stated that the card contained information about apixaban they had not previously known.

4 Discussion

This study aimed to evaluate the effectiveness of the apixaban Prescriber Guide and Patient Alert Card in terms of HCP and patient knowledge of associated bleeding risk, as well as distribution of the tools, their utilisation and target audience behaviour. It analysed responses from 385 HCP survey participants and 125 patient survey participants from 10 EEA countries.

Knowledge levels were assessed using several different survey questions. Key questions posed to HCPs on risk knowledge had the following proportions of correct answers: early recognition of symptoms requiring immediate contact with an HCP (96.1%), appropriate dosing (83.6%), contraindications (76.1%) and subpopulations at increased risk of bleeding complications (ranging from 63.5 to 85.9%). Weighted proportions of HCPs responding correctly to these questions were slightly higher for risk minimisation tool users than non-users.

Key questions for patients about risk knowledge had the following proportion of correct answers: abnormal bleeding being an important side effect (71.2%), communicating risk factors to HCPs (76.8%) and recognition of potential bleeding symptoms (‘high’ knowledge levels 22.4%, ‘moderate’ knowledge levels 49.6%). The proportion of correct patient responses was higher or similar for risk minimisation tool users compared to non-users.

Hence, the proportion of correct responses to key questions testing knowledge was generally high, with HCPs mostly scoring higher than patients. Although the patient question on recognition of potential bleeding symptoms only had a limited proportion of respondents exhibiting high knowledge level scores, this was likely impacted by the question being relatively complex and challenging with 12 response options.

These results are satisfactory when compared to rates of knowledge observed for safety concerns in other EU aRMM effectiveness evaluation surveys, which provide a benchmark indicator. A systematic review found that the proportion of correct HCP responses for 69 key safety concerns assessed in 22 studies was above 60% for most (76.8%) and above 80% for some (40.6%) items. The proportion of correct patient responses for 21 key safety concerns assessed in 6 studies was substantially lower, being above 60% for some (38.1%) and above 80% for few (9.5%) items [7].

The apixaban surveys also evaluated effectiveness of risk minimisation tool distribution and utilisation. A majority of HCP participants (58.7%) recalled receiving/obtaining the Prescriber Guide, of whom nearly all (97.8%) had read at least part of it and most (74.8%) had used it to assist patient discussion. A majority of patient respondents (59.2%) were aware of the Patient Alert Card, of whom most (89.2%) recalled receiving/obtaining a copy. A high proportion of those who received the Patient Alert Card had read it (90.9%) and kept it with them at least some of the time (93.9%).

These results for risk minimisation tool distribution and utilisation are consistent with pooled estimates reported for other EU aRMM effectiveness evaluations. In surveys covering HCP brochures/leaflets/guides, 54.1% of HCPs reported receiving them and 71.7% used them if received, whilst more generally, 89.8% of HCPs reported reading some or all of an aRMM material (mostly an HCP brochure) if received. In surveys covering patient alert cards, 55.4% of patients reported receiving them, with 88.4% reading and 62.6% using the cards if received [7].

The apixaban surveys also assessed target audience behaviour. High proportions of respondents reported appropriate actions to manage apixaban bleeding risk, including HCPs discussing the need to seek immediate medical attention for a bleeding event, patients recalling an HCP discussing the risk of bleeding, and patients answering they would seek medical attention immediately for a bleeding event that did not stop on its own.

Importantly, the apixaban surveys showed that the risk minimisation tools were valued, with very high proportions of both HCP and patient recipients finding them useful. Overall, the reach of the apixaban risk minimisation tools was consistent with levels reported for aRMMs of other medicinal products, although not optimal, since over 40% of survey participants did not recall receiving or obtaining the tools. When HCPs and patient respondents did receive the apixaban risk minimisation tools, high proportions used them and considered them beneficial.

4.1 Modifications Resulting from the aRMM Effectiveness Evaluation

As the study was conducted as a PASS in the EEA, both the study protocol and study report were assessed by the Pharmacovigilance Risk Assessment Committee (PRAC) of the European Medicines Agency (EMA). The benefit-risk balance of apixaban was judged unchanged as a result of the evaluation. There was also considered to be no need to modify the content of the aRMMs or the product information. However, since the survey results indicated that the reach of the risk minimisation tools was not optimal, action was taken to improve their distribution.

At the time of the study, the Patient Alert Card was already starting to be supplied within the product packaging as the primary mode of distribution. This change was expected to increase the reach of the card once the new stock was circulated, since patients would receive a copy with each pack of medicine rather than relying on an HCP receiving and then handing out a copy to them.

The proposed solution for improving the reach of the Prescriber Guide was to make it available in a new format, as a digital platform available in each country in appropriate language(s). This additional channel presents the content in an easily navigable, user-friendly webpage format where relevant risk information (such as indication-specific content) can be quickly selected. The digital platform goes beyond merely providing copies of the risk minimisation tools as Portable Document Format (PDF) files, although these are also available to download.

This digital platform has been rolled out in EEA countries and complements the existing paper-based educational material. It also provides opportunities to analyse data about how the digital risk minimisation tool is being used and to collect feedback from HCP users.

4.2 Study Limitations and Generalisability of Findings

This type of observational study could be influenced by several well-known types of bias. Efforts were made to minimise bias during data collection and analysis. The HCP and patient/caregiver participants were drawn from 10 countries that represented a high proportion of apixaban usage within the EEA and included a mix of large and small countries.

The source population was based on HCP lists that had been developed for each country to appropriately represent target HCP populations for sending communications about and copies of the apixaban risk minimisation tools, which took into account local factors. In the absence of being able to directly identify and target the population of all apixaban prescribers, due to privacy restrictions, this was considered a suitable approach, although to an extent signified a convenience sample. HCPs representing both primary care and a variety of secondary care specialties across different institutional settings were included and participated in the surveys. A positive was that the pre-specified HCP sample size target was met.

Available data on HCPs who were sent invitations show a 2.0% response rate in HCPs expressing interest in participating in the survey. Although this rate was low, it was consistent with low response rates generally seen for other studies that have evaluated effectiveness of aRMMs [5, 7,8,9]. An honorarium was paid, where possible, to compensate for time spent by HCPs, which was a potential source of bias.

Classifying the study as a PASS had significant implications due to GVP guidance and varying national regulations. It resulted in more stringent requirements for conducting the surveys in some countries, as has been noted by others [8, 10]. These increased the administrative burden for potential participants, decreasing the likelihood of them taking part in the research, and led to delays in starting recruitment. Ethics Committee approvals were required for conducting the patient surveys in some countries. These operational factors are likely to have had a significant negative impact on recruitment, particularly of patients, within the limited time period available for data collection.

The low response rate increased the potential for selection bias, requiring caution in interpreting the study findings more widely. Some degree of selection bias (volunteering and non-participation) within the study was anticipated. Non-response and volunteering bias was examined by comparing demographic and other available characteristics of the total sampled population of HCPs. There were differences between the invited HCPs and those HCPs who expressed interest in participating. The largest variations for specialty were a higher proportion of cardiologists and a lower proportion of general practitioners amongst HCPs who responded. This is likely to have at least partially reflected the distinction between potential apixaban prescribers who were targeted for risk minimisation tool distribution and actual prescribers who were eligible for the survey, since a greater proportion of cardiac specialists than primary care physicians would be expected to treat patients with apixaban.

Patient recruitment took place via HCPs, so no sampling frame was available. Patient recruitment was challenging, and the pre-specified patient sample size was not met. Patients invited might have differed systematically from those not invited, as well as from those who participated. Furthermore, the number of patient respondents per country and per indication varied significantly, with very low patient recruitment in some countries and for the VTEp indication. This was partially addressed by oversampling in other countries and the NVAF indication to increase the overall sample size. The patient sample had a similar gender split but a lower proportion of elderly patients in comparison to apixaban NVAF patients in a large US observational study [11]. These factors limit the generalisability of results observed in patients. However, the patient results do still provide useful information, offering an advantage over study designs that omit patients from evaluations due to anticipated recruitment challenges.

The study design aimed for equal numbers of participants per country and specified proportions per indication, which were largely achieved for HCPs but not for patients. A strength was that the overall sample reflected a diverse range of apixaban risk minimisation environments, particularly for HCPs, but this did result in relatively limited sample sizes per country and for the VTEp indication. Thus, the design increased the potential for selection bias at the individual level within countries but reduced it at the country and indication levels, compared to surveys focussing on a smaller number of countries or indications.

To provide an additional perspective for the overall sample, sampling weights (reciprocal to the probabilities of selection) were applied for HCPs to draw disproportionately from under-represented strata. The representativeness of each strata sample was assessed against the size of the initial HCP lists used for invitations, based on the assumption that those were representative of the population from which they were drawn. However, this might have not been the case for some countries.

There may have been some information bias, particularly for behaviour which was indirectly evaluated and partly relied on self-reporting by respondents. It is also possible that participants may have read the risk minimisation materials or product information around the time of the survey before responding.

Limitations of the multiple-choice response survey design include enabling correct answers to be selected by chance due to a limited number of response options and not assessing knowledge adoption to the extent of asking participants to recall and explain the safety concern and safe use advice. As a cross-sectional study, data were collected from each participant at a single point in time. Hence, the results reflect available data at the time when the survey was administered.

The significant biases described above limit the generalisability of the survey findings. These biases though are typical for cross-sectional surveys measuring RMM effectiveness, which are a well-established and commonly used methodology for conducting such assessments despite their limitations [3, 5, 8, 12]. A review found that 24 of 29 ongoing and finalised studies evaluating RMM effectiveness registered in the EU Post-Authorisation Studies (PAS) Register, including this one, used cross-sectional surveys as a primary data source [13]. Surveys remain key to evaluation because they can efficiently provide information from substantial numbers of stakeholders about process indicators at multiple levels—particularly important for knowledge, but also useful for reach and usage of the risk minimisation tools, and to some extent behaviour.

A systematic review of risk minimisation interventions found that effectiveness evaluations had only been reported for about one-third of identified interventions at the time of its publication [14]. The methodology used for the apixaban aRMM effectiveness evaluation was selected to be proportionate to the risk being minimised. Alternative approaches could have potentially been used to meet study objectives of evaluating knowledge, tool distribution, utilisation and behaviour for the educational aRMMs.

Knowledge can be ascertained by asking stakeholders direct and open-ended questions rather than using a predominantly multiple-choice response design, although data may become more difficult to interpret with larger sample sizes. Distribution could be measured by the quantity of risk minimisation materials sent out, but this is weaker evidence of reach than stakeholders confirming whether they actually received materials. Risk minimisation tool usage could be ascertained through stakeholders accessing digital versions of materials or covering this aspect in a prospective drug utilisation study.

Of 17 studies that assessed HCP prescribing or monitoring behaviour as part of evaluations in the EU PAS Register, ten had a cross-sectional survey design whilst seven used secondary data sources to perform retrospective medical chart reviews or analyses of electronic medical records [13]. Surveys provide relatively weak evidence of behaviour, particularly if self-reported due to social acceptability bias, so alternative methods such as drug utilisation studies can increase robustness. A limitation of using secondary databases is that they are typically limited to a single EEA country, which restricts the generalisability of findings.

Process indicators encompass programme execution and impact on knowledge and behaviour, which have been reported for this study. There is also a second category, namely outcome indicators, which provide an overall measure of the level of risk control achieved by the RMMs. According to GVP guidance, these are safety outcomes comprising the frequency and/or severity of relevant adverse reactions in a non-interventional study setting [3].

Although positioned at the top of the evidence hierarchy, outcome indicators can be more challenging for demonstrating RMM success. It is usually difficult to prove a direct link to the impact of RMMs unless a pre-post intervention design can be employed comparing rates before and after introduction of the RMMs, which is often not possible. A meaningful pre-post comparison for the apixaban aRMMs was not feasible. However, safety outcomes can still provide an indication of whether there is an acceptable level of the relevant risk outcome to maintain a positive benefit-risk balance.

Safety outcome studies may use existing data sources, if these already sufficiently capture data elements of interest, or collect primary data. Of the 29 studies evaluating RMM effectiveness in the EU PAS Register, only five were found to measure outcome indicators as well as process indicators. Methods used included retrospective medical chart reviews, direct observation by HCPs and data from a patient registry [13]. The incorporation of outcome indicators has been increasingly encouraged in recent years. Alternative approaches include retrospective cohort or cross-sectional studies that make use of various electronic healthcare databases available within EEA countries or to employ quasi-experimental designs. Potential challenges include the quality of available secondary data sources and the length of time needed to obtain results from prospective studies [15].

Although not linked to this aRMM effectiveness evaluation study, safety outcomes for bleeding events associated with apixaban have been evaluated in real-world settings in other studies, including those conducted within EEA countries. A nationwide registry study in atrial fibrillation patients that started close to the time of initial distribution of apixaban risk minimisation materials in Norway, one of the evaluated countries, found that apixaban was associated with a lower risk of major or clinically relevant non-major bleeding compared with warfarin [16]. Similarly, a retrospective cohort study in atrial fibrillation patients using a German claims database that covered a period starting just after initial local distribution of the materials found that apixaban was associated with a lower risk of major bleeding than the vitamin-K antagonist phenprocoumon [17].

More generally, a systematic review and meta-analysis that included some European studies found that when compared with vitamin-K antagonists for stroke prevention in atrial fibrillation, apixaban was associated with lower risk of major haemorrhage whereas the oral anticoagulants dabigatran and rivaroxaban had similar risk [18]. Hence, safety outcomes data available for apixaban provide reassurance about the risk of bleeding.

5 Conclusion

This PASS evaluated the effectiveness of the apixaban risk minimisation tools in the EEA for both HCPs and patients. Survey responses from 10 countries showed that the Prescriber Guide and Patient Alert Card were associated with satisfactory levels of knowledge on managing bleeding risk. The reach of these educational materials amongst survey participants was consistent with available literature, although not optimal. A high proportion of HCP and patient respondents who received the materials had used them and found them to be useful. In addition, high proportions of HCP and patient respondents reported appropriate behaviour for managing bleeding risk.

No changes were required to be made to the content of the risk minimisation tools as a result of this evaluation. There was, however, a change of approach for how the Prescriber Guide is distributed. To increase reach, the Prescriber Guide content has been developed as a digital platform, which provides an additional up-to-date channel for HCPs to easily access the risk minimisation information. This web-based platform has been rolled out in EEA countries. The Patient Alert Card is also now distributed within each pack of medicine.