FormalPara Key Points for Decision Makers
Table 1
Full size table

1 Introduction

Risk minimization measures are interventions intended to reduce or prevent the occurrence of adverse reactions, or their sequelae, associated with exposure to medical treatments. These are recommended as part of good pharmacovigilance practices (GVP), for example, as detailed in the European Medicines Agency (EMA) GVP Modules V and XVI [1, 2]. Routine risk minimization measures are applicable to all medicinal products and can take the form of the Summary of Product Characteristics (SmPC), appropriate medicine labeling, safety information in the package leaflet, and pack size. The majority of safety concerns are addressed by these routine measures. However, for certain medicinal products, when routine measures are deemed to be insufficient for the safe and effective use of a medicine, additional risk minimization measures (aRMM) are also necessary [1,2,3]. A number of aRMM tools are currently utilized (e.g. educational programs [2], controlled access programs, controlled distribution programs, etc.), and this field is continuously evolving with technological advances, such as interactive web- and application-based tools for healthcare professionals (HCPs) [4].

Voriconazole (VFEND®) is an extended-spectrum triazole antifungal agent indicated for the treatment of invasive aspergillosis, candidemia in non-neutropenic patients, and other serious fungal infections, in adults in both the USA and the EU since 2002. It is also approved in children over the age of 12 years in the USA [5] and in children aged over 2 years in the EU [6]. Voriconazole is also indicated in the EU for the prophylaxis of invasive fungal infections in high-risk hematopoietic stem cell transplant (HSCT) recipients [6]. Voriconazole is effective for the primary and secondary prevention of invasive fungal infections including invasive aspergillosis in patients with hematologic diseases and HSCT recipients with a favorable risk-benefit balance [7].

In light of a number of spontaneous reports, clinical trial findings, and other publications about the risks of phototoxicity [8,9,10,11], squamous cell carcinoma (SCC) of the skin [12,13,14,15,16], and hepatotoxicity [17, 18] with voriconazole, an aRMM program was implemented beginning in April 2014 based on rolling local health authority approvals in 33 EU countries (with implementation dates from April 2014 through December 2014). The goals of the aRMM tools were to increase HCPs’ awareness of these three risks and their management. Specialty care physicians who generally prescribe voriconazole, including infectious disease physicians, oncologists, hematologists, and solid organ transplant physicians, were mailed the hard copies of the aRMM tools, which included an 8-page (with 5 pages of content) HCP Question and Answer (Q&A) Brochure, a 2-page HCP Checklist, and a double-sided, wallet-sized Patient Alert Card (Table 1).

Table 1 Description of the aRMM tools
Full size table

Module XVI of EMA GVP stipulates that the effectiveness of aRMM should be evaluated for all medicinal products that utilize such tools [2]. A systematic evaluation of the effectiveness of aRMM helps to understand whether the program objectives have been met (i.e. whether an intervention has been effective) or provide evidence that further amendments to the program are needed [1, 2]. These evaluation studies of aRMM are considered post-authorization safety studies (PASS) in the EU and are regulatory commitments. We evaluated the effectiveness of the aRMM via a survey of HCPs who were prescribers of voriconazole (EUPAS12624). The overall objective of this PASS was to evaluate the effectiveness of the voriconazole aRMM tools implemented across Europe in mitigating the risks of phototoxicity, SCC of the skin, and hepatotoxicity in patients receiving voriconazole; specifically, to assess HCPs’ receipt and utilization of the aRMM tools, knowledge of the risks of phototoxicity, SCC, and hepatotoxicity, and knowledge of risk minimization practices, as well as self-reported practices, with respect to minimizing these risks in accordance with the prescribing information.

The results of the evaluation are presented and discussed in the context of the strengths and limitations of the evaluation methodology, factors influencing target population behavior, and knowledge and lessons learned from the study. As recommended by the Council for International Organizations of Medical Sciences (CIOMS) Working Group in their report [19], this manuscript was developed to add to the limited body of published evidence, and to help facilitate advances in the risk management and evaluation field.

2 Methods

2.1 Study Design

This was a voluntary cross-sectional survey of HCPs conducted 12–17 months after the distribution of aRMM tools, depending on country. A structured, self-administered questionnaire was used to collect the survey data. The HCPs were invited to take the online survey using a secure uniform resource locator requiring a unique identifier. Eligible HCPs to whom the aRMM tools were mailed, received a letter by postal mail inviting them to participate in the survey. HCPs also received an email in addition to postal mail in three countries where email addresses were available (France, UK, and Italy).

2.2 Survey Population

The survey population constituted HCPs from 10 EU countries who were mailed the aRMM tools: Austria, Denmark, France, Germany, Hungary, Ireland, Italy, the Netherlands, Spain, and the UK. The inclusion criteria included those HCPs who self-reported prescribing voriconazole at least once within 12 months preceding the survey and consented to participate in the survey. Exclusion criteria included non-practicing physicians, those who had participated in pilot testing of the aRMM tools or draft survey, and employees of Pfizer or of a research organization/vendor contracted by Pfizer to administer the survey.

2.3 Survey

Using established survey methodology [20], an online survey was developed to assess the HCPs’: (1) receipt, awareness, and utilization of the aRMM tools; (2) knowledge of the risks of phototoxicity, SCC of the skin, and hepatotoxicity with voriconazole; (3) knowledge of the practices recommended to minimize these three risks in accordance with the aRMM tools; and (4) self-reported risk minimization behaviors for these three risks (Electronic Supplementary Material).

Prior to launching the survey, moderator-guided user testing of the survey instrument was conducted to test clarity and comprehension of the draft questions, with a sample of potential prescribers of voriconazole in the UK. Overall, most physicians involved in the user testing found the voriconazole aRMM effectiveness evaluation survey instrument to be clear and comprehensive and did not propose significant revisions to the instrument. Instead, their feedback tended to focus on their clinical experiences using voriconazole. Physicians also suggested minor formatting, language choice, and editorial changes to improve the clarity of the survey. These were incorporated as appropriate prior to the implementation.

In addition, an operational feasibility assessment was completed, and relevant Competent Authority, Ethics Committee, and/or Institutional Review Board reviews were conducted according to each country’s local regulations. The ability to provide a nominal honorarium to reimburse HCPs for their time to complete the survey was also assessed. However, due to variations in local processes [21] and/or significant administrative burdens for a brief one-time survey, which would preclude survey implementation in a timely manner, it was only deemed feasible to provide an honorarium in Spain.

The final survey comprised 21 questions, which evaluated each of the 4 objectives. Other survey questions included screening/eligibility questions, demographic characteristics, and HCPs’ opinion about the usefulness of aRMM tools in clinical practice.

2.4 Statistical Analysis

Descriptive statistics were used to analyze the survey data including counts and proportion of correct/desired responses with exact two-sided 95% confidence intervals by the method of Clopper–Pearson [22]. An a priori threshold of 80% correct responses per risk questions was used to define the success of the program. The selection of this threshold for success was regarded as being subjective and not based on prior knowledge, experience, or established scientific criteria, as per previous publications [23,24,25].

3 Results

3.1 Survey Response

Of 27,396 specialty HCPs invited to participate, 26,735 invitations were successfully delivered, 447 respondents (1.7%) completed the online screening (visited the survey website and provided their unique ID from the invitation letter), 354 (1.3%) were eligible for participation, and 332 (1.2%) completed the survey (Table 2). The most frequent primary medical specialties of the completers were hematology (46.1%) and oncology (26.8%; Table 2). Almost half of the prescribers who completed the survey had been practicing medicine for 15 or more years (48.8%) and an additional 38.3% had been practicing for between 6–15 years.

Table 2 Summary of country of practice and medical specialty of survey participants
Full size table

3.2 Receipt and Utilization of Voriconazole aRMM Tools

The average proportion of undeliverable aRMM material packages across all ten study countries was 1.4% (range 0.3−4.1). However, of 332 survey completers, 19.6% (n = 65) recalled receiving the HCP Q&A Brochure, 22.6% (n = 75) recalled receiving the HCP Checklist, and 25.9% (n = 86) recalled receiving the Patient Alert Card (Fig. 1), with the majority (47.0–56.9%) of respondents stating that they did not remember receiving the aRMM tools. Of those who reported receipt of the HCP Q&A brochure, 33.8% (n = 22) reported reading the entire brochure with 53.8% (n = 35) reading some of the brochure; whereas 17.3% (n = 13) of HCPs who reported receipt of the checklist also stated that they always completed the HCP Checklist when prescribing voriconazole and 48.0% (n = 36) stated that they sometimes used it. Of HCPs who reported receipt of the Patient Alert Card, 25.6% (n = 22) always and 41.9% (n = 36) sometimes filled in and distributed the card to patients. The proportion of reported aRMM receipt and utilization was slightly higher among those HCPs who treated the highest number of patients (n > 20) with voriconazole in the past 12 months, compared with those who reported treating fewer patients (n ≤ 20) with voriconazole (Fig. 2).

Fig. 1
figure 1

Proportion of HCP respondents reporting receipt of the three aRMM tools (Question: “Did you or your hospital receive the HCP Q&A Brochure/HCP Checklist/Patient Alert Card?”). aRMM additional risk minimization measures, CI confidence interval, HCP healthcare professional, Q&A question and answer

Full size image
Fig. 2
figure 2

Proportion of correct responses to questions on receipt and utilization of the aRMM tools by HCPs, stratified by the number of patients they had treated with voriconazole in the past 12 months. AE adverse event, aRMM additional risk minimization measures, CI confidence interval, HCP healthcare professional, Q&A question and answer

Full size image

3.3 HCPs’ Knowledge of Risks

Of the 332 survey completers, the following proportions correctly identified phototoxicity [88.6% (n = 294)], SCC [44.3% (n = 147)], and hepatotoxicity [96.4% (n = 320)] as risks associated with voriconazole.

3.4 HCPs’ Knowledge of Risk Minimization Measures for Phototoxicity, SCC of the Skin, and Hepatotoxicity

Respondents were generally knowledgeable about practices recommended to mitigate the risks of phototoxicity and SCC of the skin (> 75.0% correct responses for 5/7 questions). However, some questions had lower proportions of correct responses. For example, 34.9% correctly answered that dermatologic evaluation should be performed on a systematic and regular basis, and 62.0% correctly answered that sun-exposed areas of the skin should be covered (Table 3).

Table 3 Proportion of correct responses to questions about the risks of phototoxicity and SCC of the skin and associated risk minimization practices
Full size table

Similar to the knowledge of the risk of phototoxicity and SCC of the skin, respondents were well informed about mitigating the risk of hepatotoxicity (> 78.0% correct responses for all four questions; Table 4).

Table 4 Proportion of correct responses to questions about the risk of hepatotoxicity and associated risk minimization practices
Full size table

3.5 Effect of Reading the HCP Q&A Brochure on Risk Minimization Knowledge and Behavior

Although the study was not powered for statistical comparisons, descriptive data showed that reading the HCP Q&A Brochure was associated with slightly improved knowledge of risks and risk minimization questions (Fig. 3) compared with those who indicated that they did not read the brochure. Similarly, self-reported risk minimization behavior appeared slightly improved in those who indicated that they had read some or all of the brochure, compared to those who had not read it (Fig. 3).

Fig. 3
figure 3

Proportion of correct responses to the three risks, knowledge of risk minimization questions, and associated risk minimization behavior by HCP Q&A Brochure reading status of the respondents. AEs adverse events, CI confidence interval, HCP healthcare professional, SCC squamous cell carcinoma, Q&A question and answer

Full size image

Subgroup analyses were also conducted that looked at the effects of country, medical specialty, and the frequency of voriconazole prescribing on knowledge and risk minimization practice. However, the sample sizes of many of the subgroups were too small to permit comparisons and meaningfully assess the differences. The effect of frequency of prescribing on knowledge and risk minimization practices is shown in Supplementary Tables 1 and 2.

3.6 Perceived Usefulness of the aRMM Tools

The distribution of perceived usefulness of the aRMM tools, by HCPs who reported receipt of the materials, was similar across the different tools. A small number of HCPs found them either ‘extremely useful’ or ‘not useful’ (Supplementary Table 3).

4 Discussion

This study was conducted to evaluate the effectiveness of the aRMM implemented across the EU to mitigate the risks of phototoxicity, SCC of the skin, and hepatotoxicity with voriconazole. Among respondents, we found that, in general, knowledge of the risks of phototoxicity and hepatotoxicity was high (88.6% and 96.4%, respectively), but appeared to be lower for SCC of the skin (44.3%).

The study results reported here should be interpreted with caution due to the low survey response rate, and thus unknown generalizability, a common challenge of surveys among HCPs [26]. Participation in the survey was voluntary and, despite extensive follow-up efforts, a relatively small proportion responded (1.7%) and completed (1.2%) the survey.

As a result of this low response rate, there is a possibility that the respondents may differ from those who did not respond to the survey in terms of characteristics, motivations, awareness of the aRMM, and knowledge of voriconazole risks. Indeed, when survey completers were compared with all invited HCPs in terms of country of practice and medical specialty, the results showed that the distribution of these characteristics was different between the two groups [e.g. the majority (57.5%) of survey completers were from Spain; Table 2]. Thus, a potential selection bias cannot be ruled out and the generalizability of the study results to all prescribers of voriconazole is unknown.

Some interesting differences in response rates between countries were observed in this study. We note that the highest response rate of 4.2%, which was achieved in Spain, could have been in part related to the fact that Spain was the only country where we were able to offer a small honorarium for participation. In the EU, the requirements for offering honoraria payments are variable and often quite strict due to classification of such studies as PASS. This variability has been noted by others as a significant barrier to operational success and scientific interpretation [26].

Furthermore, the differential proportions of the physician specialties in the study population compared with the overall invited population may also reflect the underlying prescriber population for whom aRMM materials for voriconazole were relevant. It is one of the well-known limitations of conducting this type of research among HCPs in the EU, that due to strict privacy regulations [21] (such as General Data Protection Regulation (GDPR) [27]), it is not possible for sponsors to obtain access to details of those prescribing voriconazole, which could have been utilized to develop more targeted lists for both distribution of aRMM materials and evaluation, and to better contextualize non-responders.

In this evaluation, the response rate (1.7%) was much lower than expected (3–7%) based on the authors’ experience with similar studies in Europe (data on file). In the USA, the Food and Drug Administration (FDA) noted that it is not uncommon to observe response rates to surveys below 10% [28]. Previous recommendations suggest that survey response rates of at least 60% are preferred to ensure that non-response bias does not influence the validity of survey findings [29]. Efforts to improve the response rate during the data collection were taken, including conducting all communications in the local language and providing reminders to complete the survey (56,080 postal reminders and 4236 email reminders). Over recent years there has been a steady downward trend in clinicians’ response rates to surveys [24, 30,31,32], which could partially explain the low response rate observed in this survey.

Another limitation of this survey was the absence of comparative data from pre-aRMM implementation (baseline data). Being able to evaluate HCPs’ knowledge and risk minimization practices pre- and post-aRMM implementation would enable stronger conclusions about the effectiveness of this aRMM program. Whilst it would have been desirable to have baseline data from before aRMM implementation, it was not feasible due to the need to implement the tools quickly and efficiently following the EMA’s request. Furthermore, the threshold for program success (80%) was also not based on published literature, prior knowledge, or established scientific criteria, because pertinent data are currently lacking. A review of all completed aRMM evaluation surveys (classified as PASS) from the EU PAS Register found that only 2 of 11 cross-sectional surveys selected 80% as a predefined threshold of success [23]. Better understanding of the suitable thresholds for aRMM success [33] and whether these thresholds differ by stakeholder, specialty, risk, or other factors, would be valuable when making conclusions about the program’s effectiveness.

Surveys are a well-established standard to measure process indicators such as distribution and utilization of educational materials, as well as knowledge and self-reported behavior [26]. However, despite many strengths, surveys also have limitations [24]. The low response rates from surveys suggest that providing feedback on a voluntary basis does not always achieve the desired level of response [33]. Many limitations are illustrated by this survey. The utility of such surveys in evaluating the effectiveness of the aRMMs and drawing meaningful conclusions is questionable considering the lack of comparative data, the extremely low response rate, unknown generalizability, and potential selection bias (e.g. HCPs who took the time to complete to this survey were likely to be better informed about or have greater interest in pharmacovigilance than those who did not respond; the majority of respondents were from one of the many countries surveyed). Other study designs and frameworks to assess effectiveness of aRMMs should be considered in future evaluation work [34,35,36].

An additional notable limitation of this effectiveness evaluation is the sole focus on process indicators (i.e. receipt and utilization of aRMM materials, knowledge, and self-reported practice pertaining to risk minimization). We were unable to measure outcome indicators (i.e. incidence of safety risks) pre- and post-implementation of the program via either primary or secondary data collection studies due to the long latency of outcomes such as SCC [37], combined with the relatively low frequency of voriconazole usage; under these circumstances, meaningful changes in risks would be nearly impossible to observe. Thus, even though assessment of both process and outcome indicators is strongly encouraged, in some situations, such as this, assessment of outcomes is not feasible and conclusions about effectiveness can only be drawn based on an incomplete set of process indicators.

The subgroup analyses indicated that those who have read some or all of the HCP Q&A brochure may have somewhat increased knowledge of the risks, compared with those who did not receive or read it. However, as the sample size of subgroups was small, caution must be applied in the interpretation of these data. This finding was not replicated when looking at the impact of reading of the brochure on risk minimization practices. In addition, no strong relationship between voriconazole prescribing frequency and HCPs’ knowledge was found, but again, the small subgroup sample size warrants cautious interpretation.

4.1 Potential Reasons Behind Low Utilization of aRMM Tools

A large proportion of survey respondents (80.4%) either did not receive or did not recall receiving the HCP Q&A Brochure.

The low recollection of receipt of the aRMM tools (HCP Q&A Brochure: 19.6%; HCP Checklist: 22.6%; Patient Alert Card: 25.9%) may have several explanations. It may be that, although delivered, the plain white envelopes were not given to the intended recipient due to administrative staff filtering or HCPs working at multiple clinics. Alternatively, the timeframe for conducting the survey (i.e. within the recommended 12–18 months after implementation of aRMMs [2]) may be too long an interval for HCPs to recall receipt of the tools. Both of these issues would be areas for future research to further inform best aRMM evaluation practice.

Of those who indicated receiving the brochure, 33.8% reported reading the entire document and 53.8% reported only reading some parts. Thus, the survey results point to a selective utilization of the aRMM tools, which could in part explain the lower knowledge of the risk of SCC compared with the risks of hepatotoxicity and phototoxicity among survey responders. One of the potential reasons behind low utilization of aRMM tools include HCPs’ own perception of familiarity with the risks. Almost half of the responding prescribers had been practicing medicine for 15 or more years (48.8%) and an additional 38.3% had been practicing for between 6–15 years. Among prescribers who treated the highest numbers of patients with voriconazole in the past 12 months (i.e. > 20 patients), slightly higher proportions reported receipt and utilization of the aRMM tools compared with prescribers who treated ≤ 20 patients with voriconazole. This observation is consistent with the finding reported by Landsberg and colleagues [38]. We note that prescriber’s general medical experience may affect aRMM awareness and utilization differently than specific experience of treating patients with voriconazole. Given the extent of the respondents’ experience in medicine and in managing patients who may require voriconazole, and that voriconazole was originally approved in 2002 in the EU, perhaps many believed they were already well aware of the risks and how to manage them. If, after a cursory review of the voriconazole HCP Q&A Brochure, HCPs assess that they are familiar with some or all of the risks that are being presented, they may not go on to read the full content and subsequently may discard the material.

Another potential reason for only partial review of the aRMM tools is that the risks presented may be considered low priority in relation to treating patients with complex underlying conditions. Patients receiving voriconazole typically experience life-threatening underlying conditions such as allogeneic HSCT, solid organ transplant, or inherited or acquired immunodeficiencies, who have also developed or are at risk for life-threatening fungal infections (e.g. invasive aspergillosis) [10, 39]. Physicians managing these critically ill patients are primarily focused on saving their lives by treating or preventing fungal infections. Within this context, the risks presented in the educational materials may be considered either already routinely monitored in the context of the underlying disease or secondary with respect to the management of the acute, life-threatening illness. Thus, reading the aRMM tools may not be a high priority for these HCPs. A similar theme was discussed in a National Comprehensive Cancer Network (NCCN) white paper, which described how oncologists, another highly specialized and trained group of physicians, perceived risk evaluation and mitigation strategies (REMS) in the USA as an unnecessary imposition on their daily workflow, which entails managing patients with serious conditions and prescribing drugs that have serious toxicities [40].

Additionally, given that a pharmaceutical company is the source of the aRMM tools, physicians may possibly view it as promotional in nature even though it was unbranded (i.e. there was no Pfizer logo on it). Some physicians may consider educational material from a pharmaceutical company as inconsequential or promotional by default, and thus may deprioritize or discard it. This explanation is supported by results from Topic 2 of the Strengthening Collaboration for Operating Pharmacovigilance in Europe (SCOPE), which indicated that of 3625 HCPs surveyed across the EU, the level of familiarity with educational materials, although still quite high (~ 66%), was considerably less than familiarity with Direct Healthcare Professional Communication (~ 90%) or National Competent Authorities (NCAs) communications (90%) [41]. The study also found that the most positively perceived senders of safety information were NCAs. This was followed by professional bodies and the EMA. Pharmaceutical companies and the press were negatively perceived when sending safety information. Electronic communications were reported by HCPs as the preferred method of communication. This finding also helps explain the low recall and utilization of the printed voriconazole aRMM tools.

4.2 What do the Survey Results Tell Us About aRMM Effectiveness?

Among the HCPs who responded to the survey, knowledge of the risks of phototoxicity and hepatotoxicity, and the associated risk minimization practices, were high, despite the low reported receipt of the aRMM tools; however, knowledge of the risk of SCC of the skin was low.

Phototoxicity and SCC of the skin are related risks and clinically linked by the role of ultraviolet exposure in both and use of the same preventive measures to minimize the risk of the two conditions. Hence, in the aRMM tools, risks of phototoxicity and SCC of the skin were presented together in the same section under one sub-heading. However, HCPs’ awareness of the risk of phototoxicity with voriconazole use was higher than for SCC of the skin. A potential reason for this lower level of knowledge about the risk of SCC of the skin may be the rarity and long latency of this risk compared with phototoxicity [42]. Another explanation could be, in part, due to a cursory review of the aRMM tools. Lastly, it is also possible that relatively lower levels of knowledge about the risk of SCC could be in part due to the fact that the risks of phototoxicity and hepatotoxicity have been documented in the SmPC for a longer period (since initial approval in 2002) than those for SCC (added in 2010).

The source of HCPs’ knowledge of the risks and associated mitigation practices for voriconazole is not clearly understood. However, the low reported rate of receipt of the aRMM tools and the relatively high knowledge of phototoxicity and hepatotoxicity compared with SCC of the skin indirectly suggests that the SmPC and other resources [e.g. the media, scientific conferences, continuing medical education (CME)] may be the sources of knowledge for safety information for these specialized HCPs. Similar to that highlighted by the US FDA in their 2013 commentary on patient education through medication guides [43], we note that it is challenging to isolate and quantify the precise impact of a specific communication medium via a knowledge survey, and especially one without baseline data. Thus, we acknowledge that the actual source of knowledge is unknown and probably multi-factorial. One action taken in agreement with the EMA to enhance HCPs’ awareness of the risk of SCC of the skin was to update the SmPC (i.e. Section 4.4 ‘Special Warnings and Precautions for Use’ was revised by reordering the adverse events to provide SCC of the skin with greater prominence) [6]. Also supporting the theory that sources of medical knowledge are multi-factorial, a study evaluating the knowledge of physicians who were sent educational materials on the thromboembolism risk of cyproterone acetate/ethinylestradiol, concluded that knowledge of the risk did not vary by receipt of the materials [25].

Of note, risk minimization measures have previously been shown to be effective. They were introduced to reduce the off-label use of tigecycline, and a retrospective study of their effectiveness concluded that this did decrease following implementation [44].

4.3 Lessons Learned

The findings from this study, in the context of the results from the SCOPE survey of HCPs, suggest that for future programs, optimization at the stage of aRMM program design and implementation may improve the uptake of aRMM tools. In retrospect, the design of the aRMM tools could have been more targeted if more extensive formative research with the key stakeholders (i.e. research with the target population to gain insights into their views of the aRMM tools planned, information presented, and incorporation of those tools into practice) had been executed prior to aRMM implementation. Therefore, for future aRMM programs, we would recommend first conducting a formative research study to investigate the optimal aRMM program design which would allow a more targeted approach to risk communication, as detailed by a relevant publication (2014) [45]. How the materials can be incorporated into existing clinical practice should be considered during this formative research, as this will greatly influence the type(s) of aRMM designed. Ideally, they should be compatible with existing patterns of care and workflows to facilitate adoption. Additionally, a multi-faceted approach to design (i.e. having multiple, integrated intervention elements delivered in unison or sequentially) could increase effectiveness [19, 45]. Considering the implementation of the aRMM program, the optimal method of distribution should be further investigated to maximize uptake, whether that involves changing the envelope in which it is posted or tailoring the number of copies of aRMM tools provided per envelope, based on stakeholder’s workflow or by using electronic modes of dissemination. Best practices recommend designing aRMM programs with dissemination in mind, to maximize external validity [45], and consider data-driven development, to avoid common pitfalls of aRMM programs [46]. In theory, these steps should take place prior to program implementation and sponsors should be encouraged to apply these approaches whenever feasible [45]. However, the timelines and urgency, which typically accompany decisions to implement an aRMM program, the need for regulatory endorsement from central health authorities, and country-specific approval requirements often make it extremely challenging to follow best practices from implementation science.

The experience with this study has also demonstrated that the addition of explanatory questions to the survey instrument (e.g. “What is your primary source of obtaining information about product safety?”, “Which aRMM tools do you find unhelpful?”, etc.) is an important element which would have helped to provide further useful context about the value of the aRMM tools in clinical practice.

We faced a number of operational challenges while implementing this evaluation survey. The most important ones stemmed from the varying applicable country-specific regulations and requirements with respect to local regulatory and ethics committee submissions. These need to be followed because of the classification of aRMM effectiveness evaluation studies as non-interventional PASS in accordance with GVP Module XVI [2]. A survey study design typically used in ‘market research’ is relatively straightforward to implement and usually does not require any additional regulatory notifications/submissions/approvals. As reviewed in a white paper by the International Society for Pharmacoepidemiology (ISPE) Special Interest Group (SIG): Benefit Risk Assessment, Communication and Evaluation (BRACE) [26], there are substantial implications of the effectiveness evaluation surveys being classified as PASS. In many EU countries, PASS are required to follow more stringent regulatory notification, submission, and approval procedures than non-PASS or market research surveys. These additional steps significantly increased the operational complexity of this PASS, delayed the time of survey launch, and reduced generalizability because at least one country was unable to be surveyed because the lengthy local ethics process prevented timely participation.

Offering financial incentives to reimburse participants for taking part has been found to increase response rates in similar surveys [47]. In our survey, the relatively greater response rate in Spain, the only country where payment was offered, corroborates this finding. However, our experience also showed this can be difficult to implement in practice, due to local regulations requiring contracts with HCPs and public disclosures stemming from the PASS classification and others which may render survey participation no longer anonymous. Further discussion of these issues can also be found in the ISPE white paper that highlights the challenges of conducting these evaluations in the EU [26].

A further operational challenge stemmed from the strict local privacy protection legislation, which resulted in difficulty in identifying a comprehensive list of HCPs who were mailed aRMM tools and thus a list to target with a survey invitation. We note that the new data protection regulations (GDPR) in the EU [27] will likely pose a further challenge in this regard.

Finally, the origin and mode of survey distribution is another factor in the success of aRMM evaluation. A recent EMA-sponsored survey of HCPs [41] demonstrated that the most preferred safety information channels were emails, point-of-care alerts (i.e. alerts through prescription systems) and, irrespective of electronic or paper-based format, reference books and national clinical guidelines. Thus, it is conceivable that those same preferences (i.e. via email or point-of-care alerts) could also be the most effective modes of inviting an HCP to a survey. The challenge faced by this study was that email lists were incomplete or absent for the purpose of research invitations, due to local privacy regulations, and therefore postal mail invitations were the best option to ensure representativeness. Furthermore, we hypothesize that many invitations were likely discarded because they were mailed on behalf of a pharmaceutical company (identified as the least-trusted source of risk communication [41]) and assumed instead to be promotional materials. Additionally, given HCPs’ preference for electronic communication, as well as our experience that a great majority of HCPs take surveys online, even if a phone survey was offered, online surveys are probably most effective. However, offering multiple methods of taking the survey (i.e. online and over the phone) might have increased participation.

5 Conclusion

Although the HCP survey response rate was low (1.7%), the results indicate that the HCPs who responded to the aRMM effectiveness survey were generally well informed about phototoxicity and hepatotoxicity as risks associated with voriconazole treatment and preventive measures. However, the level of awareness of the risk of SCC of the skin was lower than the prespecified threshold of 80%. The level of educational value of the voriconazole aRMM tools cannot be meaningfully inferred from the results, which indirectly point to the SmPC or other resources being the main source of risk information for these HCPs. The results showed that small proportions of surveyed HCPs recalled receiving the printed aRMM tools and reported reading the entire HCP Q&A Brochure. One of the key lessons learned from this study is that engaging HCPs before designing and implementing an aRMM program could ensure a useful and focused aRMM program. Given the challenges experienced with survey implementation, which compromised the ability to make meaningful scientific conclusions, future programs would benefit from a combination of: (1) early and thoughtful formative research and pilot testing; (2) minor EU regulatory reforms such as those outlined in the ISPE SIG BRACE white paper that could lessen some of the existing operational barriers; and (3) adding explanatory questions to gain further context for explaining the study results.

Finally, it is evident that further research and alternative methods are needed to overcome the limitations of survey research to help risk minimization evaluations evolve.