FormalPara Key Points

If the intent of measuring knowledge is to ensure safe use of medicinal products, the ability to inform regulators about prescribers’ knowledge of safe use within days or weeks rather than years is invaluable.

Knowledge assessments that target physicians through online platforms where they already seek information about drug-related safety risks may improve upon the <10% response rates that are typical for online surveys with traditional recruitment approaches, in addition to providing quicker access to results.

Online communities or professional societies may provide important venues in which to implement knowledge-acquisition surveys tied to training/education modules that address safety topics, including for pharmacovigilance.

1 Introduction

Ensuring that prescribers understand the safety risks of therapeutic agents and risk management strategies for potential issues is vital to public health. Accordingly, the United States Food and Drug Administration and the European Medicines Agency may request that sponsors demonstrate prescribers’ knowledge of safety risks described in the package insert or summary of product characteristics (SmPC) for marketed therapies. These knowledge assessments provide critical information to marketing authorisation holders and regulators, especially in an environment where it is challenging for physicians to remain current on the risks and benefits of the medications they prescribe as the demands of managing their clinical responsibilities increase. The utility of traditional pharmacovigilance knowledge surveys may be bounded in certain ways as a result of low response rates, selection bias, and limited generalisability and validity [1,2,3]. New assessment approaches may address some limitations by offering pragmatic, efficient, and cost-effective solutions that align with physicians’ increasing use of the Internet to access drug safety data.

Amgen Study 20110102, a traditional post-authorisation safety study (PASS), was designed to evaluate oncology practitioners’ knowledge of denosumab’s (120 mg administered every 4 weeks, XGEVA® [4]) SmPC pertaining to osteonecrosis of the jaw (ONJ). Osteonecrosis of the jaw is an adverse effect observed in some patients with advanced cancer treated with anti-resorptive therapy such as bisphosphonates or denosumab; ONJ was positively adjudicated in 89 (1.6%) of 5723 enrolled subjects across three large clinical trials: 37 (1.3%) of 2861 subjects who received zoledronic acid and 52 (1.8%) 2862 subjects who received denosumab (median follow-up months: 12.1, 12.6 respectively) [5].

Pharmacovigilance legislation introduced in Europe in 2011 provided limited guidance with respect to survey design and implementation but emphasised the importance of representativeness and response rates [6, 7]. Against this backdrop, pragmatic knowledge assessment solutions designed to balance these two considerations were tested. Traditional surveys use random selection from the ‘universe’ of eligible physicians (e.g., society membership), and a costly, time-consuming, contact and screening process, which may be inconvenient for clinicians, and ultimately yield low completion rates. To evaluate alternatives, we conducted three studies that leveraged web-based contacting, screening and knowledge delivery [including e-learning associated with continuing medical education (CME) credits] to assess pharmacovigilance knowledge. The first study was essentially a traditional pharmacovigilance survey with a more targeted contact and screening process to enhance response rates. The other two studies were conducted within an existing online medical community, with one demonstrating the implementation of a traditional online knowledge assessment in this venue, and the other examining the feasibility of implementing a targeted knowledge acquisition module in this online community.

2 Methods

In the first two studies, two different techniques in selected Nordic countries and the five largest European countries were used to conduct the surveys. Both used six multiple-choice questions on labelled ONJ-related information. The first study (PASS 20110102) survey was administered primarily online 12–18 months (1 January, 2013 to 12 June, 2014) and 24–30 months (28 August, 2013 to 15 May, 2015) after commercial availability following national reimbursement ruling of denosumab (XGEVA®) in each country. The dates of commercial availability and study 20110102 milestones are provided in Table 1. The second study’s survey was administered through Medscape, an online informational resource popular with healthcare professionals. In the third study, with Medscape Education, we developed a CME initiative for US-based Medscape users to assess the impact of an online educational programme on knowledge gaps by comparing respondent’s answers to four questions administered pre- and post-education. The studies used the same inclusion criteria: respondents were (1) active oncology practitioners; (2) treated five or more patients with bone metastases from solid tumours in the previous 3 months; and (3) prescribed denosumab within the previous 12 months.

Table 1 Commercial availability by country and corresponding milestones for Study 20110102
Full size table

2.1 Study 20110102

The 20110102 questionnaire was developed using expert opinion, in-depth qualitative interviews conducted following physicians’ completion of the online survey instrument and pilot tested with oncology practitioners. The questionnaire was translated from English to the native language of each country and tested with two physicians in each country. The final survey [Electronic Supplementary Material (ESM) 1] could be completed in 10–15 min, and covered topics consistent with the denosumab (XGEVA®) SmPC [4] pertaining to ONJ:

  • ONJ reported among patients treated with denosumab.

  • A dental examination with appropriate preventive dentistry should be considered prior to treatment with denosumab in patients with active dental and jaw conditions.

  • Patients should avoid invasive dental procedures, if possible, while receiving denosumab treatment.

  • Good oral hygiene practices should be maintained during treatment with denosumab.

  • Patients who are suspected of having or who develop ONJ while on denosumab therapy should receive care by a dentist or oral surgeon.

  • Extensive dental surgery to treat ONJ may exacerbate the condition.

Physicians were surveyed from France, Germany, Italy, Spain and the UK (the ‘European’ region), and from Denmark, Finland, Norway and Sweden (the ‘Nordic’ region).

This study used specific approaches to achieve ‘representativeness’ and ‘maximised’ response rates, as noted in the 2011 European pharmacovigilance legislation. We randomly selected oncology practitioners from country-specific master lists that combined Amgen’s country affiliate lists and the partner contract research organisation (Adelphi) lists of potential denosumab prescribers. In each survey round, oncology practitioners were randomly selected from each country; sampling continued until 150 eligible European practitioners and 60 Nordic practitioners agreed to participate. Sample sizes allowed stratification by survey round as well as by European and Nordic region based on the calculation of half-widths of 95% confidence intervals (CIs) around a range of percentages of correct responses ranging from 50 to 90%. There were two purposes for this regional stratification: to allow an adequate sample for regulatory authorities to interpret the results for oncology practitioners prescribing denosumab in the five largest European countries in the population in terms of the width of the 95% CI; and to obtain an adequate sample for the Nordic countries in terms of the CI. Round 1 participants were excluded from sampling for round 2. After confirming eligibility, up to six telephone or email attempts were made to invite the physicians to complete the online survey.

All respondents provided informed consent. All national laws that protect personal data and guidelines relevant to pharmacovigilance, physician research and/or market research practice were followed. Respondents were compensated for their time based on a fair market value in accordance with their countries’ local laws.

2.2 Medscape Survey

This study was conducted using the same or similar questions in 20110102 to confirm the feasibility of conducting a multi-round pharmacovigilance survey within an online community of physicians. Medscape (a WebMD company, New York, NY, USA), the leading online medical information resource serving approximately 3 million physicians worldwide, was selected for this study. Medscape-using physicians were recruited from France, Germany, Italy, Spain, the UK, Denmark and Sweden, consistent with study 20110102. Compensation was not provided by Medscape for involvement in this study.

2.3 Medscape Education

This study assessed the impact of an online educational programme on knowledge gaps using educational content and four survey questions related to denosumab and ONJ risk (ESM 3) developed by an independent expert and practicing oncologist in Europe. Eligible physicians were US-based users of Medscape.com or other Medscape-owned and operated sites, met all previously mentioned study inclusion criteria and received CME credit for completion. The short survey was administered before starting and then again directly after completion of the online educational module. Physicians who completed both pre- and post-assessment questions represented the linked learners for knowledge acquisition analysis.

2.4 Statistical Analysis

All descriptive statistics were reported as unweighted frequencies and percentages. Two-sided 95% CIs were estimated based on the normal approximation.

3 Results

3.1 Study 20110102

A total of 420 oncology practitioners (210 in each round) were surveyed (Table 2). Approximately 14.3% of screened physicians and 6.9% of initially identified practitioners completed the surveys. Respondents had seen a mean of 46.5 patients, new or continuing, with bone metastases from solid tumours in the prior 3 months.

Table 2 Recruitment of physicians for Study 20110102
Full size table

Per protocol, each round included 150 participants in the European region and 60 in the Nordic region (Table 3). Of the 300 total European region participants, most were from the UK (n = 74; 24.7%) and the fewest were from Italy (n = 37; 12.3%), where denosumab launched most recently (Table 1). Most of the Nordic participants were from Sweden (n = 60; 50.0%) and the fewest were from Norway (n = 12; 10.0%).

Table 3 Participating oncology practitioners by round, geographic region and country for Study 20110102
Full size table

There was some efficiency gained from round 1 to round 2 in most countries. This was evident by the fewer number of oncology practitioners that needed to be contacted, the smaller number who did not respond, and the higher number who were contacted, reached and ultimately responded to screening questions (Table 2). However, Spain, Norway and Denmark were more efficient in round 1; more attempts were required to obtain the target number of eligible oncology practitioners in round 2. The time required to recruit the sample was pre-specified as 6 months or less, and ranged from 2.5 (Denmark) to 26 (Sweden) weeks in survey round 1 and 3.5 (Italy) to 24 (Norway) weeks in round 2. The recruitment effort is summarised in Table 2, and the final sample is shown by country in Table 3.

In the combined results across both survey rounds and all respondents, the percentage of correct responses (95% CI) for each of the six questions was as follows: 76.4% (72.4–80.5) for Q1, 96.9% (95.2–98.6)—the highest correct answers selected—for Q2 (that a dental examination with appropriate preventive dentistry should be considered prior to treatment with denosumab in patients with active dental and jaw conditions), 56.7% (51.9–61.4) for Q3, 90.2% (87.4–93.1) for Q4, 81.7% (78.0–85.4) for Q5, and the lowest correct answers measured was 38.1% (33.5–42.7) for Q6 (that extensive dental surgery to treat ONJ may exacerbate the condition) (ESM 1). With Q3, an additional 30.2% (25.8–34.6) recalled that the SmPC stated, “Patients should avoid invasive dental procedures, if possible, during treatment with XGEVA® and for the following 12 months after discontinuation of XGEVA®.” Although the selection was not the exact language in the SmPC, this response was a similarly worded, albeit more conservative, answer clearly demonstrating safe use. Thus, a total of 86.9% of participants understood correctly that invasive dental procedures should be avoided, if possible, during denosumab treatment per SmPC. Correct answers by question and survey round are summarised by country in Table 4, showing the final combined Q3 correct and conservative selections.

Table 4 Percentage of correct responses by oncology practitioners for each question by country and round for Study 20110102
Full size table

Responses were reasonably consistent between the two survey rounds with some minor improvement evident for Q1 from 74.8% (68.9–80.6) in round 1 to 78.1% (72.5–83.7) in round 2—most notable in the Nordic region [75.0% (64.0–86.0) in round 1 to 86.7% (78.1–95.3) in round 2]—and Q4 from 89.5% (85.4–93.7) in round 1 to 91.0% (87.1–94.8) in round 2. In general, higher rates of correct responses were observed in the Nordic region compared with the European region (Fig. 1). Some decline was observed in the proportion of physicians selecting the correct answer for Q3 (minor decline also observed for Q5 and Q6 for European region) between the first and second rounds.

Fig. 1
figure 1

Study 20110102: percentage of correct responses by oncology practitioners for each question by round for the European (n = 150 per round) and Nordic (n = 60 per round) regions

Full size image

3.2 Medscape Survey

Five of the six questions administered in a one-time survey of qualified participants in the online Medscape community were the same as in Study 20110102, although answers were re-sequenced for Q4 and Q5. Q6 was replaced with a new question based on independent expert review and recommendation (ESM 2). A total of 207 (32.1% of 645 eligible) participants from seven European countries completed the survey between 16 May, 2013 and 5 June, 2013. Respondents were from Denmark (n = 12), France (n = 32), Germany (n = 32), Italy (n = 36), Spain (n = 33), Sweden (n = 29) and the UK (n = 33), and, on average, had seen 67.9 patients with bone metastases from solid tumours in the prior 3 months. Results were similar to those for study 20110102, but the percent correct was slightly lower in this study (e.g. Q1: 71 vs. 75% in study 20110102). A lower level of correct responses may reflect the voluntary nature of participation in the online Medscape survey that did not include an incentive such as compensation, whereas respondents were compensated for their time on the survey 20110102. Interestingly, Q3 yielded low correct responses, including 40% of respondents who selected the incorrect conservative answer that mentioned a 12-month washout period required after discontinuation of denosumab consistent with Q3 in study 20110102. The correct response rate for Q6 was improved after changes were made to the question structure (Fig. 2).

Fig. 2
figure 2

Study 20110102 and Medscape Survey: percentage of correct responses by oncology practitioners in Europe, Study 20110102 round 1 (n = 210) and Medscape participants over 3 weeks 2013 (n = 207)

Full size image

3.3 Medscape Education

Oncologists who completed the educational module and pre-/post-assessment (n = 264) over a 1-month period in 2013 were based in USA (for CME accreditation). Improvement was measured for all questions after education (Fig. 3). At baseline, 74.2% of respondents answered three or more of the four questions correctly vs. 93.6% after reading the educational article online; an increase from a mean of 3.0 correct answers to 3.5. An additional 1605 physicians from European countries completed the learning activity; however, the pre-/post-responses of individual learners were not linkable to determine score changes because CME credits were not available to these respondents. Q4 yielded lower responses compared with the other questions overall, and may reflect format and wording that differed from the preceding questions, but may also reflect oncologists’ lack of familiarity with early signs and symptoms of ONJ (ESM 3).

Fig. 3
figure 3

Medscape Education: percentage of correct answers by question (relative improvement shown) and among total comparing pre- and post-educational programme in linked learning assessments (n = 264)

Full size image

4 Discussion

New pharmacovigilance legislation was introduced in Europe in 2011, which coincided with the design phase of PASS, 20110102. At that time, physician surveys were commonly requested of marketing authorisation holders as part of risk management plans intended to measure knowledge of safety risks among prescribers. Study 20110102 for denosumab (XGEVA®) is an example of this type of study and was designed in response to the European Medicines Agency’s concerns, and conducted to measure oncology practitioners’ awareness of the SmPC statements regarding ONJ risk. This pharmacovigilance study used a traditional cross-sectional survey administered primarily online 12–18 months and 24–30 months after commercial availability of denosumab. Results from this study confirmed that oncology practitioners prescribing denosumab in nine European countries had knowledge levels exceeding 75% correct on five of six questions asked. Questions with majority correct responses were those pertinent to the incidence of ONJ, concomitant risk factors and prevention of ONJ during denosumab treatment, importance of ensuring oral hygiene, and care for patients who have or develop ONJ. The sixth question had a minority of correct responses, a result suggesting that oncology practitioners were less aware of the optimal treatment for ONJ, which may reflect their lack of involvement with ONJ treatment.

The sampling strategy for Study 20110102 was designed specifically to address the key survey considerations outlined by the EMA, at the time, namely to assure reasonable external validity and to maximise response rates [6, 7]. Measures taken to avoid bias included random sampling of potential respondents from all identified oncology practitioners in each country to ensure the sample was representative of all oncology practitioners who treat patients regularly in each country. To that end, we assembled a focused master list of prescribers to minimise sampling bias, which also resulted in greater efficiency with contact attempts by confirming and combining contract research organisation physician contact information with Amgen’s country affiliate prescriber lists. The survey was conducted in two different rounds to assess consistency or changes over time. Requiring non-overlapping participants for each survey round also enhanced the study by supporting the assessment of response consistency over time.

With use of both email and telephone in contact attempts and the low question burden (six questions to be answered in 10–15 min), over a 4-year period, the overall survey completion rate for Study 20110102 was within the expected range for this type of survey (6.9% including unsuccessful contact attempts). Of the 6079 practitioners identified for contact, 2927 were contacted successfully (3152 did not respond), 854 answered screening questions, 512 met study eligibility criteria, and ultimately 420 physicians consented to and participated in the survey. The survey completion rate was 14.3% among those who were successfully contacted, or 49.2% among those who were contacted and screened. Respondents from the Nordic region had higher levels of knowledge of ONJ risks compared with respondents from the other European countries, which may, in part, be attributed to an ongoing long-term pharmacovigilance study examining ONJ incidence in the Nordic region [8].

The survey using the online Medscape physician community achieved impressive response rates at first glance when compared with the 20110102 pharmacovigilance study (32.1% among 645 potentially eligible survey starts online); however, these different recruitment approaches do not allow for valid direct comparisons of response rates. Regardless, a key message is how the implementation of this study was much less onerous, yet the results were comparable for the five questions that also appeared in the survey fielded in 20110102. Furthermore, this approach can easily accommodate multiple survey rounds as needed, simply by keeping the online survey ‘open’ in Medscape or other online physician community over an extended period. In this way, one could monitor changes in the proportion of correct responses over time. Although future work is needed to measure the representativeness of the Medscape-accessible sample relative to the intended target population, this survey, as intended, provided a compelling proof of concept for using an existing online physician community to address pharmacovigilance needs. This Medscape study demonstrates a pragmatic 21st century survey approach, which offers the ability to generate informative results in a timely and more cost-effective manner than approaches traditionally used for pharmacovigilance knowledge assessments.

Our results highlight the need for broader consideration of the importance of timeliness and efficiency in understanding prescriber knowledge of drug safety. If the intent of such knowledge assessments is to ensure the safe use of medicinal products, a priority should be placed on efficiency and timeliness, and the ability to measure and report to regulators on the prevailing understanding of safe use within days rather than years after study start is invaluable. To illustrate, our traditional survey was initiated in August 2012 with an interim report for round 1 surveys available in October 2014 (because of the staggered commercial availability of denosumab in each country in Europe), and the final report was completed in September 2015. This timeline contrasts sharply with that for the Medscape community survey, which provided results within 3 weeks after the survey was launched online. In addition to offering quicker access to results, knowledge assessments designed to target physicians where they seek information about drug-related safety risks may also offer significant advantages in terms of better response rates and possibly enhanced generalisability. However, research targeting physicians online may in turn introduce self-selection bias with respect to a population of information seekers who are motivated to understand drug safety risk. Although Medscape is the leading online physician resource for news and education worldwide, with respect to generalisability, it does represent more physicians practicing in the United States than in other regions (i.e. approximately 500,000 physicians in the US are registered with Medscape, whereas 400,000 are practitioners in Europe). Nonetheless, physicians’ increasing use of online resources and related online professional communities can be leveraged, particularly from a pragmatic survey design, implementation and reporting perspective that can achieve external validity and maximise response rates.

Pharmacovigilance surveys may also seek to measure knowledge acquisition or change in knowledge, rather than simply measuring knowledge levels, per se. Surveys designed to be part of a risk minimisation plan may, therefore, be administered before (as a baseline) and after a learning activity is completed by participants. Our third study, the Medscape Education study with US-based practitioners registered with Medscape, illustrates this approach. In this study, we demonstrated an improvement in practitioner knowledge; from an average of 3.0 to 3.5 correctly answered questions provided by 264 physicians who completed assessments before and after completing an e-learning module with an associated CME credit. Although further work is needed and planned to assess e-learning-related knowledge acquisition in European oncology practitioners and to measure knowledge retention over time following completion of an e-learning module, physician or professional online communities offer the opportunity to couple knowledge surveys with standardised educational modules across a population of information seekers. In addition to the written materials used in this initial study, the planned/ongoing study in Europe will also include visual and video content, which has been shown to increase the effectiveness of healthcare professional educational programmes [9].

We acknowledge that differences in the recruitment approaches may have contributed to differences in the respondent populations. For example, Medscape had greater challenges recruiting enough physicians from Scandinavia; those recruited online reported seeing more patients with bone metastases but may have had less experience using denosumab. By contrast, 20110102 offered honoraria compliant with country guidelines to participating physicians, and this incentive may have contributed to larger participation as observed in Sweden. A limitation of our studies was access (i.e. limited study-related consent) to detailed demographic characteristics of the oncology practitioner respondents. Future work will address external validity by collecting more demographic information on potentially eligible survey respondents, and will aim to reach non-information seekers with programme launch announcements distributed to European oncology professional society membership listings.

Regardless of methodology, each with their own strengths and limitations, all three studies reported here documented high levels of awareness of the ONJ risks associated with denosumab among the eligible prescribing oncology practitioners who participated. During the time when Study 20110102 was being executed, additional risk minimisation measures were implemented in Europe to inform prescribers of the ONJ safety risk, including the Committee for Medicinal Products for Human Use’s strengthening of the denosumab SmPC warning language for ONJ on 1 September, 2014 [10]. This change had a minor impact on the wording featured in Q2 of the surveys used in the first two studies, but the percentages of correct responses before and after September 2014 were similarly high. In addition, an accompanying Direct Healthcare Professional Communication was distributed in the European Union to inform prescribers of the changes to the SmPC, followed by distribution of an ONJ-related patient information leaflet. These additional risk minimisation measures may strengthen the awareness of the ONJ risk among the prescribing oncology practitioners beyond the levels documented in our studies.

Current technology has increased the opportunity to evolve approaches to survey the implementation and design of knowledge assessments conducted as part of pharmacovigilance and risk management plans. Incorporating methodologies that leverage commonly used digital technologies will improve both knowledge acquisition and sponsors’ and regulatory agencies’ understanding of prevailing knowledge levels among prescribers. Research shows that physicians are increasingly using the Internet to educate themselves on clinical care issues including safety data, [11, 12] and physician online communities offer a pragmatic venue in which to conduct knowledge surveys particularly in combination with education and CME opportunities, where applicable. Guidance around the use of physician surveys as pharmacovigilance tools has improved considerably since 2011, including mention of the role of online approaches, and the use of professional and learned societies in the recruitment of healthcare professionals [6]. Nonetheless, considerations around the underlying need for a knowledge measurement, pragmatic design and interpretation of findings require further development with online opportunities. Ultimately, the findings from our work presented here for a traditional PASS and two proof-of-concepts using the Medscape online community of physicians for focused survey and educational intervention offer recommendations for future research endeavours in both pharmacovigilance and medical education.

5 Conclusions

Assessments that target physicians through online platforms where they seek information about drug-related safety risks may result in increased efficiencies, informing regulators about prescribers’ knowledge of safe use within weeks rather than years. Online communities or professional societies may provide venues in which to implement knowledge-acquisition surveys tied to training/education modules that address safety topics. However, care will need to be taken to ensure online surveyed communities are adequately representative.