FormalPara Key Points

The effectiveness of the atezolizumab additional risk minimization educational materials among physicians in six European countries was evaluated in a survey.

Receipt and usage of the Guide for Healthcare Professionals (HCP Guide) and patient alert card and knowledge and behavior of the key messages were moderately good.

The results support the safe use of atezolizumab in patients with urological and lung cancer.

However, the HCP Guide may not be necessary in the specialist hospital setting in which atezolizumab is administered as there was no difference in knowledge or behavior between those who received and did not receive the materials.

The study findings contributed to the European Medicines Agency permitting removal of the HCP Guide.

1 Introduction

Atezolizumab is an Fc-engineered, humanized, monoclonal antibody targeting human programmed death-ligand 1 on tumor-infiltrating immune cells and tumor cells. Atezolizumab was approved in September 2017 by the European Medicines Agency (EMA) for the treatment of urothelial cancer and non-small cell lung cancer. The following important immune-related adverse drug reactions (irADRs) were associated with treatment with atezolizumab: immune-related pneumonitis, hepatitis, colitis, hypothyroidism, hyperthyroidism, adrenal insufficiency, hypophysitis, type 1 diabetes mellitus, neuropathies, meningoencephalitis, pancreatitis, myocarditis, nephritis, and infusion-related reactions. The EMA’s Pharmacovigilance Risk Assessment Committee (PRAC) requested additional risk minimization measures in the form of educational materials to increase awareness and detection and to improve the management of these irADRs by physicians who prescribe and manage patients taking atezolizumab.

Roche, the Marketing Authorization Holder (MAH), developed a Guide for Healthcare Professionals (HCP Guide) and patient alert card (PAC), which aim to minimize the potential severe consequences of these adverse reactions by increasing physician awareness to facilitate early detection and prompt treatment. Table 1 shows the HCP Guide messages to physicians on how to recognize, monitor, and manage side effects related to atezolizumab, including the need to report irADRs. The PAC is to be provided by physicians to patients and details information about the risks of atezolizumab and instructions on how to use the PAC. The EMA requested the MAH to carry out a Post-Authorization Safety Study to evaluate the effectiveness of the atezolizumab educational materials using process indicators with survey methodology [1].

Table 1 Atezolizumab additional risk minimization materials
Full size table

The objectives of the study were to evaluate the self-reported receipt, understanding, use of the materials, and knowledge and behavior of key messages related to irADRs contained in the atezolizumab HCP Guide and PAC. Additional objectives were to assess knowledge and behavior according to reported receipt of the atezolizumab educational materials.

2 Methods

2.1 Study Design

A multi-country, one-wave, observational, cross-sectional, self-reported physician survey was conducted to assess the receipt, understanding, use of the materials, and knowledge and behavior in a sample of physicians with experience in the use of atezolizumab. The study, classified as a Post-Authorization Safety Study [2], was conducted according to Good Vigilance Practice Modules XVI [1] and VIII [3] and best practices based on guidelines [4,5,6] and publications [7,8,9,10,11,12,13,14,15]. The PRAC approved the study protocol. As no patients or patient-level information was involved, ethics approval from national or local ethics committees was not required. However, countries such as Germany, Denmark, and Italy required national notification to their health agencies.

2.2 Study Population and Sampling

The survey was conducted between April and August 2019 in a selection of European countries: Denmark, Germany, Italy, Spain, Sweden, and the UK. These countries were selected based on the predicted market uptake by Roche to ensure that the study was conducted where atezolizumab was available and accessible to patients and physicians. Such selection also attempted to support the external validity of the study findings, by including a wide and diverse range of healthcare systems (single-payer, public and private payers, mixed) and countries, including northern and southern Europe.

The target population included physicians who may initiate or manage patients on atezolizumab, specifically, medical oncologists, pulmonologists, and urologists. Only physician groups for whom the materials were targeted in the study period were eligible. The start of data collection in each country occurred within the 9–18 months period from the launch of atezolizumab in each country to allow physicians time for the uptake of atezolizumab, familiarity and use of the materials, as well as to ensure that the start of data collection did not vary notably amongst countries.

2.3 Sample Size

The study aimed to recruit 300 physicians to allow precision of ± 5.8% to ± 3.6% around plausible estimates (range 50–90%, respectively) of correct responses for process indicators related to the educational materials. All completed responses received by the cut-off date were included in the analysis. The ideal approach for sampling would have been a proportional split of the 300 physicians according to usage of atezolizumab in each country. However, because of the large variance in the number of physicians using atezolizumab in the targeted countries, such an approach would have resulted in very small numbers of participants in the smaller countries. Thus, the approach to sampling consisted of an approximate proportional split of the 300 physicians according to the population in each country, with a minimum of ten physicians for the smaller countries of Denmark and Sweden. Data collection stopped for each country once the country-specific target was met.

2.4 Physician Recruitment

The study was conducted by OXON Epidemiology, a scientific services company on behalf of the MAH. Physicians were selected from physician panels of oncologists, pulmonologists, and urologists in each of the participating countries. Panels were formed by voluntary participation of physicians in various ways in the different countries for market research purposes. A stratified sampling method was applied with pre-defined quotas based on the population size of participating countries and the number of materials in each country by specialty distributed by the MAH. Consecutive random samples of physicians in the panel strata were invited by e-mail to participate in the survey and asked to complete the on-line questionnaire until the quota for each stratum was reached.

Eligible physicians who agreed to participate in the survey received a link to access the e-questionnaire and were provided information on how to access the survey online. Financial compensation to physicians was based on a fair market value assessment (i.e., time and effort) in each participating country.

2.5 Questionnaire Design and Data Collection

The 20-item questionnaire was developed using best practices identified in Good Vigilance Practice Module XVI. The questionnaire was developed in English and conceptually reviewed using cognitive interviews by two UK clinicians, followed by translation and back‐translation into the languages of the participating countries by certified translators. The questionnaire was subsequently culturally adapted and reviewed by 18 oncologists, three in each of the participating countries, to ensure the clinical terminology was appropriate for each country. Qualitative techniques were used to ensure that the healthcare professional questionnaires were readable, understandable, and easy to use.

The self-reported survey questionnaire consisted of multiple-choice questions with no free text fields. The questionnaire structure included screening questions about eligibility, consent to participate, sociodemographic characteristics, specialty of the physician (oncologist, pulmonologist, urologist, other), years of experience with oncology patients, as well as experience with atezolizumab. The main questionnaire domains assessed the receipt, understanding and use of the materials, knowledge, and behavior. Questions related to knowledge included awareness, identification, monitoring, and management of irADRs, and awareness of the need to report them to their national authorities. Questions related to self-reported behavior included identification, monitoring, and management of irADRs. Questions about knowledge and behavior were capable of being answered by all respondents, whereas receipt, understanding, and use of the materials were limited to those who received the educational materials.

Responses to the survey questionnaires were collected through an on-line Electronic Data Capture system (proprietary, validated, ‘21 Case Report Form 11’ complaint, OXON software). The on-line questionnaire was self-administered and completed at the participants’ convenience. Although participants were advised to complete the questionnaire in a timely manner, once they started the questionnaire, they were able to stop at any point and, later, pick up where they left off, if necessary. Participants were not able to go back and change answers to previous questions. Participants were not allowed to access the questionnaire once it had been submitted. Multiple-choice questions included conditional branching based on responses to previous questions to improve user friendliness and reduce missing data. There was internal and external testing of the online questionnaire before release to study participants. All data collected during the survey were held confidentially. The identities of healthcare professionals taking part in the survey were pseudonymized using unique identification codes. These identification codes were held securely, and these data were used solely for the purpose of monitoring the progress of the completion of questionnaires.

2.6 Statistical Methods

Data analyses were descriptive using frequencies and percentages with 95% confidence intervals (CIs). Continuous scores were summarized using number, mean, standard deviation, median, quartiles, minimum, and maximum. No formal hypothesis testing was performed. Summary scores of usage, knowledge, and behavior were created to summarize all questions pertaining to usage, knowledge, and behavior. Summary scores were calculated as 100 times the sum of correct responses in the domain variables, divided by the number of variables in that domain.

A pre-specified threshold of 70 was used as the criterion to guide interpretation of results: ≥ 70 was deemed adequate performance and < 70 as inadequate performance. However, this criterion was not used for formal statistical testing. Results are presented overall, by country and HCP specialty. Knowledge and behavior were summarized overall and receipt of the educational materials was also calculated.

Missing, illegible, and “I don’t know”, “I’m not sure”, or “I don’t remember” responses were treated as missing values. For questions without a correct or best response, missing data were ignored in the analysis. As a sensitivity analysis, a conservative assumption considered missing values as incorrect or worst responses.

Responders and non-responders to the invitation to participate were compared by type of physician (oncologists, pulmonologists, and urologists) and country. All questionnaires received before database closure were considered responders. Participation rates were defined in two ways: number of responders divided by the number of invited, and the number of responders divided by the number of eligible (responders who fulfilled all the inclusion and none of the exclusion criteria) [4, 5]. All statistical analyses were performed using SAS® software, version 9.4 (SAS Institute Inc., Cary, NC, USA).

3 Results

3.1 Participation

Of the 6881 physicians (oncologists, pulmonologists, and urologists) invited in the physician panels of the participating countries, 760 HCPs were successfully contacted. A total of 378 physicians fulfilled the inclusion criteria and were eligible for the study. Among the eligible physicians, 371 accessed the electronic Case Report Form to answer the online self-administered questionnaire. The full analysis set included a total of 313 subjects who completed the online questionnaire, and for which there was at least one response for any of the domains related to the objectives (Fig. 1). Participation rates were 11.0% among responders/invited and 82.8% for responders/eligible. The distribution of the 313 physicians varied by country and specialty. Some physicians did not answer all the questions and the denominator is less than 313 for some of the results.

Fig. 1
figure 1

Flow chart picturing the study population and datasets within the full analysis set

Full size image

3.2 Participant Characteristics

The characteristics of the 313 physicians who completed the survey are depicted in Table 2. The percentage of physicians by country ranged from 3.5% in Denmark to 29.1% in Germany. Physician specialties comprised 81.6% oncologists, 9.6% pulmonologists, and 9.0% urologists. In Denmark, Spain, and the UK, all the physicians (100%) were oncologists, while in Italy and Sweden, pulmonologists also participated in the study at 14.8% and 7.7%, respectively. Germany was the only country with physicians from the three different specialties: oncologists (51.7%), pulmonologists (17.6%), and urologists (30.8%). Hospital-based physicians comprised 72.8% with 12.8% being both hospital and office based. There were 14.4% of physicians who were only office based and all were from Germany. Male physicians comprised 73.2% of the study sample and almost all physicians were aged 30–45 years (47.3%) and 46–65 years (50.8%).

Table 2 Physician’s sociodemographic and patient management characteristics
Full size table

Experience in managing cancer was 6–10 years in 16.9%, and more than 10 years in 78.3%. The majority of physicians (89.1%) had had contact with at least one patient taking atezolizumab in the 3 months before completing the questionnaire. Almost one half of the physicians (47.6%) had participated in a trial with atezolizumab; however, this varied by country. While a high percentage of physicians in Spain (68.1%) and the UK (69.8%) had participated in clinical trials, the majority of physicians in Germany (74.7%) and Sweden (84.6%) had not.

3.3 Receipt of Materials

The HCP Guide was received by 77.4% (240/310) of physicians while the PAC was received by 74.2% (230/310) of physicians (Fig. 2). Receipt of the HCP Guide was similar between countries, and above 70% among physicians with > 10 years of experience of managing patients with cancer, and physicians who had last seen a patient taking atezolizumab within 3 months of completing the questionnaire. Nonetheless, there were minor differences in the receipt of materials by specialty (Table 1 of the Electronic Supplementary Material [ESM]). While more than 70% of oncologists received both the HCP Guide and the PAC, the HCP Guide was received by 59.3% of urologists and 66.7% of pulmonologists, and the PAC by 51.9% of urologists and 63.3% of pulmonologists. Receipt of the PAC was above 70% in Denmark, Italy, Spain, and the UK, among oncologists, physicians with >10 years of experience of managing patients with cancer, and physicians who had last seen a patient taking atezolizumab within 3 months of completing the questionnaire (Fig. 3).

Fig. 2
figure 2

Percentage of physicians reporting having received the patient alert card (PAC), Guide for Healthcare Professionals, both the PAC and Guide for Healthcare Professionals, one of the materials, and none of the educational materials within the full analysis set

Full size image
Fig. 3
figure 3

Receipt of the patient alert card by subgroups of country, specialty, years of experience, and time since last contact. Values are shown in percentage of physicians (%) and the 70% prespecified threshold is indicated by the red vertical broken line

Full size image

3.4 Understanding of Materials

Figure 4 shows that 85.4% (146/171) of physicians who had read the HCP Guide reported having “completely understood” it while 9.4% (16/171) found it too difficult to understand. Results were similar across country and physician specialty (Table 1 of the ESM).

Fig. 4
figure 4

Percentage of physicians who read and understood the Guide for Healthcare Professionals

Full size image

3.5 Use of Educational Materials

Of the physicians who received any of the materials, 71.3% (171/267) had read the HCP Guide. Among these physicians, 31.6% (54/171) stated that they frequently referred to it (Fig. 5). Among the 230 physicians who provided the PAC, 54.8% gave it to all their patients and 30.4% to “most patients”. The mean usage score was 69.5 (95% CI 66.0–72.9), and 57.1% of physicians reached usage scores of ≥ 70. Mean usage score results were similar across country (Denmark [55.9], Germany [70.8], Italy [72.4], Spain [62.1], Sweden [69.7], UK [70.8]) and physician specialty (Table 1 of the ESM).

Fig. 5
figure 5

Use of educational materials. The figure depicts the percentage of physicians who referred to the Guide for Healthcare Professionals and implemented the advice in the Patient Alert Card (PAC). The overall usage score is shown in blue with median and interquartile range

Full size image

3.6 Knowledge

Across all questions in the knowledge domain, the mean overall knowledge score was 63.9 (95% CI 62.1–65.7). There were 39.4% of physicians with knowledge scores ≥ 70. Figure 6 shows that most of the point values were below the threshold score of 70 for all the questions. Mean overall knowledge score results were similar across countries (Denmark [74.5], Germany [60.7], Italy [62.6], Spain [65.0], Sweden [62.5], and UK [67.9]) and by physician specialty (Table 1 of the ESM).

Fig. 6
figure 6

Level of knowledge of key messages in the Guide for Healthcare Professionals. The figure depicts the percentage scores across the different questions. The overall knowledge score is shown with median and interquartile range. irADRs important immune-related adverse drug reactions, IRR infusion-related reaction

Full size image

Figure 7 shows that knowledge was not related to receipt of the educational materials, with knowledge scores of 66.8 (95% CI 64.9–68.7) for reported receipt of both the HCP Guide and PAC, 59.4 (95% CI 55.5–63.4) for one of the materials, and 60.8 (95% CI 55.4–66.2) for having received none of the materials. Knowledge scores above the 70 thresholds were also not related to receipt of the educational materials and were reached by 43.8% who received both the HCP Guide and PAC, 28.1% who received one of the materials, and 37.5% who received none of the materials.

Fig. 7
figure 7

Percentage knowledge scores for physicians who received and did not receive the Guide for Healthcare Professionals (HCP Guide) and patient alert card (PAC). The shaded bars indicate the median and upper and lower interquartile range with ranges. The point estimates represent the mean and 95% confidence intervals (CIs). The overall knowledge score by receipt of materials is shown in green

Full size image

3.7 Behavior

The mean overall behavior score was 78.9 (95% CI 76.8–81.0). Behavior on key concepts was ≥ 70 in 74.8% (231/309) of physicians. Using the experience of their last patient taking atezolizumab, 92.9% (289/311) counseled their last patient about the signs and symptoms of irADRs associated with atezolizumab and 94.5% (293/310) counseled their last patient to contact their treating doctor immediately if they experience symptoms of irADRs (Fig. 8).

Fig. 8
figure 8

Physician behavior. The figure depicts the percentage of physician behavior related to immune-related adverse drug reactions. The overall behavior score is shown in blue with median and interquartile range. IRR infusion-related reaction

Full size image

Knowledge that temporary interruption of atezolizumab should take place in patients who develop new isolated hypothyroidism, without an alternative cause, was reported by 87.1% (269/309), and 78.3% (242/309) correctly indicated permanent discontinuation of atezolizumab for a patient who develops meningoencephalitis without an infectious or other cause. However, a smaller percentage of physicians (40.7%) were aware that permanent discontinuation is needed if a patient develops dyspnea, hypoxia, and patchy infiltrates on a chest x-ray, consistent with immune-related pneumonitis. These results were similar across country and physician specialty (Table 1 of the ESM).

Behavior scores were similar regardless of whether materials were received. The mean behavior score was 79.0 (95% CI 76.5–81.5) for those who received both the HCP Guide and PAC, 76.9 (95% CI 72.2–81.5) for receipt of one of the materials, and 81.5 (95% CI 75.0–88.0) for those who received none of the materials. The proportion of physicians who had mean behavior scores ≥ 70 by receipt of materials was 74.9% (152/203) for both the HCP Guide and PAC, 73.4% (47/64) for one of the materials, and 75.0% (30/40) for having received none of the materials.

4 Discussion

The study employed a one-wave, observational, cross-sectional, self-completed survey design to evaluate the effectiveness of the atezolizumab additional risk minimization educational materials (HCP Guide and PAC) in physicians using process indicators. Recruitment of the target sample size was reached, permitting the overall degree of precision planned. Receipt of both the HCP Guide and PAC was moderately high. Receipt was particularly high among oncologists and physicians with > 10 years of experience of managing patients with cancer. This is not surprising as oncologists are likely to initiate and manage more patients with atezolizumab than pulmonologists and urologists.

Understanding of the materials was very high, but only a minority of physicians referred to the HCP Guide frequently. This may reflect the units in which these specialists manage patients with atezolizumab with access to other sources of materials, such as local protocols. However, the PAC was provided to most patients by most physicians.

Knowledge of irADRs was moderate, but a majority of physicians did not have knowledge scores ≥ 70. Knowledge was poorer for the following irADRs: nephritis, myocarditis, neuropathies, meningoencephalitis, and all the (false) risks in the questionnaire that are not considered related to atezolizumab: amnesia, insomnia, urethritis, and stomatitis. Surprisingly, there was no clear difference in knowledge scores between physicians who had and had not received one or both of the educational materials.

The poorer than expected knowledge scores for irADRs may be due to the inclusion in the questionnaire of a significant proportion of false risks, as these types of questions are known to score less well than questions with positive answers. Conversely, behavior in the management of irADRs was very good and a high percentage of physicians had behavior scores ≥ 70. Nevertheless, as with knowledge, there was no clear difference in behavior scores between physicians who had and had not received one or both of the educational materials. Access to resources outside of the current educational materials such as the European Union Summary of Product Characteristics, other literature, and prior experience with immunotherapy and use of local treatment protocols available in oncology departments may explain this lack of difference.

While knowledge was lower than desired, behavior fared much better, inferring that physicians were safely managing irADRs. There is no one ‘threshold’ that is used by the EMA to assign success or failure of additional risk minimization measures and the results as a whole and their implications for patient safety are evaluated. Thus, while a 70 threshold of 70 was used in this study, it is arbitrary and needs to be assessed with this perspective.

The lack of differences in knowledge and behavior between physicians who did and did not receive the materials may be due to the information being available in other sources, such as departmental protocols and the Summary of Product Characteristics, which are used more frequently even if the additional materials are read initially. The questions for both knowledge and behavior were not too insensitive as the scores did not demonstrate a ‘ceiling effect’ or ‘floor effect’. The range of countries with different healthcare systems and multiple sites provides a global overall picture of the performance of the atezolizumab HCP Guide and PAC among oncologists, pulmonologists, and urologists in Europe (although most of the data were from oncologists), and wide generalizability of the findings.

The study has some limitations. While the planned number of HCPs was recruited, it involved a significant recruitment effort requiring invitations to 6881 potential HCPs. How many of the non-responders were not eligible is unknown, but we do know that many of physicians who responded were not eligible because of a lack of experience with atezolizumab. The generalizability of these results may be limited to more experienced oncologists, as a large proportion had participated in clinical trials. While a wide range of countries was included to increase generalizability, smaller countries with less usage of atezolizumab necessarily had fewer physicians in the survey. Thus, the generalizability of the results has some uncertainty. Nonetheless, low response rates in surveys involving HCPs are a well-known limitation and has been previously acknowledged in other studies of this type [16,17,18,19]. Clinical or safety outcomes linked, at the participant level, with process indicators in surveys are the best way to assess the effectiveness of additional risk minimization measures [16]. Differences in results in those who receive and do not receive the educational materials are arguably the strongest evidence from one-wave surveys of the effectiveness of additional risk minimization measures. However, they may not be wholly unconfounded comparisons. Thus, the results need to be interpreted cautiously. Furthermore, it is possible that there may have been some information bias, particularly as behavior was self-reported by the participants. While this potential bias was minimized by including questions with clinical scenarios for managing irADRs and experience with the last patient taking atezolizumab they managed, it cannot be entirely excluded.

A more comprehensive evaluation of the effectiveness of the PAC would require an assessment in patients and this was not part of the study. The education and management of atezolizumab irADRs are also likely to be provided by nurses, as they administer the infusions every 3 weeks in most centers and not physicians, and this HCP group was not covered in the study. Potential response bias was minimized with ‘best practice’ qualitative techniques to develop the questions and their implementation in the online survey such as sequencing of questions, skipping questions not permitted, and questions could not be changed once submitted. There were few missing data and a sensitivity analysis treating missing values as incorrect or worst responses made little difference to the results.

As new indications are added to atezolizumab, there may be a need to consider evaluations in additional physician groups involved and an evaluation of the PAC in patients. The practical impact of this study resulted in no modifications to the content of the atezolizumab HCP Guide or PAC or further evaluations being requested by EMA regulators.

5 Conclusions

The study assessed the effectiveness of the atezolizumab additional risk minimization educational materials among physicians in six European countries, using process indicators. The educational materials reached over 70% of target physicians, 57.1% of whom reported using them. Despite demonstrating knowledge scores below the threshold, physicians reported scores for understanding and behavior that exceeded the threshold. Thus, it appears that the additional risk minimization measures may not impact knowledge nor change physician behavior related to irADRs for atezolizumab. The results support the safe use of atezolizumab by these physician groups and contributed to the decision of the EMA to permit the MAH remove the mandatory distribution of the Guide for Healthcare Professionals.