Advertisement

Drug Safety

, Volume 42, Issue 4, pp 477–489 | Cite as

Recommendations on the Use of Mobile Applications for the Collection and Communication of Pharmaceutical Product Safety Information: Lessons from IMI WEB-RADR

  • Carrie E. Pierce
  • Sieta T. de Vries
  • Stephanie Bodin-Parssinen
  • Linda Härmark
  • Phil Tregunno
  • David J. Lewis
  • Simon Maskell
  • Raphael Van Eemeren
  • Alicia Ptaszynska-Neophytou
  • Victoria Newbould
  • Nabarun Dasgupta
  • Antoni F. Z. Wisniewski
  • Sara Gama
  • Peter G. M. MolEmail author
Open Access
Special Article

Abstract

Over a period of 3 years, the European Union’s Innovative Medicines Initiative WEB-RADR (Recognising Adverse Drug Reactions; https://web-radr.eu/) project explored the value of two digital tools for pharmacovigilance (PV): mobile applications (apps) for reporting the adverse effects of drugs and social media data for its contribution to safety signalling. The ultimate intent of WEB-RADR was to provide policy, technical and ethical recommendations on how to develop and implement such digital tools to enhance patient safety. Recommendations relating to the use of mobile apps for PV are summarised in this paper. There is a presumption amongst at least some patients and healthcare professionals that information ought to be accessed and reported from any setting, including mobile apps. WEB-RADR has focused on the use of such technology for reporting suspected adverse drug reactions and for broadcasting safety information to its users, i.e. two-way risk communication. Three apps were developed and publicly launched within Europe as part of the WEB-RADR project and subsequently assessed by a range of stakeholders to determine their value as effective tools for improving patient safety; a fourth generic app was later piloted in two African countries. The recommendations from the development and evaluation of the European apps are presented here with supporting considerations, rationales and caveats as well as suggested areas for further research.

Key Points

A mobile application (app) designed for adverse drug reaction (ADR) reporting and product safety alerts can help to augment pharmacovigilance activities and extend a health authority’s reach to patients and healthcare professionals.

Of particular value to app users was the ability to learn about the safety profiles of medicines through user-friendly, interactive graphics within the app as well as privacy and data protection features.

While uptake and use of the app to date seems modest in comparison with other ADR-reporting modalities, it is reasonable to expect that app-based reporting will grow in importance as a younger generation of app-literate patients matures and smartphone owners increasingly use their mobile devices to access the Internet.

1 Introduction

The Innovative Medicines Initiative (IMI) WEB-RADR (Recognising Adverse Drug Reactions) consortium was a public–private partnership supported by the IMI Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115632. Participating members were from European regulatory agencies, the pharmaceutical industry, academia, patient groups and other organisations with an interest in pharmacovigilance (PV). The aim of WEB-RADR was to develop and evaluate two digital tools to support PV activities and promote patient safety, and ultimately provide policy, technical and ethical recommendations on implementing such technology. Full details of IMI WEB-RADR’s objectives and approaches have been detailed elsewhere [1]. The outputs from WEB-RADR arose from four work packages: two work packages undertook original research in social media and mobile application (app) technology; a third evaluated the scientific impact of the original research to determine where it had potential to add value to existing PV methodologies. A fourth work package addressed governance and policy, including personal data protection and ethical and societal considerations related to the use of mobile apps and social media for PV. This paper will focus on recommendations resulting from the work on apps. Recommendations resulting from the research conducted using social media will be the subject of a separate publication.

The profusion of smartphones and other mobile devices offers an opportunity to further engage the public in reporting suspected adverse drug reactions (ADRs) to national competent authorities (NCAs) and to improve communication of emerging safety issues from NCAs to the public and healthcare professionals (HCPs). Together with web-based forms, mobile devices offer a platform for developing real-time PV systems that can enable near-instantaneous transmission of patient safety information at the point of need, potentially improving health outcomes. Data collection in PV typically relies on voluntary reporting of suspected ADRs submitted to NCAs and pharmaceutical companies by HCPs. Over the last 20 years, patient reporting has been increasingly recognised as a valuable addition to safety monitoring activities [2, 3]. This growing interest in the patient perspective has coincided with a proliferation of technology, which in turn has increased access to data, accelerated information exchange, and fostered transparency. It is therefore reasonable to assume that efforts to improve ADR reporting and patient safety communication could benefit from the adoption of new technologies such as mobile apps.

Early experiences with mobile apps for medical product safety have been promising and have demonstrated potential for broad engagement of the public in safety reporting [4]. At the same time, concerns over security and potential for misuse of personal data have grown with several well-publicised breaches such as those at Yahoo, CEX and Facebook [5, 6, 7]. Aside from developing and evaluating digital tools themselves, the WEB-RADR project also examined the ethical, societal and privacy concerns related to mobile apps, as well as technological and promotional aspects, from a European perspective.

The WEB-RADR mobile apps were designed to facilitate instantaneous reporting of suspected ADRs directly to NCAs by patients and HCPs, as well as timely dissemination of accurate PV information from NCAs to HCPs and the public, thereby allowing potential two-way exchange of safety information. The apps have been made freely available by PV centres in the United Kingdom (UK; Medicines and Healthcare products Regulatory Agency [MHRA]), the Netherlands (NL; Lareb), and Croatia (HR; HALMED) in both Google Android and Apple iOS formats; links to download the apps are available at https://web-radr.eu/mobile-applications-for-adr-submission. A fourth version of the app has been piloted in Burkina Faso and Zambia in collaboration with the World Health Organization (WHO); at the time of writing, this fourth version was still undergoing validation and assessment.

The use of mobile apps for ADR reporting and the downstream use of these data for PV is a relatively new and unexplored concept. The anticipated impact of these apps will depend on the degree of uptake by the public and HCPs, the quality and volume of ADRs reported, and the utility of the data in safety signal detection and evaluation. In WEB-RADR, each NCA devised its own strategy for mobile app promotion best suited to its national context and providing a range of engagement models for assessment. Focus group discussions and face-to-face interviews were conducted with potential app users from multiple countries [8]. Enthusiasm for the app was high and the feedback received was constructive, targeted and generally encouraging. Comments from the participants have been used to inform further app development to improve uptake and encourage continued use of the app. As of 31 December 2017, there have been over 17,000 app downloads and 838 suspect ADR reports received since the first app was launched on 14 July 2015 (Table 1).
Table 1

Breakdown of app downloads and reported ADRs by country to 31 December 2017a

 

United Kingdom (MHRA)

Netherlands (Lareb)

Croatia (HALMED)

Date of app launch

14 July 2015

29 Jan 2016

18 May 2016

Number of ADR reports

505

173

160

App downloads for iOS

7498

4172

422

App downloads for Android

2701

2497

876

ADR adverse drug reaction, app application, MHRA Medicines and Healthcare products Regulatory Agency

aAs of Q4 2017, mobile operating system penetration within these three countries was as follows: UK: Android 46.7%, iOS 50.6%; the Netherlands: Android 56.0%, iOS 41.9%; and Croatia: Android 82.3%, iOS 15.0%. See http://gs.statcounter.com/os-market-share/mobile

To integrate ADR reports submitted via the apps into existing ADR processing workflows at NCAs, a direct connection was made between each app and the respective NCA’s existing electronic gateway. When a user submits an ADR report through the app, the reported data are automatically converted into standard International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use (ICH) E2B(R2) format, allowing it to be submitted directly to the NCA’s PV portal for triage, assessment and signal detection alongside ADR reports received through other reporting channels. The apps also provide a means for NCAs to broadcast to users personalised safety alerts and news about user-selected drugs. App users can also explore aggregate ADR report information in interactive graphs that display volumes of reports received by the NCA (at the national level in the UK and NL and at the global or EU level for HR) for medicines of interest according to Medical Dictionary for Regulatory Activities (MedDRA®; https://www.meddra.org) system organ class and, where available, by patient demographics such as age and sex. NCAs submitted suspected ADRs from the mobile apps to EudraVigilance by batching app reports with reports submitted through other channels, in accordance with usual reporting requirements [9].

The following recommendations1 relate to policy considerations and original research pertaining to the development and utility of a mobile app platform for PV conducted under the auspices of the IMI WEB-RADR project between September 2014 and September 2017. In total, we present 27 separate recommendations with the intention of informing PV professionals, particularly those with an interest in the development of digital technologies to enable PV.

2 Summary of the Research and Recommendations

The recommendations outlined in this paper have been developed based on evidence from the activities undertaken through IMI WEB-RADR. For a full description of the studies, refer to the cited original publication or technical report. For the sake of brevity, unless explicitly stated to the contrary, the terms ‘mobile app’ (or simply ‘app’) are used to cover software designed for mobile devices, irrespective of the means of presentation, i.e. via a smartphone or a tablet. Recommendations were drafted by the authors of the various outputs that collectively described the outcomes of the app work package. Draft recommendations were subjected to several rounds of review by other work package members, resulting in either their revision or rejection. Recommendations were typically rejected on the grounds that they were too general in nature or not adequately supported by insights arising from the work of WEB-RADR or if they were considered not implementable in practice. Final-draft recommendations were circulated to the WEB-RADR General Advisory Board2 for review and comment. Finally, all consortium participants were invited to review a mature draft of this paper.

2.1 Drivers for Mobile Application Development and Other Pre-development Considerations

Prior to committing the effort and investment needed to develop any mobile app, it is important to gain an understanding of the setting in which the app is to be used and the intended target group. Prior knowledge of technical and regulatory limitations and likely barriers to uptake and usage can inform app design and implementation. The recommendations provided in this section could be deemed good practice for app development in any setting; however, it is pertinent to share the specific experiences from WEB-RADR as a means of emphasising the value of undertaking a scoping exercise prior to full-scale app development. The value versus opportunity cost of app development has not been factored into these recommendations since it is likely to vary from country to country given differences such as availability of established means of ADR reporting, infrastructure maturity (internet, mobile networks, etc.), education and awareness of healthcare systems generally and PV more specifically.

Consideration of the intended use of the technology is also important. For example, the key benefits in the EU settings evaluated were related to easy access to trusted regulatory information and the impact in terms of reporting considered additive to the suite of tools available to reporters [8]. However, early feedback from the pilots in Burkina Faso and Zambia points to the value of off-line functionality (reporting and news) independent of access to a website.

Furthermore, there are efficiencies in the costs associated with standardised/shared platforms/code, with significantly reduced maintenance costs associated with the app version used in Burkina Faso and Zambia developed towards the end of the WEB-RADR project. Technology now enables the use of common code to deliver the same functionality or multiple modalities; this offers a more flexible and cost-effective framework to deliver on user needs independent of the end user device. See Table 2 for app development recommendations.
Table 2

Recommendations for application (app) development

Recommendation

Rationale

Prior to designing a mobile app, it is recommended that smartphone and device use among target populations is assessed as this can inform basic decisions regarding development

Decisions over which reporting modalities (e.g. app, website, etc.) and operating systems are supported should account for the prevalence of wired versus wireless connectivity and the devices in use (e.g. iOS vs Android vs PC) in the target setting. An evaluation should be conducted to ensure that the platform proposed is the most effective means of delivering the functionality to the target audience

For example, smartphone penetration and use of different types of devices vary greatly from country to country. Android devices are more commonly used in Europe, South America, and Asia, while iOS devices are more commonly used in Australia and in North America, use of both operating systems is broadly even [10]. Additionally, while two-thirds of the global population use the Internet or own a smartphone, this is less common among adults in Africa and South Asia [11]

Organisations deploying apps should be aware of the differing regulations in different regions and countries

The General Data Protection Regulation (GDPR) introduces EU-wide legislation on personal data and security and developers of apps need to be familiar with the rules and obligations. In particular, they should assess the impact of data protection at the time of design concept and review compliance periodically. In addition, an ePrivacy Regulation has been proposed [12]. It is also important that developers pay attention to any specific national requirements, and testing of application security is advised

Other regulations should be considered. Although the apps developed by WEB-RADR are not currently impacted by EU Medical Device regulation, it would be advisable to maintain awareness of device regulations, particularly if the apps were to be enhanced to an extent where medical device regulations may become relevant. In case of doubt, legal advice should be sought [13]

Any new app or subsequent updates to the app or operating platform should be extensively tested before release to ensure that technical issues are avoided and to assure the app design remains intuitive, does not inhibit usage and minimises user error

A survey indicated that users experienced difficulties downloading some versions of the app. New software requires testing and validation to ensure that it will perform as expected on release. A formal validation model requiring user specification and technical specification tested using a formal validation strategy with a final report should be employed

Consultation with target user groups should be considered when undertaking app development efforts for pharmacovigilance

Prior to app launch, WEB-RADR had access to patients who contributed important suggestions based on their experience testing the app prototype. Focus groups and face-to-face interviews were conducted in multiple countries where feedback was documented formally to later inform app design [8]. Consultation may be less important if targeting well characterised populations of users or similar geographic regions

2.2 Mobile Application Design

Traditional reporting mechanisms require significant time and effort to complete and submit, inhibiting ADR report submission to Marketing Authorisation Holders (MAHs) and regulators [14]. Mobile app design for PV offers the opportunity to improve upon traditional ADR-reporting forms for patients and HCPs alike. The app design process in WEB-RADR focused on developing an uncluttered visual design incorporating feedback from patient and HCP groups, while maintaining enough consistency to enable comparison with suspected ADRs obtained through other mechanisms. Since the reports submitted through the app would be passed along to NCAs, the app screens had to include the minimum necessary data fields to comply with ICH E2B(R2) standards and requirements in line with the guideline on good pharmacovigilance practices (GVP) Module VI, which served as one of the starting points for their design [15]. The four mandatory reporting elements were the focus of data collection: (1) patient information, (2) reporter information, (3) suspected medicinal product and (4) suspected adverse reaction.

The digital platform afforded opportunities to have adaptive forms. Since ADR reports contain information about medical products and ADRs, structured report fields may require knowledge of medical terminology; however, it is unlikely that all patients will be familiar with this type of vocabulary. In the UK, work was done as part of WEB-RADR to investigate the use of patient-friendly MedDRA® terms for reporting ADRs, and a list of patient-friendly terms is available on the MedDRA Maintenance and Support Services Organization (MSSO) website [16].

There are also several features that could be integrated into an app to facilitate processing and analysis by NCAs, including structured and hierarchical event ontologies. Autofill features, an adaptive suggestion that appears in a field automatically when a user types a few letters, for symptoms and product names may expedite the report submission process and ensure that these data points are spelled correctly and adhere to standardised terminologies.

While structured text fields such as drop-down menus or radio buttons promote data standardisation, these may not be ideal in every situation. For example, patients may not comprehend ontology-based medical terms suggested in a drop-down menu or they may find that the options do not best describe their experiences. From a cognitive design perspective, overly structured fields may negate useful nuance in the patient’s experience, reducing the usefulness of the report. Early testing suggested that patients may find that completing free-text fields to describe ADRs in their own words preferable to selecting options from drop-down menus with multiple choice and radio buttons [17]. Such design considerations can conceivably impact report completeness and accuracy.

It is important to ensure that a mobile PV app can be used by as wide a population as possible. Technology industry standards can provide guidance on design elements such as preferred font size for legibility, preferred colours or shades for text and image visibility and button placement for touchscreens. Adhering to such standards as much as possible ensures that a mobile app can be used by patients and HCPs with a variety of abilities. Additionally, when designing an app for ADR reporting, any constraints imposed by regulatory requirements, database or technology limitations and resource implications should be carefully evaluated.

Beyond the visual design of the app, the digital environment permits the creation of a positive feedback loop whereby reporters receive confirmation and positive reinforcement for submitting their report via email or in-app messaging. In early experiences with PV reporting, such feedback mechanisms were believed to have improved the user experience [4]. This also creates a conduit for follow-up reporting to NCAs as well as a mechanism for push notifications when new safety alerts are issued by NCAs. Some elements of this approach to enhancing user engagement were assessed in WEB-RADR.

Since the base WEB-RADR mobile app was to be deployed in multiple countries, yet evaluated centrally by the project, a certain level of design flexibility was required to allow country-specific language and branding. In some countries the NCA’s ADR-reporting programme already has a recognisable identity and the app was therefore designed to be extensible with colours, logos, etc. to mimic extant reporting forms and branding. See Table 3 for design recommendations.
Table 3

Recommendations for application (app) design

Recommendation

Rationale

General design recommendations applicable to health apps

 If a mobile app is to have different target groups (e.g. HCPs and members of the public) customisation for each target group should be considered

WEB-RADR explored the use of different report designs tailored for different types of users. For example, the users of the UK’s Yellow Card app can select MedDRA terms from a drop-down list to describe their ADR, ensuring that the reaction data are already coded when they are submitted to the MHRA. For the patient version, the MHRA is currently exploring ways to make this feature easier for patients to use with a patient-friendly MedDRA list that contains descriptions of medical concepts in layman’s terms [17, 18]. Additionally, distinct questions could be presented to different types of users who may be more likely to be able to provide certain details; for example, a patient may be able to report that a medical product was purchased from an Internet pharmacy, whereas that patient’s HCP may not have that level of knowledge

 If different versions of a mobile app are needed to best meet the differing needs of multiple target groups, it is recommended that a single app prototype is developed with as many standard features as possible. This can then be configured at lower overall cost

In WEB-RADR, three apps were developed for three pilot member states, all sharing a similar framework. Ultimately, for the second and third pilots in the Netherlands (Lareb) and Croatia (HALMED), the NCAs worked together to determine a set of common design standards to create a more generic app that would require less development than the first pilot in the UK (MHRA). The Lareb app was developed and launched 6 months after Yellow Card; the HALMED app was developed and launched 3 months after Lareb

The lower cost of using established applications has contributed to roll-out of the app to lower income countries; see the WEB-RADR portal (https://webradr.files.wordpress.com/2017/09/web-radr-stakeholder-event_theme2.pdf)

 In order to provide a user-friendly experience for as many users as possible, ensure that the app design complies with accessibility requirements or industry best practices that facilitate use of the app among patients and HCPs with different abilities

Best practices exist for ensuring maximum accessibility through web and app design. Both Apple and Android devices have documented design principles for developers [19, 20]. In addition, the European Commission has published a set of Web Content Accessibility Guidelines from the Web Accessibility Initiative [21]. Furthermore, many modern devices and operating systems have built-in accessibility features. It will be necessary for app developers to understand how the app can take advantage of and interact with these features, rather than ignore or override them

 The mobile app should give users the option of receiving push notifications from the health authority or PV centre when new safety information is available

In a quantitative survey study, more than half of respondents indicated interest in an optional push notification feature [22]. However, not all users expressed interest in this feature, for example, patients in Croatia were generally less interested, and there were also national differences in preference amongst HCPs

 Mobile apps that provide safety information should include the ability for the user to configure or customise the type of information that s/he receives (e.g. information at the individual product level or the therapeutic area level)

The quantitative survey study found that users were interested in receiving safety news regarding a wide variety of medicines. HCPs generally preferred to receive information on all approved drugs, while the largest group (40%) of patients preferred to receive only information pertaining to their own prescriptions [22]

 Mobile apps for ADR communication should provide users with the option of bypassing the login screen to automatically access the app after the information has been entered once

The quantitative survey study and feedback from the Netherlands app indicated that most patients and professionals preferred to access the app without having to login each time they opened it (57% and 70%, respectively) [22]. In addition, respondents to a user experience survey indicated that they would prefer not to create an account or enter login details. Providing patients and HCPs with the option to choose whether to login automatically would allow users to access the app quickly and easily—removing a hurdle to real-time ADR reporting and facilitating quick access to timely medicine news. However, should app developers choose to take this approach, they should work closely with stakeholders to determine the appropriate level of patient consent or privacy protection required for app functions that will not require password protection

 Mobile apps for ADR communication should be designed with the possibility to integrate with other health/drug safety applications and/or systems through, e.g. APIs, since this will extend their uptake by a wider range of organisations and a larger group of users, consequently enhancing sustainability

Several organisations including NHS trusts, EHR system providers and developers of health apps have approached MHRA with an interest in integration of different components of the WEB-RADR app into their own platforms. Their rationale is that such an approach enables them to provide validated forms or data in a format agreed by the regulatory authority, but without the need for them to develop, maintain and update the code themselves

Specific design recommendations applicable to ADR-reporting apps

 At a minimum, mobile reporting apps should provide an immediate acknowledgement message indicating that an ADR report has been successfully received

Many responders to both a quantitative survey study and a qualitative study expressed interest in receiving feedback or a confirmation following ADR report submission. Users appreciated that, upon report submission, the WEB-RADR apps displayed a simple notification informing the user that the report had been successfully sent. Earlier research by Lareb indicated a simple acknowledgement of receipt would satisfy most reporters [23]

Some users indicated that additional types of feedback would be appreciated (e.g. an overview of how frequently the ADR had been reported previously; feedback on what was done with the report), while others preferred to not receive this type of response as they found it intrusive. Therefore, giving users the option to select their preferences regarding ADR feedback within the app ought to be considered [8]. Also see experiences from the SCOPE project for feedback on report templates from reporters [24]

 Mobile apps for ADR reporting should be designed with an emphasis on two-way information exchange features, as this may encourage adoption and increase the likelihood of continuous user engagement and repeated use of the app

Findings from a qualitative study revealed that patients would view the news feature in the app favourably. Patients indicated that they might use the app as a quick and accessible source of information about various ADRs. For example, 84% of patients and 71% of HCPs specified that they would want to receive safety notifications on medicines from their local NCA [22]

 When considering which functionalities to add to an app targeting HCPs, it is suggested to focus on features that facilitate the provision of safety information rather than features facilitating ADR report submission

According to the quantitative study, HCPs expressed a greater interest in app features that would provide them with safety information than features that would enable ADR reporting [22]

 Strive for a report form design that captures all essential information in a user-friendly manner. For instance, free-text fields may be more appropriate for some questions than drop-down menus with a long list of options. It may be possible to reduce the time it takes to complete a report by keeping mandatory fields to a minimum. User-friendly phrases and descriptions may help to ensure that ADR reports are as accurate and complete as possible

Lareb operates a web-based ADR report form that includes many structured fields (e.g. radio buttons, drop-down menus). While this ensures data completeness, Lareb has also received complaints from users that reporting is time consuming. In addition, patients might find that their experiences are not well reflected in drop-down menu options and may find it preferable to provide this custom information themselves

In order to keep the length of the ADR-reporting form to a minimum, it was shown that a shortened form in the app still produces reports with sufficient clinical quality. The proportion of reports of at least moderate quality was high in both user groups (HCPs and patients) for all countries, but relatively lower for app reports: 83% vs 92% in the UK (p = 0.08); 85% vs 98% in the Netherlands (p < 0.01); and 78% vs 78% in Croatia (p = 1.0) [25]

Although free-text fields may provide patients and HCPs with the opportunity to describe ADRs in their own words, unstructured text resulting from these fields will need to be automatically coded or may require additional manual review [25]

 For mobile apps aimed at HCPs, it is recommended to keep adverse reaction report forms concise and clinically focused, in contrast to those provided for patients

The qualitative study found that time constraints were more likely to be a barrier to ADR reporting among HCPs than among patients [8]

 The ability to add attachments to reports is considered valuable—with the implementation of the ICH E2B(R3) format, this will become feasible and should be considered for future apps or upgrades

Mobile devices typically have built-in cameras, potentially making it easy for app users to include a photo or video along with ADR reports. Photos or videos could help capture product information (brand name, batch number, barcode) or provide a visual reference for certain ADRs (e.g. a photo of a skin rash, a video recording movement related to dyskinesia). For several years, Lareb has allowed reporters to attach additional files to ADRs submitted via web-forms. This not only decreases the burden on the reporter by allowing a simple way to relay medical information (e.g. uploading a medication list or a discharge letter from the hospital), but also increases Lareb’s understanding of the experienced ADR. The ICH E2B(R3) format that supports the transmission of additional files was not available for use during the WEB-RADR project. It follows that changes to databases and appropriate processes for receipt and analysis of such data would be needed to exploit information captured through these device features

ADR adverse drug reaction, API application programming interface, EHR Electronic Health Record, HCP healthcare professional, ICH International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, MedDRA Medical Dictionary for Regulatory Activities, MHRA Medicines and Healthcare products Regulatory Agency, NCA national competent authority, NHS National Health Service, PV pharmacovigilance, SCOPE Strengthening Collaboration for Operating Pharmacovigilance in Europe

2.3 Mobile Application Content and Data-Related Considerations

The WEB-RADR app has been developed to improve the two-way exchange of information (i.e. the reporting of ADRs and the provision of patient safety information). Therefore, the app contains an information-sharing component and an ADR-reporting component. The information-sharing component enables app users to receive safety alerts and news, as well as view aggregated statistics on previously reported ADRs.

A qualitative study that involved focus group discussions and face-to-face interviews showed that patients and HCPs were generally positive about the existing content of the app and suggested some additional functions that could also be useful [8]. Perspectives on the content of the app were further assessed in a large online survey among patients and HCPs in Europe [22]. It was shown that potential app users were somewhat more interested in an app for two-way exchange of safety information than in an app that only served one purpose (i.e. either just the information-sharing component or just the ADR-reporting component). The survey provided many suggestions on how to further improve the app to increase ADR reporting to NCAs.

A second online survey was conducted among current users of the WEB-RADR app. The survey asked about their experiences with the app. Most responders had a positive view of the value of the app in general, for both the ADR-reporting feature and the patient safety information provided in the app. Some suggestions were also received on how to further improve the app.

A quantitative survey study found that respondents from different countries preferred access to different types of information, e.g. drug stock levels (Croatia), alternative medicines (the Netherlands), and ADR mitigation strategies (the UK). Information on newly identified drug–drug interactions was of common interest across the three countries surveyed [8, 22]. This echoes our earlier recommendation in the pre-development considerations section (Sect. 2.1): when determining what kind of information to provide to users in the app, consideration should be given to user- and country-specific content. Interests among different user groups could differ, and countries may vary greatly in terms of what safety information may need to be broadcast; for example, local regulatory communication of newly identified drug–drug interactions may vary where there are formulary differences across countries. See Table 4 for a summary of the recommendations regarding content and data-related considerations.
Table 4

Recommendations regarding application (app) content and data-related considerations

Recommendation

Rationale

The ability to access all information contained in Patient Information Leaflets (or Product/Patient Package Inserts) should be considered as an additional feature of any app designed for patient use

The quantitative survey study found that patients who were interested in other app functions expressed most interest in being able to use the app to access both Patient Information Leaflets and an overview of previously reported ADRs, when presented with both these possibilities [22]. However, this approach should only be considered where the information can be appropriately formatted and indexed for use on a mobile device

The app should provide users with the ability to access, save a draft and review their own submitted reports as well as summary information regarding ADRs previously reported to the NCA

Users preferred having the ability to look at their previous ADR reports. In addition, patients indicated that a major benefit of the app would be if it allowed them to check whether a symptom has previously been reported as an ADR. Several studies have shown that patients are sometimes uncertain about an association between a symptom and a drug and having this information in the app could reduce their uncertainty about an association between a drug and their symptoms [8, 22, 26, 27, 28, 29]. To respect users’ desire for anonymity, however, there should exist an option not to store personal details when saving a report

Apps should provide information to users about the importance of ADR reporting. For patients, this should contain an explanation of its importance for patient safety and that the information they voluntarily submit can benefit other stakeholders

The quantitative study found that patients were mainly motivated to report ADRs to contribute to patient safety knowledge and to share their experiences for the benefit of others [22]

ADR adverse drug reaction, NCA national competent authority

2.4 Mobile Application Implementation, Awareness and Uptake

NCAs can use various strategies to inform potential users about the existence of the mobile app, but it is most efficient to use those channels and sources that reach the audience with potentially the highest interest in the app.

To assess which patients and HCPs are most interested in an app with capabilities for the two-way exchange of safety information, data from a European-wide online survey study were used [22]. In the survey, patients and HCPs were asked to what extent they were interested in the app. Analyses of the HCP data showed that HCPs who already (at least sometimes) use a health app were very interested in the app. This was also shown among the patients. In addition, patients that were younger were very interested in the app.

See Table 5 for a summary of recommendations on implementation, awareness and uptake.
Table 5

Recommendations on application (app) implementation, awareness and uptake

Recommendation

Rationale

Prior to app launch, host organisations should devise a publicity strategy to inform HCPs and the public of the app and encourage uptake. Continuously publicising the app over a longer period—and not just following app launch— may also serve to encourage app uptake

The Yellow Card app was launched in July 2015, and the highest number of app users registered around this time. The second-highest number of registrants occurred Sept 9–10, 2015, immediately following a brief update about the app on the gov.uk website [30]. Similarly, the Lareb app was launched at the end of January 2016. While the highest number of users registered immediately following the launch in February 2016, there was a second surge of app registrants in April following an article published in a Dutch nursing magazine [31]

The app should be hosted by an organisation that is trusted by and preferably familiar to its target users. If not familiar, steps should be taken to increase awareness of the host organisation in the peri-launch period. For example, the app could be promoted to HCPs via NCAs and/or professional bodies

A qualitative study found that respondents would be more likely to use an app that was associated with a familiar and trustworthy source [8]

NCAs and professional bodies are preferred senders of risk information for professionals. Information about the app seems to be sufficiently related to that topic to apply the findings of the SCOPE survey [24]

Efforts to inform HCPs and patients about the app should focus on those who use other health apps. Hence, it is likely to be beneficial to advertise the app in other health apps

Patients and HCPs who already use other health apps were more interested in the app than those that did not use other health apps. It was also shown that younger patients tended to be more interested in the app [22]

Country-specific communication plans should be developed

Possible channels and sources for promoting the app will differ per country; therefore, country-specific strategies are needed. In the quantitative survey study, there were quite a few patients who had never heard of their national agency [22]

HCP healthcare professional, NCA national competent authority, SCOPE Strengthening Collaboration for Operating Pharmacovigilance in Europe

2.5 Mobile Application Impact and Miscellaneous Recommendations

The recommendations presented in Table 6 relate to the assessment of the overall impact of mobile reporting apps for PV and other stand-alone recommendations.
Table 6

Recommendations on assessment of the overall impact of applications (apps) for pharmacovigilance and other stand-alone recommendations

Recommendation

Rationale

The app should be considered an additional channel for ADR reporting

Analysis has shown very little qualitative difference between reports from ‘conventional means’ and reports from the app [25]

Consideration could be given to having the app tested by an independent reviewer that specialises in health apps prior to launch

Some countries have healthcare specific app stores that contain apps that have been tested by the relevant national health services and certified for usability, utility, and security. This can generate confidence in and increase awareness of the app

Consideration should be given to the means of measuring the public health impact of the app after launch with HCPs and patients

Measuring the impact of any mobile app can be challenging. It is easy to capture the number of downloads or reports, although these figures do not provide insights into the value of news feeds and safety data to patients or the value provided to HCPs and whether it impacts their prescribing. Valuable metrics may include:

 How often reports are sent

 How often the app is opened, how many new downloads, how many ADRs reported

 Impact of signals arising from the app compared to traditional reporting mechanisms

ADR adverse drug reaction, HCP healthcare professional

3 Discussion

A series of recommendations relevant to the development of mobile apps for the reporting of ADRs are presented with the intention of informing PV professionals and other stakeholders, particularly those with an interest in the development of research methods and digital technologies. Each recommendation is based on the experiences and outcomes of the research conducted under the auspices of IMI WEB-RADR. Collectively, the recommendations should not be considered a comprehensive treatise on the subject, rather they should be considered within the context of the entire corpus of research and regulatory guidance documents that exist outside of IMI WEB-RADR.

Since the start of the WEB-RADR project, mobile ADR-reporting apps have been launched in five separate countries. This group of participating countries has presented both diverse target populations of HCPs and non-HCP users, predominantly patients, as well as widely varying reporting systems and diverse technical resources that have impacted how each app has been configured. It is our hope that, having launched these apps in five unique settings, the recommendations, when taken together with their caveats, will be applicable to many other settings. That said, careful consideration should be given to the generalisability of the recommendations in circumstances that are substantially different from those described in the original research.

Overall, our experience has indicated that a deep understanding of the target app-user population, through local knowledge and potential user outreach prior to embarking on development activities, can facilitate effective design from the start. However, it is equally important to continue improving the app based on user feedback and evaluation of outputs after any public release—particularly following initial launch and major updates. Additionally, preferences and needs for patient safety information and reporting can vary greatly among different groups, and a balance must be struck between providing enough features that are useful to a wide range of users without attempting to create an app that ‘does it all’.

Communication in support of the launch of the app is important, followed by further provision of information at appropriate intervals to keep stakeholders informed about enhancements and new features of the software. Once established, a good place to inform patients about the app is at first prescription or first dispensing of a medicine. Patients are most likely to experience ADRs when starting new medications [32].

It is important to note that even if an app for ADR reporting is well-designed and user-friendly, it will not serve its purpose if the target population is not familiar with that purpose. Patient awareness of reporting ADRs is therefore critical to adoption and utilisation of the app and may be a factor beyond the scope of designing and developing the app itself. It might be necessary to raise public awareness of ADR reporting prior to evaluating the need for an app; in other cases, the existence of a mobile app could serve to drive awareness of ADR reporting among the public, and it may be useful to address underreporting barriers, as part of a publicity strategy. More research is needed to explore ways to encourage patient reporting of ADRs in a balanced way that does not lead to bias or soliciting.

Since release, each app has seen varying degrees of exposure. The Yellow Card mobile app was launched as an extension of MHRA’s Yellow Card Scheme—a safety reporting programme for human medicinal products that has been running for over 50 years. To date, MHRA has received 505 ADR reports through its mobile app, launched in July 2015 (see Table 1). Objectively speaking, this number pales in comparison to the 26,000 ADR reports the MHRA received in 2016 directly from HCPs and the public, and there is a tendency to view this as a shortcoming of the mobile app itself [33]. However, the value identified by users was much broader than just reporting and extended to engagement with the wider PV system and accessing timely safety information. As smartphone users increasingly use their mobile devices to access the Internet (in the UK, for example, 73% of adults browsed the Internet using their mobile phones in 2017, compared to 36% in 2011), so grows the opportunity for health authorities to reach patients through digital channels where they are already seeking information [34].

It is not practical to expect the outputs from novel, patient-centric reporting channels to mimic those from traditional channels, particularly as the value of patient-reported individual case safety reports (ICSRs) has only been recognised relatively recently [3]. There is therefore a need to re-calibrate expectations and perspectives when evaluating mechanisms for patient ADR reporting and establish a methodology that can allow an assessment of these initiatives entirely on their own, and not in the context of traditional reporting channels. For instance, Lareb has found that its Bijwerking app has encouraged participation from new members of the public, specifically HCPs, who have never submitted an ADR report previously, suggesting that the existence of the mobile app has increased the breadth and diversity of reporters [35]. Additionally, evaluation of reports originating from the apps revealed that while the resulting data had lower completeness scores, the app reports demonstrated clinical quality that did not differ significantly from that of reports submitted via conventional reporting channels and contributed to signal detection and safety evaluation activities.

Probably the most important recommendations are those that relate to engagement of users of the app, in particular, establishing the two-way interchange of safety information. Users of the app valued the provision of graphs and charts showing the safety profile of their medicines. This positive reinforcement of the value of transmitting ICSR was a feature of the feedback received during focus groups and interviews. However, a commitment to provide automated feedback should not override the current practices in terms of providing feedback for other reporting routes [22]. In general, provision of more detailed feedback information should be well thought through, as it may introduce questions or concerns relating to the complexity of mandatory ADR reporting requirements that may require time to address.

Equally important was the protection of data privacy, in accordance with national and European laws. Patients wanted assurance that the highest standards of confidentiality were maintained during the data reporting and transfer processes. Patients also found it important to be able to review their own previously submitted reports and to submit corrections or follow-up information later, if needed. Overall, the app and surrounding publicity brought the importance of PV to the attention of various stakeholders in public health. In the case of HCPs, this served as a reminder to report suspected ADRs, but for patients and caregivers, the app provided an alerting mechanism, highlighting the value of the public in recognising potential safety concerns.

The app could be considered an important mechanism to facilitate compliance with the European Union PV framework legislation which became operational in July 2012, obliging all NCAs and MAHs to record and report cases of suspected ADRs received from patients. Careful consultation with a range of patients and patient support groups helped in the design and delivery of a functional app to facilitate patient reporting. An app provides an additional means of reporting suspected ADRs and communication and, furthermore, projects an innovative and progressive image; in turn this fosters engagement with stakeholders. Health authorities might consider providing an app as an additional means of reporting and/or communication; however, the value versus opportunity costs, which were not specifically evaluated in WEB-RADR, are likely to vary from country to country. It is important that regulators are accessible through means that patients and HCPs use in their day-to-day lives; it should not be any more challenging to report an ADR than to use any other service. Furthermore, responsiveness to new technologies projects an innovative and progressive image that in turn fosters trust and improves dialogue with stakeholders.

A maintenance and support programme has been established with the aim of ensuring the sustainability of the app; however, questions remain about its long-term utility. From the patient perspective, there may be no need to repeatedly use the app to submit an ADR report; many patients will not experience multiple ADRs (if at all); thus, most users will not find it necessary to keep the app on their device, let alone use it regularly. Conversely, for HCPs, the app may prove useful on multiple occasions and could be of even greater utility if the app was integrated with other software, such as prescribing records, dispensing systems and other healthcare systems. The WEB-RADR project has established a follow-up programme to develop the app using additional medical ontology mappings and by evaluating additional application programming interfaces for linking to other software.

In the future, the potential value of the app in supporting the notification of product quality issues and/or product complaints should be investigated. This could yield important information about manufacturing issues or batch-related concerns, particularly if used in conjunction with built-in image capture technologies. Similarly, specialist use of the app should be considered if an emerging safety issue arises with a product or class of medicines. Any situation such as this would require the close monitoring of a cohort of patients in a defined setting, and the app could provide a useful adjunct to traditional methods of PV by providing safety data in real time for rapid evaluation.

4 Conclusions

Over a period of 3 years, IMI WEB-RADR has addressed several important research questions relevant to the development of an app for reporting suspected ADRs. The resultant recommendations point to a series of pragmatic steps that those working in the PV community should consider before designing and building an app in support of safety reporting. The recommendations support the idea that the app is an important adjunct to existing ADR reporting pathways; however, engagement of app users is key to successful app development and adoption, in particular, in establishing the two-way interchange of safety information. As a younger generation of app-literate patients matures, it seems likely that app-based reporting will grow in importance and that, at some point, structured electronic reports will supplant the use of traditional paper forms.

Footnotes

  1. 1.

    The non-binding recommendations presented in this report represent the views of the authors and do not represent the views or policies of the authors’ respective affiliations (unless by coincidence), even if employees of those organisations at the time of preparing this paper.

  2. 2.

    The General Advisory Board (GAB) consists of experts from organisations that are independent of those delivering and managing the project, as well as work package leaders. The GAB has an important role, providing advice on the WEB-RADR project, and identifying other initiatives which may be relevant to the project. For further details see https://web-radr.eu/governance/.

Notes

Author Contributions

The research leading to these results was conducted as part of the WEB-RADR consortium, http://webradr.eu), which is a public–private partnership coordinated by the Medicines and Healthcare products Regulatory Agency. In addition to the authors, the following persons contributed to mobile app development and/or research within the various work packages that form the basis for these recommendations. All contributors were invited to review the final draft manuscript of this article prior to submission. Andrew Cochrane (Novartis), Rajesh Ghosh (Novartis), Susana Goncalves (Novartis), Thierry Guet (Novartis), Christiane Michel (Novartis), Bita Rezaallah (Novartis), Eric Scalfaro (Novartis), Miguel Teixeira (Novartis), Alastair Sutcliffe (Institute of Child Health), François Houÿez (EURORDIS), Raphael van Eemeren (AMGEN), Sandra Fernandes (Sanofi), Lisa Wong (Institute of Child Health), Faiza Afsal (Institute of Child Health), Carmen Lasheras Ruiz (EURORDIS), Denis Costello (EURORDIS), Karin Hace (AMGEN), Lucas Pedron Baptista (Booz Allen Hamilton/Epidemico Ltd.), Kyra McKenna (Booz Allen Hamilton/Epidemico Ltd.), Harold Rodriguez (Booz Allen Hamilton/Epidemico Ltd.), Petar Mas (HALMED), Henric Taavola (UMC), Ola Caster (UMC), Magnus Wallberg (UMC), Ingrid Oosterhuis (Lareb). The views expressed in this paper are those of the authors only and not of their respective institution or company.

Compliance with Ethical Standards

Funding

The WEB-RADR project has received support from the Innovative Medicine Initiative Joint Undertaking (http://www.imi.europa.eu) under Grant Agreement no 115632, resources of which are composed of financial contributions from the European Union’s Seventh Framework Programme (FP7/2007-2013) and EFPIA companies’ in-kind contribution.

Conflict of interest

The following authors have declared no potential conflicts of interest: Sieta T. de Vries, Stephanie Bodin-Parssinen, Linda Härmark, Phil Tregunno, Simon Maskell, Raphael Van Eemeren, Alicia Ptaszynska-Neophytou, Victoria Newbould, Nabarun Dasgupta, and Peter G.M. Mol. Carrie Pierce is an employee of Booz Allen Hamilton, formerly Epidemico Inc.; David Lewis is an employee of Novartis Pharma AG and is a shareholder in Novartis and GlaxoSmithKline; Antoni Wisniewski is an employee of AstraZeneca and shareholder of AstraZeneca and GlaxoSmithKline; Sara Gama is an employee of Novartis Pharma AG.

References

  1. 1.
    Ghosh R, Lewis D. Aims and approaches of Web-RADR: a consortium ensuring reliable ADR reporting via mobile devices and new insights from social media. Expert Opin Drug Saf. 2015;14(12):1845–53.  https://doi.org/10.1517/14740338.2015.1096342.CrossRefGoogle Scholar
  2. 2.
    de Langen J, van Hunsel F, Passier A, de Jong-van den Berg L, van Grootheest K. Adverse drug reaction reporting by patients in the Netherlands: three years of experience. Drug Saf. 2008;31(6):515–24.CrossRefGoogle Scholar
  3. 3.
    Avery AJ, Anderson C, Bond CM, Fortnum H, Gifford A, Hannaford PC, et al. Evaluation of patient reporting of adverse drug reactions to the UK ‘Yellow Card Scheme’: literature review, descriptive and qualitative analyses, and questionnaire surveys. Health Technol Assess. 2011;15(20):1–234, iii–iv.  https://doi.org/10.3310/hta15200.CrossRefGoogle Scholar
  4. 4.
    Bahk CY, Goshgarian M, Donahue K, Freifeld CC, Menone CM, Pierce CE, et al. Increasing patient engagement in pharmacovigilance through online community outreach and mobile reporting applications: an analysis of adverse event reporting for the Essure device in the US. Pharm Med. 2015;29(6):331–40.  https://doi.org/10.1007/s40290-015-0106-6.CrossRefGoogle Scholar
  5. 5.
    British Broadcasting Corporation. ‘One billion’ affected by Yahoo hack. 2016. http://www.bbc.co.uk/news/world-us-canada-38324527. Accessed 20 Mar 2018.
  6. 6.
    British Broadcasting Corporation. Customer data stolen at Cex online games store. 2017. http://www.bbc.co.uk/news/technology-41095162. Accessed 20 Mar 2018.
  7. 7.
    British Broadcasting Corporation. Facebook broke German privacy laws, court rules. 2018. http://www.bbc.com/news/technology-43035968. Accessed 07 June 2018.
  8. 8.
    de Vries ST, Wong L, Sutcliffe A, Houyez F, Ruiz CL, Mol PG, et al. Factors influencing the use of a mobile app for reporting adverse drug reactions and receiving safety information: a qualitative study. Drug Saf. 2017;40(5):443–55.  https://doi.org/10.1007/s40264-016-0494-x.CrossRefGoogle Scholar
  9. 9.
    European Medicines Agency. EudraVigilance: electronic reporting. http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_000686.jsp&mid=WC0b01ac0580a69261. Accessed 20 Mar 2018.
  10. 10.
    Piejko P. Android grew faster than iOS in Q1 2016. 2016. https://deviceatlas.com/blog/android-grew-faster-ios-q1-2016. Accessed 20 Mar 2018.
  11. 11.
    Center PR. Smartphone ownership and internet usage continues to climb in emerging economies. In: Global attitudes & trends. Pew Research Center. 2016. http://www.pewglobal.org/2016/02/22/smartphone-ownership-and-internet-usage-continues-to-climb-in-emerging-economies/. Accessed 22 Mar 2018.
  12. 12.
    European Commission. Proposal for an ePrivacy Regulation. 2018. https://ec.europa.eu/digital-single-market/en/proposal-eprivacy-regulation. Accessed 31 Oct 2018.
  13. 13.
    Medicines and Healthcare products Regulatory Agency. An introductory guide to the medical device regulation (MDR) and the in vitro diagnostic medical device regulation (IVDR). https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/640404/MDR_IVDR_guidance_Print_13.pdf. Accessed 31 Oct 2018.
  14. 14.
    Herdeiro MT, Figueiras A, Polonia J, Gestal-Otero JJ. Physicians’ attitudes and adverse drug reaction reporting: a case-control study in Portugal. Drug Saf. 2005;28(9):825–33.CrossRefGoogle Scholar
  15. 15.
    European Medicines Agency. Guideline on good pharmacovigilance practices (GVP) module VI—Collection, management and submission of reports of suspected adverse reactions to medicinal products (Rev 2), EMA/873138/2011 Rev 2 (2017).Google Scholar
  16. 16.
    MedDRA Maintenance and Support Services Organization. Patient-friendly term list. MedDRA MSSO. 2018. https://www.meddra.org/patient-friendly-term-list. Accessed 22 Mar 2018.
  17. 17.
    De Vries ST, Harrison J, Revelle P, Ptaszynska-Neophytou A, Radecka A, Ragunathan G, et al. Use of a patient-friendly terms list in the adverse drug reaction report form: A database study. Drug Saf. 2019.  https://doi.org/10.1007/s40264-019-00800-x.Google Scholar
  18. 18.
    Harmark L, Raine J, Leufkens H, Edwards IR, Moretti U, Sarinic VM, et al. Patient-reported safety information: a renaissance of pharmacovigilance? Drug Saf. 2016;39(10):883–90.  https://doi.org/10.1007/s40264-016-0441-x.CrossRefGoogle Scholar
  19. 19.
    Apple. Human Interface Guidelines. iOS design themes. In: Human interface guidelines. Apple Inc. https://developer.apple.com/ios/human-interface-guidelines/overview/themes/. Accessed 25 Mar 2018.
  20. 20.
    Android.com. Android developers guides: accessibility. Android.com. https://developer.android.com/guide/topics/ui/accessibility/index.html. Accessed 26 Mar 2018.
  21. 21.
    European Commission. WAI—Web Content Accessibility Guidelines (WCAG) 2.0. In: Web Accessibility Initiative. European Commission. 2016. http://ec.europa.eu/ipg/standards/accessibility/wcag-20/index_en.htm. Accessed 26 Mar 2018.
  22. 22.
    de Vries ST, Denig P, Lasheras Ruiz C, Houÿez F, Wong L, Sutcliffe A, et al. Interest in a mobile app for two-way risk communication: a survey study among European healthcare professionals and patients. Drug Safety. 2018;41(7):697–712.  https://doi.org/10.1007/s40264-018-0648-0.CrossRefGoogle Scholar
  23. 23.
    Rolfes L, van Hunsel F, van Grootheest K, van Puijenbroek E. Feedback for patients reporting adverse drug reactions; satisfaction and expectations. Expert Opin Drug Saf. 2015;14(5):625–32.  https://doi.org/10.1517/14740338.2015.1021775.CrossRefGoogle Scholar
  24. 24.
    SCOPE. SCOPE Work Package 4 ADR Collection: feedback to patient ADR reports. In: The Strengthening Collaboration for Operating Pharmacovigilance in Europe (SCOPE) Joint Action. SCOPE. 2018. Accessed 25 Mar 2018.Google Scholar
  25. 25.
    Oosterhuis I, Taavola H, Tregunno PM, Mas P, Gama S, Newbould V, et al. Characteristics, quality and contribution to signal detection of spontaneous reports of adverse drug reactions via the WEB-RADR mobile application: a descriptive cross-sectional study. Drug Saf. 2018.  https://doi.org/10.1007/s40264-018-0679-6.
  26. 26.
    Britten N. Medication errors: the role of the patient. Br J Clin Pharm. 2009;67(6):646–50.  https://doi.org/10.1111/j.1365-2125.2009.03421.x.CrossRefGoogle Scholar
  27. 27.
    de Vries ST, Haaijer-Ruskamp FM, de Zeeuw D, Denig P. The validity of a patient-reported adverse drug event questionnaire using different recall periods. Qual Life Res. 2014;23(9):2439–45.  https://doi.org/10.1007/s11136-014-0715-7.CrossRefGoogle Scholar
  28. 28.
    de Vries ST, Mol PG, de Zeeuw D, Haaijer-Ruskamp FM, Denig P. Development and initial validation of a patient-reported adverse drug event questionnaire. Drug Saf. 2013;36(9):765–77.  https://doi.org/10.1007/s40264-013-0036-8.CrossRefGoogle Scholar
  29. 29.
    Chaipichit N, Krska J, Pratipanawatr T, Uchaipichat V, Jarernsiripornkul N. A qualitative study to explore how patients identify and assess symptoms as adverse drug reactions. Eur J Clin Pharmacol. 2014;70(5):607–15.  https://doi.org/10.1007/s00228-014-1653-6.CrossRefGoogle Scholar
  30. 30.
    Medicines and Healthcare products Regulatory Agency. Download the Yellow Card mobile app to report suspected adverse drug reactions. Government Digital Service (UK), GOV.UK. 2015. https://www.gov.uk/drug-safety-update/download-the-yellow-card-mobile-app-to-report-suspected-adverse-drug-reactions. Accessed 26 Mar 2018.
  31. 31.
    Loghum BSv. Verpleegkundigen melden bijwerkingen te weinig. In: nursing. SPRINGER Nature. 2016. https://www.nursing.nl/verpleegkundigen-melden-bijwerkingen-te-weinig/. Accessed 26 Mar 2018.
  32. 32.
    Frew A. General principles of investigating and managing drug allergy. Br J Clin Pharmacol. 2011;71(5):642–6.  https://doi.org/10.1111/j.1365-2125.2011.03933.x.CrossRefGoogle Scholar
  33. 33.
    British Pharmacopeia Commission. Human Medicines Regulations 2012 Advisory Bodies—Annual Report 2016. Medicine & Healthcare products Regulatory Agency.Google Scholar
  34. 34.
    Office of National Statistics. Internet access—households and individuals, Great Britain: 2017. 2017. https://www.ons.gov.uk/peoplepopulationandcommunity/householdcharacteristics/homeinternetandsocialmediausage/bulletins/internetaccesshouseholdsandindividuals/2017/. Accessed 31 Oct 2018.
  35. 35.
    European Medicines Agency. Innovative Medicines Initiative WEB-RADR Workshop Report: Mobile technologies and social media as new tools in pharmacovigilance. In: Pharmacovigilance and epidemiology. European Medicines Agency. 2017. http://www.ema.europa.eu/docs/en_GB/document_library/Report/2017/02/WC500221615.pdf. Accessed 26 Mar 2018.

Copyright information

© The Author(s) 2019

Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Carrie E. Pierce
    • 1
  • Sieta T. de Vries
    • 2
  • Stephanie Bodin-Parssinen
    • 3
  • Linda Härmark
    • 4
  • Phil Tregunno
    • 5
  • David J. Lewis
    • 6
    • 7
  • Simon Maskell
    • 8
  • Raphael Van Eemeren
    • 9
  • Alicia Ptaszynska-Neophytou
    • 5
  • Victoria Newbould
    • 10
  • Nabarun Dasgupta
    • 11
  • Antoni F. Z. Wisniewski
    • 12
  • Sara Gama
    • 6
  • Peter G. M. Mol
    • 2
    • 13
    Email author
  1. 1.Booz Allen Hamilton (formerly Epidemico, Inc.)BostonUSA
  2. 2.Department of Clinical Pharmacy and PharmacologyUniversity of Groningen, University Medical Center GroningenGroningenThe Netherlands
  3. 3.UCB PharmaceuticalsBrusselsBelgium
  4. 4.Netherlands Pharmacovigilance Centre Lareb‘s-HertogenboschThe Netherlands
  5. 5.Vigilance, Intelligence and Research Group, Medicines and Healthcare Products Regulatory Agency (MHRA)LondonUK
  6. 6.Chief Medical Office and Patient Safety, Global Drug Development, NovartisBaselSwitzerland
  7. 7.Department of Pharmacy, Pharmacology and Postgraduate MedicineUniversity of HertfordshireHertfordshireUK
  8. 8.Department of Electrical Engineering and ElectronicsUniversity of LiverpoolLiverpoolUK
  9. 9.Amgen Europe BVBredaThe Netherlands
  10. 10.Pharmacovigilance Department, Inspections and Human Medicines Pharmacovigilance Division, European Medicines Agency (EMA)LondonUK
  11. 11.University of North Carolina at Chapel HillChapel HillUSA
  12. 12.AstraZeneca, Global Regulatory Affairs, Patient Safety and Quality AssuranceCambridgeUK
  13. 13.Dutch Medicines Evaluation Board (CBG-MEB)UtrechtThe Netherlands

Personalised recommendations