Skip to main content
SpringerLink
Log in

Evolution of Cross-Sectional Survey Protocol Quality Over Time: A Case Series of Index U.S. REMS Knowledge Survey Protocols (2007–2020)

  • Original Research Article
  • Published:
Drug Safety Aims and scope Submit manuscript

This article has been updated

Abstract

Introduction

Surveys are commonly used to assess effectiveness of FDA-required risk evaluation and mitigation strategies (REMS) for drugs and biologics in the United States.

Objective

The aim of this study was to assess the scientific rigor of REMS knowledge survey protocols submitted to FDA and compare protocols before and after FDA’s 2012 public workshop and 2019 draft guidance.

Method

A content analysis of index survey protocols submitted to FDA (2007–2020) for single-product REMS with elements to assure safe use (39 programs, 78 protocols) was conducted. Each protocol was scored against 52 core essential elements (CEE), abstracted from FDA’s guidance and grouped into six domains: study objective (n = 5), study design (n = 18), survey instrument (n = 9), participant recruitment (n = 7), survey administration (n = 9), and statistical analysis plan (n = 4). Scores were collected by time periods: (A) Oct 2007 to Jul 2012; (B) Aug 2012 to Feb 2019; (C) Mar 2019 to Dec 2020; and compared using logistic generalized linear mixed models adjusting for domain, survey population, vendor, program, and protocol.

Results

There were 30 (38.5%), 40 (51.3%), and 8 (10.3%) protocols submitted in time period A, B, and C, respectively. Adjusted marginal means of elements present (on the probability scale) by time period were 0.5816 (SE = 0.0242), 0.6429 (SE = 0.0229), and 0.7543 (SE = 0.0394). The likelihood of missing a CEE declined over time (adjusted p-value = 0.0094, time period A vs C). The statistical analysis plan domain had the most improvement; study design remained the weakest domain with the scientific justification CEE particularly underrepresented.

Conclusion

The rigor of REMS knowledge survey protocols improved over time consistent with FDA’s efforts to advance regulatory science, but gaps remain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Change history

  • 29 September 2023

    The original article has been corrected. Supplementary material was missing from this article and has now been uploaded.

References

  1. United States Food and Drug Administration, “What We Do,” 2018. https://www.fda.gov/about-fda/what-we-do#missionhttps://www.fda.gov/about-fda. Accessed 01 Feb 2023.

  2. United States Food and Drug Administration, “PDUFA REAUTHORIZATION PERFORMANCE GOALS AND PROCEDURES FISCAL YEARS 2023 THROUGH 2027,” 2022. [Online]. Available: https://www.fda.gov/media/151712/download. Accessed 01 Feb 2023.

  3. United States Food and Drug Administration, “Advancing Regulatory Science at FDA: FOCUS AREAS OF REGULATORY SCIENCE (FARS),” 2022. [Online]. Available: https://www.fda.gov/media/161381/download#:~:text=we%20identified%20priorities%20in%20women’s,%2C%20rare%20diseases%2C%20and%20oncology. Accessed 01 Feb 2023.

  4. United States Food and Drug Administration, “Risk Evaluation and Mitigation Strategies | REMS,” 2021. https://www.fda.gov/drugs/drug-safety-and-availability/risk-evaluation-and-mitigation-strategies-rems. Accessed 01 Feb 2023.

  5. United States Food and Drug Administration, “FDA’s Role in Managing Medication Risks,” 2018. https://www.fda.gov/drugs/risk-evaluation-and-mitigation-strategies-rems/fdas-role-managing-medication-risks. Accessed 01 Feb 2023.

  6. Huynh L, Toyserkani GA, Morrato EH. Pragmatic applications of implementation science frameworks to regulatory science: an assessment of FDA Risk evaluation and mitigation strategies (REMS) (2014–2018). BMC Health Serv Res. 2021;21(1):779. https://doi.org/10.1186/s12913-021-06808-3.

    Article  PubMed  PubMed Central  Google Scholar 

  7. United States Food and Drug Administration, “FDA Risk Evaluation and Mitigation Strategy (REMS) Public Dashboard,” 2023. https://fis.fda.gov/sense/app/ca606d81-3f9b-4480-9e47-8a8649da6470/sheet/dfa2f0ce-4940-40ff-8d90-d01c19ca9c4d/state/analysis?utm_medium=email&utm_source=govdelivery. Accessed 01 Feb 2023.

  8. United States Food and Drug Administration, “REMS Assessment: Planning and Reporting Guidance for Industry,” 2019. Accessed: Feb. 01, 2023. [Online]. Available: https://www.fda.gov/media/119790/download

  9. Aday LA, Cornelius LJ. Designing and conducting health surveys : a comprehensive guide. 3rd ed. Hooboken: John Wiley & Sons Inc.; 2006.

    Google Scholar 

  10. Morrato EH, et al. Glucose testing for adults receiving medicaid and antipsychotics: a population-based prescriber survey on behaviors, attitudes, and barriers. Psychiatr Serv. 2016;67(7):798–802. https://doi.org/10.1176/appi.ps.201500181.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Brasel K, Haider A, Haukoos J. Practical guide to survey research. JAMA Surg. 2020;155(4):351. https://doi.org/10.1001/jamasurg.2019.4401.

    Article  PubMed  Google Scholar 

  12. Ponto J. Understanding and evaluating survey research. J Adv Pract Oncol. 2015;6(2):168–71.

    PubMed  PubMed Central  Google Scholar 

  13. Brewer SE, Campagna EJ, Morrato EH. Advancing regulatory science and assessment of FDA REMS programs: a mixed-methods evaluation examining physician survey response. J Clin Transl Sci. 2019;3(4):199–209. https://doi.org/10.1017/cts.2019.400.

    Article  PubMed  PubMed Central  Google Scholar 

  14. Amend KL, et al. Assessment of risk evaluation and mitigation strategies (REMS) for VARENICLINE (Chantix): a multistage patient survey. Pharmacoepidemiol Drug Saf. 2018;27(3):253–62. https://doi.org/10.1002/pds.4373.

    Article  CAS  PubMed  Google Scholar 

  15. Brandenburg NA, Bwire R, Freeman J, Houn F, Sheehan P, Zeldis JB. Effectiveness of risk evaluation and mitigation strategies (REMS) for lenalidomide and thalidomide: patient comprehension and knowledge retention. Drug Saf. 2017;40(4):333–41. https://doi.org/10.1007/s40264-016-0501-2.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Dal Pan GJ. Commentary on ‘The effectiveness of varenicline medication guide for conveying safety information to patients: a rems assessment survey’ by Enger et al. Pharmacoepidemiol Drug Saf. 2013;22(7):716–8. https://doi.org/10.1002/pds.3450.

    Article  CAS  Google Scholar 

  17. Enger C, Younus M, Petronis KR, Mo J, Gately R, Seeger JD. The effectiveness of varenicline medication guide for conveying safety information to patients: a REMS assessment survey. Pharmacoepidemiol Drug Saf. 2013;22(7):705–15. https://doi.org/10.1002/pds.3400.

    Article  CAS  PubMed  Google Scholar 

  18. Ishihara L, Beck M, Travis S, Akintayo O, Brickel N. Physician and pharmacist understanding of the risk of urinary retention with retigabine (Ezogabine): a REMS assessment survey. Drugs Real World Outcomes. 2015;2(4):335–44. https://doi.org/10.1007/s40801-015-0042-5.

    Article  PubMed  PubMed Central  Google Scholar 

  19. Hays RD, Liu H, Kapteyn A. Use of Internet panels to conduct surveys. Behav Res Methods. 2015;47(3):685–90. https://doi.org/10.3758/s13428-015-0617-9.

    Article  PubMed  PubMed Central  Google Scholar 

  20. Groves R. Survey methodology. 2nd ed. Hoboken: John Wiley & Sons Inc; 2009.

    Google Scholar 

  21. Dillman D, Hox J, Leeuw E. International handbook of survey methodology. European Association of Methodology 2008.

  22. Dillman D, Smyth J, Christian L. Internet, phone, mail, and mixed-mode surveys: the tailored design method. Hoboken: John Wiley & Sons Inc; 2014.

    Google Scholar 

  23. United States Food and Drug Administration., “Risk Evaluation and Mitigation Strategy Assessments: Social Science Methodologies to Assess Goals Related to Knowledge; Public Workshop; Issue Paper,” 2012. https://www.federalregister.gov/documents/2012/05/03/2012-10646/risk-evaluation-and-mitigation-strategy-assessments-social-science-methodologies-to-assess-goals. Accessed 01 Feb 2023.

  24. United States Food and Drug Administration, “Survey Methodologies to Assess REMS Goals That Relate to Knowledge: Guidance for Industry,” 2019.. [Online]. Available: https://www.fda.gov/media/119789/download. Accessed 01 Feb 2023.

  25. United States Food and Drug Administration, “Identification of Drug and Biological Products Deemed to Have Risk Evaluation and Mitigation Strategies for Purposes of the Food and Drug Administration Amendments Act of 2007,” Mar. 2008.. [Online]. Available: https://www.federalregister.gov/documents/2008/03/27/E8-6201/identification-of-drug-and-biological-products-deemed-to-have-risk-evaluation-and-mitigation. Accessed 01 Feb 2023.

  26. Columbia University’s Mailman School of Public Health, “Content Analysis,” Jan. 10, 2023. https://www.publichealth.columbia.edu/research/population-health-methods/content-analysis Accessed 01 Feb 2023.

  27. Baxter J. Content analysis. International Encyclopedia of Human Geography. 2020;391–396.

  28. National Institutes of Health, “Protocol Templates for Clinical Trials.” [Online.] https://grants.nih.gov/policy/clinical-trials/protocol-template.htm. Accessed 03 July 2023.

  29. European Medicines Agency,” Guidance for the format and content of the protocol of non-interventional post-authorization safety studies. Sep. 2012 [Online]. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUKEwjck6PNn4eAAxWOD1kFHSC5ArwQFnoECCQQAQ&url=https%3A%2F%2Fwww.ema.europa.eu%2Fen%2Fdocuments%2Fother%2Fguidance-format-content-protocol-non-interventional-post-authorisation-safety-studies_en.pdf&usg=AOvVaw1D2I9lkqvc2CK5UYLXM7dL&opi=89978449. Accessed 03 July 2023.

  30. United States Food and Drug Administration, “Administrative Practices and Procedures; Good Guidance Practices,” Feb. 2000.. [Online]. Available: https://www.federalregister.gov/documents/2000/02/14/00-3344/administrative-practices-and-procedures-good-guidance-practices. Accessed: 01 Feb 2023.

  31. United States Food and Drug Administration, “Format and Content of a REMS Document Guidance for Industry,” 2023. [Online]. Available: https://www.fda.gov/media/77846/download. Accessed: 01 Feb 2023.

  32. United States Food and Drug Administration, “REMS: FDA’s Application of Statutory Factors in Determining When a REMS Is Necessary,” Apr. 2019. [Online]. Available: https://www.fda.gov/media/100307/download. Accessed: Feb. 01, 2023.

  33. United States Food and Drug Administration, “Development of a Shared System REMS Guidance for Industry,” 2018. [Online]. Available: https://www.fda.gov/media/113869/download. Accessed: 01 Feb 2023.

  34. United States Food and Drug Administration, “Use of a Drug Master File for Shared System REMS Submissions Guidance for Industry,” 2017. Accessed: Feb. 01, 2023. [Online]. Available: https://www.fda.gov/media/109124/download. Accessed 01 Feb 2023.

  35. United States Food and Drug Administration, “Evaluating the Effect of the Opioid Analgesics Risk Evaluation and Mitigation Strategy Education Program on Prescribing Behaviors and Patient Outcomes-Exploring the Path Forward for Assessment; Public Workshop; Issues Paper; Request for Comments,” Federal Register, Nov. 05, 2020. https://www.federalregister.gov/documents/2020/11/05/2020-24542/evaluating-the-effect-of-the-opioid-analgesics-risk-evaluation-and-mitigation-strategy-education. Accessed 01 Feb 2023.

  36. Esposito DB, et al. Patient knowledge of safe use of ER/LA opioid analgesics following implementation of the class-wide REMS: a survey study. Patient Prefer Adherence. 2021;15:431–42. https://doi.org/10.2147/PPA.S286935.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Johnson TP. Handbook of health survey methods. John Wiley & Sons, Inc, 2014. Accessed: Feb. 01, 2023. [Online]. Available: 978-1-118-00232-2.

  38. Salas M, et al. Evaluation of patient/caregiver and healthcare provider knowledge, attitudes and behavior for safety and use of pexidartinib. J Comp Eff Res. 2021;10(12):953–67. https://doi.org/10.2217/cer-2020-0253.

    Article  PubMed  Google Scholar 

  39. Salas M, et al. A qualitative evaluation of patient and healthcare provider knowledge, attitudes, and behavior for safety and use of pexidartinib. J Comp Eff Res. 2021;10(3):187–92. https://doi.org/10.2217/cer-2020-0147.

    Article  PubMed  Google Scholar 

  40. Toussi M, Isabelle B, Tcherny-Lessenot S, Voogd H, Dimos V, Kaplan S. Effectiveness of risk minimisation measures for valproate: a cross-sectional survey among physicians in Europe. Pharmacoepidemiol Drug Saf. 2021;30(3):283–91. https://doi.org/10.1002/pds.5119.

    Article  PubMed  Google Scholar 

  41. CIOMS Working Group report, “Patient involvement in the development, regulation and safe use of medicines,” Geneva, Switzerland, 2022. doi: https://doi.org/10.56759/iiew8982

  42. Heads of Medicine Agencies and European Medicines Agency Science Medicines Health, “Guideline on good pharmacovigilance practices (GVP): Module XVI – Risk minimisation measures: selection of tools and 5 effectiveness indicators (Rev 3),” Feb. 2021. [Online]. Available: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/guideline-good-pharmacovigilance-practices-gvp-module-xvi-risk-minimisation-measures-selection-tools_en.pdf. Accessed 01 Feb 2023.

  43. National Institutes of Health, “Clinical Trial E-Protocol Tool and Template Documents,” 2017. https://osp.od.nih.gov/policies/clinical-research#tab1. Accessed 01 Feb 2023.

  44. BassMaloy PJW. How to determine if a project is human subjects research, a quality improvement project, or both. Ochsner J. 2020;20(1):56–61. https://doi.org/10.31486/toj.19.0087.

    Article  Google Scholar 

  45. United States Department of Health and Human Services, “Chart01: Is an Activity Human Subjects Research Covered by 45 CFR Part 46?” [Online]. Available: https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts-2018/index.html#c1. Accessed 03 July 2023.

  46. Quality Improvement Organization Review, 42 CFR Part 476.1 (2023). [Online]. Available: https://www.ecfr.gov/current/title-42/chapter-IV/subchapter-F/part-476. Accessed 03 July 2023.

  47. United States Department of Health and Human Services. “2018 Requirements (2018 Common Rule).” [Online]. Available: https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/revised-common-rule-regulatory-text/index.html. Accessed 03 July 2023.

  48. Kalkbrenner M. A practical guide to instrument development and score validation in the social sciences: the MEASURE approach. Pract Assess Res Eval. 2021. https://doi.org/10.7275/svg4-e671.

    Article  Google Scholar 

  49. Federal Food, Drug, and Cosmetic Act Subchapter V-Drugs and Devices Part A-Drugs and Devices. United States, 2007. [Online]. Available: https://uscode.house.gov/view.xhtml?req=%28title:21%20section:355-1%20edition:prelim%29%20OR%20%28granuleid:USC-prelim-title21-section355-1%29&f=treesort&edition=prelim&num=0&jumpTo=true#sourcecredit. Accessed 01 Feb 2023.

Download references

Acknowledgments

We are grateful to FDA staff Brian Gordon, Shelly Harris, and Derrick Beasley for their contributions to the study. We also thank Drs Claudia Manzo and Cara Dooley for their review and revisions to the text. The findings in this work is the sole responsibility of the authors and does not represent the views, opinions, and recommendations from the United States Food and Drug Administration.

Author information

Authors and Affiliations

  1. Food and Drug Administration, 10903 New Hampshire Avenue, Building 22, Room 1106, Silver Spring, MD, 20993, USA

    Gita A. Toyserkani, Samuel B. Ewusie & Janifer Quick

  2. Oak Ridge Institute for Science and Education (ORISE) Program, Oak Ridge, TN, USA

    Samuel B. Ewusie & Janifer Quick

  3. University of Mississippi Medical Center, Jackson, MS, USA

    Philip Turk

  4. Parkinson School of Health Sciences and Public Health, Loyola University Chicago, Chicago, IL, USA

    Elaine H. Morrato

Authors
  1. Gita A. Toyserkani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Samuel B. Ewusie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Philip Turk
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Janifer Quick
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Elaine H. Morrato
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gita A. Toyserkani.

Ethics declarations

Funding

SE and JQ received financial support from the Oak Ridge Institute for Science and Education (ORISE) through an interagency agreement between the U.S. Department of Energy (DOE) and FDA as part of a Research Participation Program at the U.S. Food and Drug Administration (FDA). EM received financial support from the U.S. Food and Drug Administration, Office of Surveillance and Epidemiology, administered through the University of Colorado, Denver.

Conflict of Interest

SE is currently an employee of Bristol Myers Squibb. EM has received funding for consulting services from United Healthcare Services, Eli Lilly, and Reatta Pharmaceuticals. JQ is currently an employee of AstraZeneca. GT is employed by the U.S. Food and Drug Administration. The other authors declare no conflicts of interest.

Availability of Data and Materials

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Ethics Approval

Not applicable.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Code Availability

Not applicable.

Authors’ Contributions

GT contributed to the study conception, design, data analysis, data interpretations, and helped critically revise the manuscript. JQ initially developed the study protocol, analysis tool, data structure, and data collection. SE developed, collected, analyzed, and interpreted data resulting from the content analysis, helped write the first draft of the manuscript, and presented in part preliminary findings at the 2021 ICPE All Access. PT made significant contributions to the statistical plan, data analysis, data interpretation, and helped critically revise the manuscript. EM provided critical intellectual direction on the design of the study, data interpretation, and critically revised the manuscript. All authors read and approved the final version.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 337 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Toyserkani, G.A., Ewusie, S.B., Turk, P. et al. Evolution of Cross-Sectional Survey Protocol Quality Over Time: A Case Series of Index U.S. REMS Knowledge Survey Protocols (2007–2020). Drug Saf 46, 1073–1087 (2023). https://doi.org/10.1007/s40264-023-01344-x

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40264-023-01344-x

Search

Navigation