Skip to main content
Log in

Optimization of alert notifications in electronic patient-reported outcome (ePRO) remote symptom monitoring systems (AFT-39)

  • Published:
Quality of Life Research Aims and scope Submit manuscript



Clinical benefits result from electronic patient-reported outcome (ePRO) systems that enable remote symptom monitoring. Although clinically useful, real-time alert notifications for severe or worsening symptoms can overburden nurses. Thus, we aimed to algorithmically identify likely non-urgent alerts that could be suppressed.


We evaluated alerts from the PRO-TECT trial (Alliance AFT-39) in which oncology practices implemented remote symptom monitoring. Patients completed weekly at-home ePRO symptom surveys, and nurses received real-time alert notifications for severe or worsening symptoms. During parts of the trial, patients and nurses each indicated whether alerts were urgent or could wait until the next visit. We developed an algorithm for suppressing alerts based on patient assessment of urgency and model-based predictions of nurse assessment of urgency.


593 patients participated (median age = 64 years, 61% female, 80% white, 10% reported never using computers/tablets/smartphones). Patients completed 91% of expected weekly surveys. 34% of surveys generated an alert, and 59% of alerts prompted immediate nurse actions. Patients considered 10% of alerts urgent. Of the remaining cases, nurses considered alerts urgent more often when patients reported any worsening symptom compared to the prior week (33% of alerts with versus 26% without any worsening symptom, p = 0.009). The algorithm identified 38% of alerts as likely non-urgent that could be suppressed with acceptable discrimination (sensitivity = 80%, 95% CI [76%, 84%]; specificity = 52%, 95% CI [49%, 55%]).


An algorithm can identify remote symptom monitoring alerts likely to be considered non-urgent by nurses, and may assist in fostering nurse acceptance and implementation feasibility of ePRO systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data availability

Investigators can submit a request for data from this trial by completing an Alliance Data Sharing Request Form, found at, and submitting it to


  1. Cleeland, C. S., Zhao, F., Chang, V. T., et al. (2013). The symptom burden of cancer: Evidence for a core set of cancer-related and treatment-related symptoms from the Eastern Cooperative Oncology Group Symptom Outcomes and Practice Patterns study. Cancer, 119(24), 4333–4340.

    Article  PubMed  Google Scholar 

  2. Reilly, C. M., Bruner, D. W., Mitchell, S. A., et al. (2013). A literature synthesis of symptom prevalence and severity in persons receiving active cancer treatment. Supportive Care in Cancer, 21(6), 1525–1550.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Panattoni, L., Fedorenko, C., Greenwood-Hickman, M. A., et al. (2018). Characterizing potentially preventable cancer- and chronic disease-related emergency department use in the year after treatment initiation: A regional study. Journal of Oncology Practice/American Society of Clinical Oncology, 14(3), e176–e185.

    Article  Google Scholar 

  4. Henry, D. H., Viswanathan, H. N., Elkin, E. P., et al. (2008). Symptoms and treatment burden associated with cancer treatment: Results from a cross-sectional national survey in the US. Supportive Care in Cancer, 16(7), 791–801.

    Article  PubMed  Google Scholar 

  5. Schmidt, T., Valuck, T., Perkins, B., et al. (2021). Improving patient-reported measures in oncology: A payer call to action. Journal of Managed Care and Specialty Pharmacy, 27(1), 118–126.

    Article  PubMed  Google Scholar 

  6. Centers for Medicare & Medicaid Services (CMS). Enhancing oncology model.

  7. Basch, E., Schrag, D., Henson, S., et al. (2022). Effect of electronic symptom monitoring on patient-reported outcomes among patients with metastatic cancer: A randomized clinical trial. JAMA, 327(24), 2413–2422.

    Article  PubMed  PubMed Central  Google Scholar 

  8. National Cancer Institute. Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events.

  9. Dueck, A. C., Mendoza, T. R., Mitchell, S. A., et al. (2015). Validity and reliability of the US National Cancer Institute’s Patient-Reported Outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE). JAMA Oncology, 1(8), 1051–1059.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Al-Rashdan, A., Sutradhar, R., Nazeri-Rad, N., et al. (2021). Comparing the ability of physician-reported versus patient-reported performance status to predict survival in a population-based cohort of newly diagnosed cancer patients. Clinical Oncology (Royal College of Radiologists), 33(7), 476–482.

    Article  CAS  Google Scholar 

  11. Stover, A. M., Tompkins Stricker, C., Hammelef, K., et al. (2019). Using stakeholder engagement to overcome barriers to implementing patient-reported outcomes (PROs) in cancer care delivery: Approaches from 3 prospective studies. Medical Care, 57(5), S92–S99.

    Article  PubMed  Google Scholar 

  12. Kiernan, K. (2018). Insights into using the GLIMMIX procedure to model categorical outcomes with random effects. SAS Technical Papers, 2018, 1–23.

    Google Scholar 

  13. Basch, E., Stover, A. M., Schrag, D., et al. (2020). Clinical utility and user perceptions of a digital system for electronic patient-reported symptom monitoring during routine cancer care: Findings from the PRO-TECT trial. JCO Clinical Cancer Informatics, 4, 947–957.

    Article  PubMed  Google Scholar 

  14. Mandrekar, J. N. (2010). Receiver operating characteristic curve in diagnostic test assessment. Journal of Thoracic Oncology, 5(9), 1315–1316.

    Article  PubMed  Google Scholar 

Download references


The sponsor, Alliance Foundation Trials, LLC, coordinated contracting with participating study practice sites and submission of regulatory documentation. The sponsor was not involved in the design or conduct of the study; collection, analysis, or interpretation of the data; preparation of the manuscript; or decision to submit the manuscript for publication. The Alliance Publications Committee reviewed and approved this manuscript for publication and did not have the right to veto publication or to control the decision regarding to which journal the manuscript was submitted. All statements in this publication, including its findings, are solely those of the authors and do not necessarily represent the views of PCORI or its Board of Governors or Methodology Committee. Dr. Mazza presented results from this article at the International Society for Quality of Life Research Conference in Calgary, AB, Canada on October 20, 2023.


This work was funded by the Patient-Centered Outcomes Research Institute (PCORI) award IHS-1511-33392. The trial made use of technology systems provided by the Patient-Reported Outcomes Core (PRO Core; at the Lineberger Comprehensive Cancer Center of the University of North Carolina, funded by NCI Cancer Center Core Support grant P30CA016086 and the University Cancer Research Fund of North Carolina. The trial was sponsored by Alliance Foundation Trials, LLC (

Author information

Authors and Affiliations



GLM and EB formulated the research question and designed the study. EB and ACD provided supervision for the research activity planning and execution. BG, ACD, and GLM prepared the data for analysis. GLM and BG analyzed the data. GLM led the writing of the manuscript. GLM created the tables and figures. GLM, ACD, BG, JJ, AMD, PC, VSB, GT, MJ, LJR, DS, and EB contributed to the conduct and primary analysis of the PRO-TECT trial. All authors interpreted the results as well as reviewed and approved the manuscript for publication.

Corresponding author

Correspondence to Gina L. Mazza.

Ethics declarations

Conflict of interest

Dr. Basch reported receipt of personal fees from Navigating Cancer, Sivan, AstraZeneca, Resilience, and the Research Triangle Institute for serving as a scientific advisor; receipt of research funding from the National Cancer Institute (NCI); being owner of Carolina Informatics; and employment by the University of North Carolina. Dr. Schrag reported receipt of personal fees from Pfizer, nonfinancial support from Grail for serving as a site principal investigator, and grants from the American Association for Cancer Research awarded to Memorial Sloan Kettering Cancer Center. Dr. Mody reported consultancy and receipt of research funding from Sivan. All remaining authors have declared no conflicts of interest.

Consent to participate

Informed consent was obtained from all participants included in the PRO-TECT trial.

Ethics approval

The PRO-TECT trial was conducted in accordance with the Declaration of Helsinki and Good Clinical Practice guidelines. The trial’s protocol was approved by the central institutional review board and the institutional review board at each participating site.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (DOCX 233 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mazza, G.L., Dueck, A.C., Ginos, B. et al. Optimization of alert notifications in electronic patient-reported outcome (ePRO) remote symptom monitoring systems (AFT-39). Qual Life Res (2024).

Download citation

  • Accepted:

  • Published:

  • DOI: