Background

Patients with COVID-19 (coronavirus disease) can experience rapid and unpredictable clinical deterioration. This concern was heightened in the early months of the pandemic when the clinical course of COVID-19 was unknown, treatment was entirely supportive, and vaccines were unavailable. Simultaneously, office-based outpatient practices largely used telephone or telemedicine encounters to provide guidance for providing reassurance or managing symptoms at home or directing patients with concerning symptoms to acute care settings. To manage the large volumes of encounters, particularly during high community case counts, several health systems developed remote patient monitoring programs to support home dwelling COVID-19 infected patients [1,2,3,4,5,6,7,8,9].

Our 6-hospital health system with over 500 outpatient practices enrolled adult patients with test-confirmed COVID-19 or symptoms of COVID-19 in COVID Watch, a 14-day remote patient monitoring program which resulted in lower patient mortality compared to matched control patients not enrolled in the program [7, 10]. While the program was clinically effective, we elected to ask patients and clinicians how it could be improved. Their perspectives could lead to improvements in this program or in remote engagement programs more generally [11,12,13,14,15].

This study investigates the perspectives of patients and clinician groups who interacted with COVID Watch: (1) patients enrolled in the program, (2) primary care and Emergency Department (ED) clinicians who enrolled patients in or had their COVID-19 patients managed by COVID Watch; and (3) administrators in primary care or the Emergency Department. The study’s aim was to understand patients’ and clinicians’ experiences interacting with the COVID Watch program, how the program could be improved, and the lessons from COVID Watch that could be extended to the design and implementation of future remote patient monitoring programs.

Methods

COVID Watch Overview

Patients were enrolled in COVID Watch by outpatient or ED clinicians through an application embedded in Penn Medicine’s electronic health record. Patients were additionally offered COVID Watch via an automated text message if they received a positive result from a COVID-19 test conducted by a Penn Medicine laboratory.

Once enrolled, COVID Watch sent twice daily text messages in English or Spanish that asked “How are you feeling compared to 12 hours ago: better, same, or worse?” Patients who replied “worse” were subsequently asked, “Is it harder than usual for you to breathe: yes or no?”. Patients who responded “yes” generated an electronic health record (EHR) alert, monitored by a team of telemedicine clinicians (nurses, advanced practice providers, and physicians) available 24/7, who contacted the patient within one hour. Interpretation services were used when needed. Patients could also text the word “worse” at any time to connect with a clinician, also within one hour. During the study period, some participants were automatically randomized to receive a home fingertip pulse oximeter or not [10]. Those who received a pulse oximeter were asked to report their symptoms and their ambulatory oxygen saturation levels. Patients whose oxygen saturation, SpO2, was below 90% or had decreased by more than 3% and below 95%, were called by the clinical staff who triaged the patient using standardized protocols. The program was free and did not require a patient to have established care with a Penn Medicine clinician (e.g., primary care) or insurance. We describe the COVID Watch interventions, including the pulse oximeter trial, in previous publications [7, 10, 16].

Study Participants and Setting

We conducted semi-structured interviews with three groups of people who interacted with COVID Watch: (a) patients who had been enrolled in COVID Watch within the prior 90 days, (b) primary care and ED clinicians who directly enrolled patients in COVID Watch or had their patients enrolled into COVID Watch, and (c) administrators in primary care or the ED. Ethical approval was provided by the Institutional Review Board at the University of Pennsylvania.

Patient Recruitment and Sampling Strategy

Patients were recruited via phone between February and June 2021, a timespan that included the randomized controlled trial of fingertip pulse oximetry among COVID Watch enrolled patients [10]. Patients were purposively sampled across two patient-level strata to gain a diversity of patient perspectives on their COVID Watch experience: (a) having a pulse oximetry device mailed to their house (or not), and (b) level of engagement in the program (high vs. low, defined as responding to twice-daily, automated text message prompts at least 10 out of 14 days of enrollment versus fewer). The cutoffs were based on the median level of engagement. We monitored and recruited patients to attempt balance across racial, ethnic, and language sub-groups. Patients provided verbal informed consent were compensated $50 USD (United States dollar) for their time.

Clinician and Administrator Recruitment and Sampling Strategy

Clinicians (e.g., physicians, nurse practitioners, and physician assistants) and administrators (e.g., medical directors, physician leads, or non-clinical practice managers) were recruited using emails between July and November 2021. Clinicians and administrators were purposively sampled across two health system level strata with a goal of gaining a diversity of clinician perspectives of COVID Watch: (a) clinicians who primarily worked in an ED setting vs primary care setting, and (b) clinicians that enrolled a high versus low number of patients in COVID Watch. For both settings, high enrolling clinicians were defined as those who enrolled > 15 patients and low enrollers were defined as less than 10 patients. These cutoffs were based on the median level of enrollment. Administrators were recruited from the clinical sites of the clinicians. Snowballing techniques were used to identify additional administrators who might have been influential in encouraging clinicians or staff members to enroll patients in COVID Watch. Clinicians and administrators provided verbal informed consent and were compensated $50 USD for their time. Interviews were conducted before the publication of COVID Watch’s evaluation [7, 10].

Interview Guide Development

We created three semi-structured, open-ended interview guides (see Table 1, Supplementary 1, Supplementary 2, Supplementary 3) for each cohort: patients, clinicians, and administrators. Guides were created by core members of the research team (KHC, JAS, MW, MN, JR, MKD, AF, SK, MG, AUM) and reviewed by the larger team. They were pilot tested with at least two participants in each cohort. Questions were open-ended and included follow-up probes to allow participants to expand upon answering. After the interview, participants self-reported sociodemographics.

Table 1 Patient and Clinician Interview Question Examples According to Themes in Participant Responses
Full size table

Data Collection and Analysis

Researchers (MW, MN, and ZB) conducted audio-recorded phone interviews in English or Spanish. Verbal informed consent was obtained prior to all interviews. English and Spanish audio recordings were transcribed by Datagain Services (Seacaucus, NJ), with the Spanish audio transcribed into English. Transcripts were then entered into NVivo 1.5 (QSR International) for coding and analysis.

Separately for patients, clinicians, and administrators, early interview transcripts were used to develop an initial codebook using a modified content analysis [17, 18] approach that relied upon the structure of the interview guide but allowed for emergent themes. The codebook was applied to all transcripts. Recruitment and emergent findings were reviewed in bi-weekly team meetings. We conducted interviews until members of the study team (KHC, JAS, AUM) concluded that we had achieved thematic saturation within groups (e.g., patients, clinicians, and administrators). The achieved inter-rater reliability was ĸ=0.81 across co-coded transcripts. Patient interviews lasted an average of 53 min (range 37–74 min) and clinician and administrator interviews lasted an average of 36 min (range 23–54 min).

Results

Participant Characteristics

In total, 85 interviews were completed. Forty-seven patients were interviewed, who were on average 50 years old and mostly female, White, non-Hispanic, and English speaking (see Table 1). Because most administrators were clinicians and no major thematic differences were identified between clinicians and administrators, we combine these groups for presentation and refer to both groups as “clinicians” henceforth. The sample of 38 clinicians was primarily female, White, non-Hispanic, and physicians, and had been in practice for 11 years or more (see Table 2). Across all cohorts, themes aligned into three categories: (1) sentiments about COVID Watch, (2) feedback for improving COVID Watch and only asked of clinicians, (3) lessons learned from COVID Watch that have implications for future remote patient monitoring programs (see Table 3). There was no notable thematic difference by level of patient engagement or clinicians’ predilection (high vs. low) for enrolling patients into COVID Watch, therefore themes are aggregated across strata. Results specific to patients with pulse oximetry devices and clinicians’ practice setting (ED vs. primary care) are noted below. Quotes to illustrate each theme are presented in Table 4 for patients and Table 5 for clinicians.

Table 2 Patient Characteristics
Full size table
Table 3 Clinician and Administrator Characteristics
Full size table
Table 4 Summary of Patient Themes and Illustrative Interview Excerpts
Full size table
Table 5 Summary of Clinician Themes and Illustrative Interview Excerpts
Full size table

Sentiments About COVID Watch

Patient Perspectives

Comforting

Most patients described text messages as a comforting reminder that healthcare professionals were monitoring their well-being. COVID Watch was viewed as a positive alternative to being admitted and monitored in a hospital inpatient setting. Even among individuals who never required the support of a clinician, being able to contact a clinician or escalate their care while they had COVID-19 provided peace of mind. Patients also appreciated that the regular text messages helped them monitor their symptoms over time and track when their symptoms were improving.

Irritating

Some patients felt that the text messages were excessive, intrusive, or annoying. These feelings were common for patients at the ends of the illness spectrum, either with mild to no symptoms, or conversely so ill they did not have the energy to respond to messages in a timely manner. Those who had a low response rate to the twice daily messages were often patients who felt the text messages were unhelpful.

Insufficient

Some patients who were worried about their COVID-19 symptoms or those who were fearful about the risk of severe illness expressed a desire for real-time support from human clinicians instead of the automated, routine text messages provided by COVID Watch. Some patients wanted to report additional symptoms beyond feeling short of breath.

Clinician Perspectives

Comforting

ED clinicians specifically described taking comfort in knowing that COVID Watch could monitor patients discharged home. Many described it as a valuable “safety net” for patients especially when COVID-19 was a new illness. This feeling of a safety net was particularly true in ED clinicians’ discussions of patients with significant social needs (e.g., without a primary care provider, socially isolated) and patients who did not meet admission criteria but whom they worried might decompensate at home after being discharged from the ED.

ED and primary care clinicians believed that COVID Watch also gave their patients a sense of comfort. They knew the program would monitor them and provided an alternative to the ED as the sole source of COVID-19 care.

Some clinicians, both ED and primary care, saw the provision of pulse oximetry devices to use at home as an important way to give clinicians and patients additional data points about the severity of a patient’s illness. For example, one ED clinician described being able to trust a patient’s report of dyspnea more if the patient used a pulse oximeter at home.

ED clinicians specifically did not believe that the ability to enroll patients in the program influenced their decisions to admit versus discharge, a sentiment that was independent of knowing that COVID Watch could monitor patients discharged home.

Increased Access to Care for Patients

COVID Watch’s accessibility was seen as a key benefit to the program. Patients had quick and easy access to a clinician if needed; patients were not alone in their health decision-making; the program alleviated patient fear of the unknown; and the program was free of charge. Additionally, a few clinicians reported COVID Watch increased access for their Spanish-speaking patients, which was described as a key need at some sites.

Reduced Follow-up Burden

Many primary care clinicians found COVID Watch to be an important tool for managing follow-up care when the volume of patients’ needs was high. They perceived it reduced the volume of patients contacting them about worrisome symptoms by rerouting calls to a centralized team of clinicians, offloading COVID-19 concerns during surges in COVID-19 cases.

Feedback for Improving COVID Watch

Patient Perspectives

Improve the Enrollment Process

Some patients did not recall when or how they were enrolled in COVID Watch. Patients who knew they were enrolled tended to describe more positive feelings about starting the program. 

A misconception about the program was that some patients thought it was their own doctor who had enrolled them in COVID Watch and was personally monitoring their symptoms.

Clarify the Monitoring and Escalation Process

Many patients desired clearer, more concise information about what symptoms would result in clinical escalations and found the subjective nature of the daily text message (e.g., “Are your symptoms the same, better, or worse than 12 hours ago”) to be challenging. Some patients desired more quantitative measures such as a 0–10 number scale for their dyspnea. This desire for quantitative measures was also reflected in patients’ positive reception to the pulse oximetry device. The device provided an objective measure that enabled most patients to feel more confident about their clinical course.

Some patients expressed a personal preference for phone calls over text messages or a desire for phone call options to be available for other patients, referencing the difficulty that older patients can have with texting, or not having phones that are equipped for text messaging. Other patients thought phone calls would be preferable because they would give clinicians more clarity about how patients are feeling.

Spanish-speaking patients more often felt COVID Watch was not able to fully meet their needs because the format of communication was not concordant with their cultural expectation. Some Spanish speakers expressed how Spanish-speaking cultures tended to be more phone-call oriented, so an option to choose the modality of the messages may provide a better cultural fit for some.

Clinician Perspectives

Improve the Enrollment Process

Clinicians tended to describe the process of enrolling into COVID Watch as relatively easy, but there was a desire to make enrollment even easier. Some believed it was tedious to, for example, go into the patient’s exam room to ensure patients received the program’s initial text message or to ensure the patient’s phone number was correct. In addition, some clinicians felt that their own familiarity with the enrollment process waned if they had not enrolled a patient recently. Some also expressed uncertainty about the program’s details such as the ability to enroll patients over the weekends.

Provide Solutions for Patients With Limited Device Access or Hesitancy

Clinicians highlighted barriers related to patients’ accessing COVID Watch and hesitancy to use their phones for engaging in care. Participation required the ability to use a cell phone with text messaging. Access to the required technology was particularly challenging for elderly patients (especially those who lived alone) and patients experiencing homelessness.

Address Low-literacy and Language Preferences Among Patients

A key access-related barrier clinicians discussed was that the program required patients to have comfort in reading and writing in English or Spanish; other languages should be considered. Other enhancements for accessibility included offering an option for patients to use a landline; offering access to a central hotline phone number those patients could call; or distributing cell phones for patients to use. Finally, some clinicians recommended that patients be given the ability to enroll themselves.

Create a Feedback Loop for Clinicians

ED and primary care clinicians discussed a desire to know the clinical course of their patients after enrollment in COVID Watch and were interested in knowing which patients did not escalate, those who escalated to a COVID Watch nurse, or those who unenrolled from the program early. This would serve as a mechanism to inform clinicians about the quality of their care, particularly if enrolling in COVID Watch was effective, and a reminder that the COVID Watch program was still enrolling patients.

Lessons for Future of Remote Patient Monitoring Programs from Clinicians

Enhanced Data Collection

Clinicians felt that remote patient monitoring programs will be an important part of practicing medicine in the future. However, data collected should have concrete benchmarks. Clinicians were hesitant about COVID Watch’s subjective self-reports and felt more objective measures should be used in future programs, COVID-19 related or otherwise.

Both ED and primary care clinicians felt remote patient monitoring programs should provide patients with the appropriate health data collection tools, like a home pulse oximeter or blood pressure cuff, to collect and report data back to their healthcare team. Some also suggested greater integration with existing electronic health records, directly embedding remotely recorded results into the medical record.

A Guide for Patients

ED clinicians felt that remote patient monitoring could reduce the number of ED visits by giving patients more accurate, objective data about when not to come to the ED. By using objective data and clear cutoffs, patients could be clearly guided to seek the right level of care. In addition, the ability to provide reassurance to patients with a remote monitoring program might help patients being discharged home from the ED.

Extend Remote Patient Monitoring to Non-COVID-19 Conditions

Many clinicians also felt remote patient monitoring will be particularly valuable for certain chronic and acute conditions. For example, clinicians tended to perceive targeted data collection to be practical and effective for monitoring and treating conditions like congestive heart failure, diabetes, asthma, weight management, and post-surgery recovery. To evolve for other use cases, however, clinicians emphasized that remote patient monitoring tools needed to be easy to use and equitable for patients, and clinicians must be confident in the quality of the data collected.

Discussion

Overall, while patients and clinicians found COVID Watch to be comforting and beneficial, improvements to the design and implementation of the program will be important for the program’s future and have implications for the design of future remote patient monitoring programs.

COVID Watch often provided a sense of comfort and reassurance to both patients and clinicians. This is a shared sentiment across prior qualitative evaluations of remote patient monitoring programs developed for COVID-19 [19, 20]. For COVID Watch specifically, the increased access to care due to its free cost and Spanish-language availability were vital. Despite their enthusiasm, patients and clinicians desired a better user experience, at times sharing common suggestions for the program. Patients wanted more clarity about when and how they were enrolled in the program. The confusion over enrollment may have contributed to lower engagement in the program and may have lowered perceptions of its utility. Both patients and clinicians desired a guide for patients, indicating when care would likely be escalated or when staying at home would be safe. Another suggestion from both patients and clinicians was a desire for quantitative data that would provide a threshold for when to escalate care. In addition, similar to other remote patient monitoring programs for COVID-19, [19] patients and clinicians thought it would be beneficial to monitor symptoms other than breathing. Patients and clinicians also desired the program to build in flexibility, expressing concerns about the universal use of automated text messages for patients. For example, the program may have had limited uptake among patients who had reticence about using technology to communicate their health needs, limited English or Spanish literacy, or preferred additional languages. These subgroups are among those who are historically at increased risk for limited access to care and experienced worse COVID-19 outcomes [21].

Our findings have generated three key insights for future remote patient monitoring programs to manage COVID-19 or other clinical conditions. First, remote patient monitoring programs should not be static, one-time builds or implementations. While these programs may have automation or use digitized algorithms, they are human facing programs that should evolve as technology advances, patient and clinician expectations of technology evolve, and standards for managing targeted disease conditions change. Developing systems for monitoring program performance and patient engagement and seeking patient and clinician feedback to continuously refine these programs are as important as achieving intended health outcomes.

Second, health systems must acknowledge the human resources needed to support remote patient monitoring programs, even if automation is embedded in the program. While automation can improve efficiency for some patients, successful programs will need to marry technology with options for human interaction. Tech-first approaches may not always be welcomed. Some patients in our study indicated the desire to connect with a human clinician, wanting to avoid the automated text message system more generally. Yet, at the same time, automation allowed for a team of 3–4 nurses to simultaneously manage efficiently and effectively over one thousand patients during the staffing shortages of the pandemic, [7] a challenge projected to remain over the coming years [22, 23]. Using default pathways (e.g., text messages) that are automated, complemented by alternative pathways (e.g., interactive voice recordings or human-to-human phone calls) that are customized to the user’s needs, may be one solution for greater engagement while not overburdening current clinicians.

Finally, future programs must be designed with equity as a primary principle, recognizing patients who have the most limited access to care may need additional design considerations. For example, programs should be offered in multiple languages. In our study both patients and clinicians expressed concerns that vulnerable populations may have been excluded because more direct human-to-human connections (e.g., telephone calls) were not made available. Developing programs that correctly balance patients’ desires for human-to-human connections, promoting inclusivity, with the efficiency gains of automated processes will be important for future remote patient monitoring programs.

This study has limitations. Our analysis took place in one large academic institution. However, the institution includes six hospitals and over 500 outpatient practices across a wide geographic area, allowing us to sample participants from multiple hospitals, encompassing urban and suburban settings across the large catchment area. These interviews took place relatively early in the pandemic – within the first year – and therefore reflect the stress that both patients and clinicians felt when faced with an unprecedented crisis. Finally, patients’ and clinicians’ experiences with COVID Watch and COVID-19 evolved during the study period such as the implementation of automated and opt-out enrollment in the fall of 2020. In addition, surges of infection, the increased availability of vaccination, and effective treatments might have influenced our participants’ responses.

Remote patient monitoring programs are increasingly available and are being tested for a variety of clinical conditions including hypertension management, in-home administration of chemotherapy, and transitions between hospital and home. In addition, success of these programs hinges on user-centered design to enhance experiences for both patients and clinicians [24] as well as intentional design for marginalized groups who have not historically been considered as early adopters of new technology-based care programs [25,26,27]. Understanding how a diverse group of patients engage with and experience remote patient monitoring programs, and how clinicians integrate them into daily workflow and clinical decision-making will be informative for future remote patient monitoring programs.