Background

The COVID-19 pandemic dramatically increased the demand for telemedicine and telehealth services [1,2,3], including in outpatient settings to support patients with confirmed or suspected COVID-19 [4]. Several health systems have introduced remote patient monitoring (RPM) programs as a vital adjunct to in-person COVID-19 clinical care. Such programs seek to guide patients to the appropriate level of care, whilst minimizing unnecessary pathogen exposure to clinical staff and non-COVID-19 patients [5,6,7,8,9,10,11,12].

RPM directs patient data collected in non-clinical settings to clinical teams providing care oversight [13, 14]. It has traditionally been used to guide care for patients with chronic disease [14,15,16,17,18,19,20,21], and its impact may depend on the disease and monitoring approach [15, 22]. Outside of chronic disease, the use of RPM to manage short-term conditions has been primarily limited to post-operative care [14, 23]. Prior to COVID-19, its use in acute infectious illness was virtually non-existent.

COVID-19 expanded RPM possibilities from both the clinical [5,6,7,8,9, 24] and technical [25, 26] perspectives. Recent RPM efforts have combined digital and/or sensor technology (i.e., pulse oximetry) with clinical oversight to determine the need for care escalation. Evaluations of these efforts thus far are limited. Most notably, RPM enrollment reduced 30 day hospital readmissions in COVID-19 patients discharged from the hospital [11, 12]; similar findings were also noted for cancer patients with COVID-19 [27]. Another evaluation found an association between RPM enrollment and a reduced admittance rate following an ED visit [5]. Patient engagement and satisfaction with programs have also been documented [5,6,7].

This nascent body of work suggests that RPM programs are acceptable to patients and may decrease undesirable healthcare utilization, though additional evaluation is needed to understand how perceived benefits amongst stakeholders compare with actual care utilization and patient outcomes. Such analysis can inform the future direction of such programs, particularly given the non-trivial resources they require. We evaluated a specialized COVID-19 clinic with an integrated RPM program in an academic medical center using a mixed-methods approach. This approach included quantifying the program’s impact on acute care utilization and patient outcomes as well as qualitatively analyzing patient, provider, and administrator perspectives of the program.

Methods

Design

We conducted a convergent mixed methods [28] evaluation of a dedicated COVID-19 outpatient clinic at a large academic medical center, Stanford Health Care (Palo Alto, CA, USA). A quantitative retrospective cohort analysis matched patients enrolled and unenrolled in the clinic to understand the clinic’s impact on downstream emergency department (ED) utilization and hospitalization rates. Semi-structured interviews with patients, frontline clinicians, and administrative stakeholders and subsequent analysis were guided by Proctor et al. (2011) Outcomes for Implementation Research [29]. The Stanford Institutional Review Board approved the present retrospective evaluation and determined it did not meet the definition of human subjects research (Protocol #56054). Informed consent was obtained from all interview participants.

Specialized COVID-19 “CROWN” clinic with RPM

In April 2020, Stanford Health Care (SHC) launched a specialized COVID-19 clinic called CROWN (standing for Care and Respiratory Observation of patients With Novel coronavirus) to provide RPM to adult outpatients recently diagnosed with COVID-19 [30]; this clinic combined with RPM services entail the intervention that is the focus of this evaluation. Any positive COVID-19 laboratory test throughout the health system went into a central pool monitored by ED nurses. Patients with a positive COVID-19 test and fewer than 14 days of symptoms were offered enrollment in the program that included periodic check-ins by phone and/or video. The frequency and method of communication (phone or video) was based on a risk stratification tool developed by lead clinicians, which incorporated age, pre-existing conditions, clinical severity of illness and the clinical course of disease to place patients into low, medium and high risk categories (Additional file 1: Appendix A1). These tiers were used to guide the frequency of outreach through phone calls and video visits as well as the distribution of pulse oximeters to medium and high-risk patients (Additional file 1: Appendix A2). Clinicians at any point could designate a patient receive a higher level of care (e.g. shorter intervals between outreach), including escalation to in-person care at CROWN or the ED if needed. CROWN clinicians followed patients from the date of their enrollment to 14–21 days from first symptoms and provided patients a dedicated phone number to contact the clinic if concerns or questions arose. Biometric data was shared verbally by the patient in the course of each encounter and was not electronically transmitted. Further details regarding the risk stratification tool and RPM outreach protocol have been previously described [30].

Quantitative evaluation - creation of the matched retrospective cohort

We extracted electronic health record demographic and acute care utilization data of all patients who received a positive test result from a Stanford testing facility between May 1, 2020 and September 30, 2020. The time between the COVID-19 test result date and enrollment varied, with a median of 2 days. To ensure a similar duration of disease progression for the purposes of matching, a “pseudo enrollment” date defined as 2 days following a COVID-19 test result was assigned to the unenrolled comparator group. Patients were excluded if they were under age 18, lived more than 50 miles from Stanford Hospital or had a missing zip code, or if their positive test date was after September 2, 2020, to ensure complete follow-up of all patients; follow-up was defined as 28 days following the date of program enrollment (or pseudo-enrollment). Patients were also excluded if they were hospitalized in the 7 days preceding and inclusive of the enrollment (or pseudo-enrollment) date, a decision made to focus the evaluation on the intervention’s effectiveness at directing patients to the appropriate level of care early in their disease course.

Manual chart reviews were conducted by physician authors (with regular audits by author SV). These were used to identify the binary presence of comorbidities based on the COVID-19 risk stratification protocol (Additional file 1: Appendix A) in addition to emergency room and hospitalization encounters in the exclusion and observation periods that were not captured in the extracted dataset but were viewable within the electronic health record (Epic, Wisconsin, USA).

The matched cohort was finalized by matching each eligible enrolled patient to an unenrolled patient considering home distance from hospital (0–15, > 15–30, > 30–50 miles), health system affiliation (Stanford academic, Stanford non-academic made up of affiliated community practices, and unaffiliated), insurance type (private, Medicare, other) and race/ethnicity (Hispanic, White, Asian, other), age (closest match), and binary presence of relevant comorbidities (technical details in Additional file 1: Appendix B).

Quantitative data analysis

The outcomes of interest for the matched analysis were COVID-related unique patient and total ED encounters, unique patient and total hospital admissions (inclusive of an admission to observation status), and death in the 28 days following the date of program enrollment (or pseudo-enrollment). The ratio of a positive admission following an ED visit was reported for each group (defined as # inpatient admissions / (# inpatient admissions + # ED encounters)); these rates were statistically compared using a mixed effects logistic regression model with a random effect for patient. We also compared rates of COVID-related ED and hospital admissions by risk severity level for enrolled patients as an informal validation of the clinical risk assignment protocols. Analyses were conducted using R version 4.0.5 software and SAS version 9.4 software. Statistical significance was set at 0.05; p-values were adjusted for multiple comparisons using Tukey-Kramer correction where appropriate. Poisson regressions were used to determine statistical significance for counted events, including to validate the risk stratification tool where risk level was an explanatory variable and COVID-related ED and hospital admissions were outcomes.

We conducted a detailed chart review to extract clinical details of all COVID-19-related ED encounters and hospitalizations to provide a descriptive analysis only, as our evaluation was not powered to detect changed at this level. Data extracted included chief complaint, referring party (self, clinic, other), and suboptimal oxygenation (defined as a pulse oximeter reading of 95% or below at first clinical presentation) and are described as a percentage of a given event type. Given ever-shifting COVID-19 treatment guidelines at the time, for the purposes of analysis and discussion, we relied on a crude definition of “appropriate care” in which a hospital admission followed an ED encounter.

Qualitative stakeholder groups & data collection

We interviewed clinic stakeholders including enrolled patients, providers, and administrators. Interviews with unenrolled patients were not pursued given their experiences have been described elsewhere [31, 32], as well as our need to direct limited resources to better understand the clinic’s implications for the purposes of quality improvement. Enrolled patients were selected to include a diverse representation of gender, primary language spoken, and COVID-19 risk severity level from a list of those who completed clinic enrollment within the previous 3 weeks with no prior hospitalization for COVID-19. Patients were approached by telephone, given a verbal description of the evaluation, and gave their informed consent to participate in the interview. All providers who made up the core clinical team were contacted for interview, as were key administrators who were familiar with ambulatory clinical operations, acute care clinical operations, and finances related to clinic operations.

Semi-structured interview protocols were adapted for each stakeholder group based on its relevant perspective (Additional file 1: Appendix C). Interviews were conducted by either of two researchers (AL, SV), recorded and transcribed for analysis. Transcripts of interviews conducted in Spanish were first transcribed in Spanish and then translated to English by health system affiliates with professional translation training.

Qualitative analysis

We conducted thematic analysis of interviews using a combined inductive and deductive approach with separate codebooks developed for each stakeholder group. The deductive codes were derived from the topic guide content and key implementation outcomes [29]. All transcripts were imported into NVivo (released March 2020) for analysis. Two coders (SV, ASL) first independently coded one transcript then met to discuss and align coding practice. The process was repeated until the researchers agreed coding alignment was achieved for each stakeholder group. Coders continued to independently code transcripts and met regularly to review emerging themes and revise the codebook [33]. Coded data were then summarized into a thematic matrix with color-coding to identify positive and negative sentiments where rows represented individual participants and columns represented key themes; this visualization further supported the identification of intra and inter- group convergence and divergence of opinion [34].

Results

Quantitative analysis

Characteristics of enrolled and unenrolled patients

A total of 719 patients (89 Stanford academic, 20 Stanford non-academic, and 610 unaffiliated) enrolled in the clinic during the inclusion period. After applying exclusion and selection criteria for all populations, 137 eligible enrolled patients and 126 unenrolled matched patients were included in the analysis. There was no statistically significant difference between enrolled and unenrolled patients in terms of key demographic characteristics, suggesting successful matching of the groups on these variables (Table 1). There was also no statistically significant difference in enrolled and unenrolled patients who were hospitalized for non-COVID-19 related reasons, suggesting that these groups were reasonably balanced regarding comorbidities and their general hospital utilization (Additional file 1: Appendix D).

Table 1 Characteristics of Enrolled and Unenrolled Patients in the Specialized COVID-19 Clinic

Covid-19 related ED encounters and hospitalizations

The enrolled and unenrolled groups had similar proportions of unique patients with a COVID-related ED-only encounter (7% enrolled vs 6% unenrolled; p = .76) as well as the total number of these encounters (13 enrolled and 9 unenrolled, p = .51) (Table 2). A larger proportion of enrolled patients had COVID-related hospitalizations, however, than unenrolled (8% vs 2%; p = .02); the enrolled patients also had a higher number of total hospitalizations (11 vs 3, p = .04). Chart review showed each admission event was preceded by an ED event (and direct admissions by primary care providers are not supported in this system), this reflects an increased admittance rate of 46% in the enrolled group (11 admittances of 24 possible events) versus 25% in the unenrolled group (3 admittances of 12 possible events). This could represent more appropriate use of the ED by enrolled patients, though this difference was not statistically significant (p = .32). Finally, there was no difference in length of stay during the observed hospitalizations (p = .67) or in mortality (p > .99).

Table 2 COVID-Related Events Within a 28-Day Observation Period by Enrollment Status in the Specialized COVID-19 Clinic

Chart reviews revealed a total of 6 ED encounters for asymptomatic repeat COVID-19 testing, (4 of 13; 31% ED visits in the enrolled and 2 of 9; 22% in the unenrolled). Events in which patients self-referred to acute care and were not admitted were similar between enrolled and unenrolled patients (8 of 13; 62% vs 6 of 9; 67%, respectively). However, events in which patients self-referred to acute care and were ultimately admitted were slightly higher in the enrolled than in the unenrolled group (11 of 24; 46% vs 3 of 12; 33%, respectively). Finally, all admission events for unenrolled patients were associated with poor oxygen status (defined as a pulse oximeter reading of 95% or below) at first clinical presentation (3 of 3; 100%), whereas this proportion was slightly lower for enrolled patients (9 of 11; 82%).

Validation of COVID-19 risk severity clinic protocol

Most enrolled patients were determined to have low (39%) or medium (43%) COVID-19 Risk Severity based on the clinic protocols (Table 3). There was not a statistically significant difference between risk severity levels and COVID-19-related ED visits or admissions after adjusting for multiple comparisons. However, there was a statistically significant difference between high and low risk severities in rate of ED-only events (p = .05) and combined ED and hospital admission events (p < .01), thus suggesting the risk stratification tool (Additional file 1: Appendix A) holds some validity.

Table 3 Enrolled Patients COVID-related Events Within the 28-Day Observation Period by COVID-19 Risk Severity

Interview findings

Characteristics of interviewed stakeholder groups

We conducted a total of 38 qualitative interviews June to October 2020 across the three stakeholder groups, including 21 patients, 9 (of 12 available) providers, and 8 (of 12 available) administrative stakeholders, 2 of whom had overlapping provider and administrative responsibilities and were therefore interviewed with both the provider and administrator protocols. Patient characteristics were predominantly female (67%), Hispanic/Latino (52%), Spanish speaking (52%) and were moderate risk based on clinical criteria (48%), though low and high risk patients were also represented (33, 19%, respectively). Reflecting the personnel serving in the CROWN clinic, providers were predominantly female (78%), physicians (56%) while administrators were also predominantly female (75%) and had diverse areas of operational focus (Table 4). Duration of interviews varied low to high (median) across stakeholder groups: 10–27 (19), 18–45 (25), 17–39 (24) minutes for patients, providers and administrators, respectively. The data below focuses on key themes related to care utilization that complement the quantitative analysis above, system considerations impacting the perceived benefits of the clinic, and clinic features impacting the acceptability of the clinic from patient and provider perspectives. We compare findings across methods to gain insight where applicable.

Table 4 Patient, Provider, and Administrative Stakeholder Characteristics

Perceptions and utilization patterns suggest an increase in appropriate care escalation

Both providers and administrators believed the clinic played a crucial role in identifying patients who required care escalation but who may not have otherwise sought or been able to access this care. Providers felt this to be particularly important given the nature of the COVID-19 illness where low blood oxygen levels can be present without associated symptoms, so called “silent hypoxia”. Such cases were detected through symptom monitoring and pulse oximeter information, which was “… super helpful. We do a resting and an ambulatory pulse ox. And then if the O2 sat is worrisome, we either have the patient come to clinic or send them to the emergency room” (Provider 3).

Patients who had access to pulse oximetry overwhelmingly reported the devices were easy to operate, and they used them approximately 1–3 times a day. One patient explicitly described using the device to determine whether to seek care (though no patient interviewed reported the need for escalation of care). Patients expressed universally positive sentiment in having the pulse oximeter as a resource during their illness.

Another mechanism that may have increased appropriate care utilization was the interim step that an in-person evaluation in CROWN offered in place of the ED:

…I've said, ‘You need to go to the ER,’ and [patients] are like, ‘No, I'm not going to the ER.’ Then I say, ‘Well, if you refuse to go there, I can offer you the CROWN in person, but I might still tell you to go to the ER [emergency department] after I see you.’ They seem more willing to do that than to go to the ER.’ (Provider 2)

Some patients expressed concerns about possibly being contagious and appreciated the remote care and streamlined in-person care the clinic offered.

These perceptions were supported by the quantitative data described above in which enrolled patients had a significantly higher number of inpatient admissions as well as a higher admittance rate upon presenting to the ED. The lack of a significant difference in length of stay between enrolled and unenrolled patients does not support the notion that patients presented for care earlier because of the program. Clinic providers were a source of referral to ED evaluation in one third (8 of 24 referrals) of the encounters for enrolled patients, suggesting the clinic played a substantial role in escalation of care.

Perceptions of a reduction in inappropriate care escalation without corresponding quantitative evidence

Providers and administrators believed the clinic also played a strong role in redirecting patients away from inappropriate care, specifically in the ED. One administrator shared he would “panic” without this resource given it was the primary place to send patients with confirmed or suspected COVID-19 (Administrator 1). Another provider reported that keeping patients out of the ED had become an unexpected predominant role of the clinic:

We thought that we would be picking up hypoxia that people weren’t noticing and sending them to the ER. And instead, we’re almost doing the opposite where we have patients who may feel a little shortness of breath and then they get their [oxygen saturation] and it’s 99% [optimal] and that actually helps me keep people at home. (Provider 5)

Without the clinic, providers perceived that patients would be otherwise “completely lost” given restricted access for potentially contagious persons and the reported reluctance of PCPs and specialists to see these patients in person (Provider 2). Several patients corroborated these challenges in accessing primary care, stating their doctors were either slow to return their calls and/or were unable to physically see them in regular clinic: “…they [clinic] said no, that they could not attend to me unless it was an emergency” (Patient 10). CROWN was perceived by providers as offering “everything the patient needs in one place,” including imaging, labs, and other studies, thereby optimizing safety and efficiency (Provider 4).

Despite these beliefs and reported behaviors, the actual utilization data described above does not corroborate these beliefs, as there was no difference between enrolled and unenrolled ED-only utilization. Indeed, chart reviews suggested similar numbers between enrolled and unenrolled patients for an inappropriate ED presentation –asymptomatic repeat COVID-19 testing—despite provider efforts to educate patients during their enrollment.

Systems considerations impacting perceived value of the clinic

Several administrators and providers reported the clinic’s greatest impact was possibly its role in supporting access to health services for non-COVID patients. They reported that safety was increased throughout the system by directing potentially infected patients to a single site for non-acute services, thereby facilitating a gradual reopening of services following the initial lockdown:

Our ability to reopen all the sites…was dependent on the fact that we had a place that we could also appropriately case manage and track people with COVID …So the financial viability of the clinic itself can't be measured in the clinic financial performance, but the rest of organization’s capability to keep pace or open up quickly and stay open. (Administrator 7)

Administrators also reported challenges capturing reimbursement for the clinic due to structural challenges. Having been rapidly launched, the clinic lacked its own budget and instead operated under the umbrella of its neighboring urgent care clinic with “borrowed” resources (Admin 3). Many RPM activities, such as outreach by non-provider staff were not reimbursable, and when they were, the patient mix reportedly skewed towards the uninsured or underinsured, further limiting reimbursement.

Clinic features impacting acceptability of the specialized COVID-19 clinic

Patients and providers described how several clinic characteristics impacted their perceived acceptability of the clinic (Table 5). Patients and providers both expressed generally favorable perceptions related to improved access to care, benefits of patient education, benefits of concentrating COVID-19 expertise within a specialized clinic, support for mental health services, and use of pulse oximeters.

Table 5 Clinic Features Impacting Acceptability of the Specialized COVID-19 Clinic Across Patients and Providers

Acceptability of other aspects was mixed. Patients overwhelmingly reported the frequency of outreach was appropriate, apart from one asymptomatic patient who felt outreach was too frequent. For providers, some debate existed as to the appropriate frequency of outreach, with a few providers questioning whether a proactive outreach approach was the optimal use of scarce resources, noting that many patients felt fine when they were contacted.

In addition, while most patients accepted virtual care via video or phone, noting community-wide shelter-in-place orders and their own need to isolate given their diagnosis, two participants expressed concerns that remote care could contribute to missed important clinical changes. Providers usually preferred video to phone so they could “eyeball” a patient to visually assess their clinical status (Provider 1). In addition, a few providers also regretted the system’s inability to provide home social services to patients in a time of significant need—an ED visit was sometimes the only solution when a patient could not adequately care for him- or herself.

Finally, acceptability was limited by the challenges associated with specialist care, particularly for patients with serious comorbid conditions (i.e., cancer, post-transplant). Patients wanted to know what their COVID-19 diagnosis meant for them in terms of their pre-existing condition; providers’ efforts to connect with these specialists on behalf of patients were sometimes fruitless. Particularly challenging for providers was convincing specialists that patients were no longer infectious after their 10-day quarantine and therefore qualified for in-person specialty care.

Discussion

The specialized COVID-19 clinic and RPM program was launched early in the COVID-19 pandemic to fulfill multiple stakeholder needs—primarily to promote access to non-emergency services for patients suffering from COVID-19 and secure patient and staff safety systemwide [30]. This convergent mixed methods analysis suggests stakeholder beliefs that the clinic reduced acute care utilization (ED and admission events) was not supported by the quantitative analysis. Rather, observed hospitalizations were actually higher in the enrolled group than the unenrolled group. Further, no difference was observed between groups for ED events that did not result in an admission. Despite these negative quantitative findings, qualitative data suggests healthcare access increased for patients with confirmed or suspected COVID-19 who were otherwise restricted from non-emergency care. Finally, the clinic’s unmeasured benefit supporting the safe reopening of health services systemwide was felt to be substantial.

These findings seem to contradict the reduction in acute care utilization previously seen following COVID-19 RPM efforts in which patients were enrolled following an ED encounter or hospitalization [11, 12]. Our analysis differed from these studies in that it focused on patients who were enrolled following an outpatient diagnosis. This difference, as well as variation in how RPM was implemented may contribute to our observation that hospitalization was higher in the enrolled patient group [22].

Such increase in the utilization of health services has been previously seen with increased healthcare access [35, 36]. Increased exposure to the health system through the COVID-19 clinic may have increased patient comfort with the health care system overall, thereby lowering patients’ perceived barrier to seeking a higher level of care. Though clinician perspectives presented here suggest increased utilization was often an appropriate escalation of care, we did find patients presenting to the ED for asymptomatic repeat COVID-19 testing in both the enrolled and unenrolled groups. This suggests increased opportunity for patient education regardless of enrollment status, and possibly at the time of their initial diagnosis.

At the time of the analysis, outpatient care for COVID-19 was also primarily supportive. Simply increasing the monitoring for clinical deterioration (rather than combining this effort with treatment) was perhaps unlikely in retrospect to change the clinical course of the disease. Instead, clinicians had more opportunity to identify worsening disease and therefore escalate patients to a higher level of care. Future work in this area should account for COVID-19 outpatient treatments that have become available since this analysis [37].

Other possible reasons for increased utilization in the enrolled group include increased patient psychological need for health services at a time of great vulnerability [38], ED physician bias to admit a patient sent in by another provider, and residual selection bias towards “sicker” patients in an enrolled population (i.e. patients with underlying disease not already accounted for in the analysis are more likely to opt into a monitoring program) [39].

The present work represents an example of a collaboration between researchers, clinicians and health system administrators to evaluate an ongoing initiative in order to inform future improvements and the direction of limited resources [40]. Evaluating the value of such a clinic from a mixed-methods, multi-stakeholder perspective is particularly important in this setting where typical value-oriented data (i.e., cost and outcomes) is missing or overshadowed by unmeasured benefits —for example, benefits from the gradual reopening of a health system following an initial lockdown where no comparator exists.

The present work is also notable in its examination of an important new use case for RPM: monitoring of acute infectious disease. All stakeholders largely believed that RPM added value in this clinical context, and patients and providers described how it shifted their care-seeking behavior and clinical decision-making, respectively. This novel use of RPM touches on growing trends to explore and expand “hospital-at-home” models in which patients who would otherwise meet inpatient criteria received care in the comfort of their own home, with the support of remote clinical experts [41,42,43]. Understanding the bridge between hospital-at-home and traditional RPM will be an important area for future research. Further, careful consideration should be given to which types of patients may benefit most from RPM given limited resources. For example, targeting clinic resources towards patients who are higher risk for adverse outcomes (given the link between increased risk and escalation of care demonstrated in Table 3), and/or with a lower level of health literacy may be more cost-effective than a uniform approach and is an area for future work.

Evaluation limitations include the retrospective case-control design in which patients had the option to opt into the program; we were also unable to verify through the data that each patient was equally offered enrollment following standard protocols. Our analysis therefore relies on successful matching across several variables. The absence of any statistically significant difference in these variables and acute care events unrelated to COVID-19 between these groups provides some reassurance against selection bias. Individuals without access to at least a phone were unreachable and therefore not enrolled, though prior work suggests this is a small population [44]. Manual chart review captured all ED encounters and hospitalizations that occurred within any health system in the area that used the predominant electronic health record system (Epic Systems), though uncaptured encounters outside this network were possible. We further note that diverse clinical circumstances involving a novel disease limited our ability to fully define “appropriate” care. Our assumption therefore focused on two extremes—an admission following an ED visit suggests that ED visit was likely appropriate, whereas an asymptomatic patient presenting to the ED for a repeat COVID-19 test was likely better served in an alternative setting. We recognize possible exceptions to these generalizations, including our inability to draw conclusions about cases between those extremes (i.e. value-added ED visits [21]). For these reasons, conclusions should be interpreted with caution.

Conclusion

Acute care utilization data and multi-stakeholder interviews suggest heightened outpatient observation through a specialized COVID-19 clinic and RPM program may have contributed to an increase in appropriate acute care utilization. A reduction in inappropriate care utilization was not seen, despite provider and administrator beliefs in this benefit. The clinic’s role securing safety systemwide, leading to a gradual reopening of health services systemwide was endorsed as a primary, if unmeasured, benefit. Additional evaluation is needed to understand the growing role of RPM in the novel support of acute infectious disease.