Introduction

The SARS2-Coronavirus 2019 (COVID) crisis has severely impacted public health and disrupted the provision of healthcare, including organizing, mobilizing, and deploying extra resources to effectively address emerging needs [1]. For instance, healthcare service delivery has been impacted in numerous ways [2], changing many essential elements vital to providing person-centered care (PCC) [3, 4], and implementing widespread use of virtual care.

Virtual care is defined as any interaction between patients and/or members of their circle of care, occurring remotely, using any forms of communication or information technologies (e.g., phone calls, videoconferences, and secure messages), to facilitate or maximize the quality and effectiveness of patient care [2, 5, 6]. Virtual care can play a vital role in emergencies by supporting healthcare needs remotely [7], streamlining the necessity of healthcare services, conserving medical resources [8], directing the medical supply on the basis of priority [9], and providing telecommunication for visitor-patient interaction[10, 11].

The COVID pandemic resulted in changes to the patient care environment, impacting the delivery of PCC [2,3,4]. PCC promotes adherence to treatment, improved care, better health outcomes, enhanced relationships between providers and patients, improved perceptions of doctor performance, and patient trust [3]. PCC is advocated by both patients and providers as it supports a higher quality of care [3].

Worldwide, over fifty-eight percent of the countries that experienced service disruption during the pandemic adopted virtual care delivery to continue to meet healthcare needs [12]. Hence, it is crucial to understand the impact of virtual care delivery on patient experiences and outcomes. Additionally, to deliver good patient-centred care, we need to understand what barriers or challenges present and how virtual care can be optimized. Thus, we conducted this systematic review to identify evidence on how virtual care delivery has impacted patient experiences and outcomes, both measured using validated Patient-Reported Experience Measures (PREMs) and Patient-Reported Outcome Measures (PROMs) respectively, during the first 2 years of the COVID pandemic across a spectrum of diseases and healthcare settings.

PROMs are used to assess a patient’s health status at a particular point in time, which can be completed either during an illness or while treating a health condition, or pre-and post-event to measure the impact of an intervention [13]. Capturing patient experiences is an important part of an overall effort to measure health system performance and is integral to delivering patient-centred care. Routinely applying PROMs and PREMs can enhance communication between patients and care providers, inform decisions for value-based healthcare, and improve patient care experiences and outcomes. To achieving health system goals, PROMs and PREMs are increasingly recognized for providing valuable and essential information [13]. With the onset of the COVID-19 pandemic, as the healthcare systems evolve, it becomes increasingly significant to measure healthcare delivery, PREMs, PROMs, and clinical outcomes towards a strong person-centred healthcare system.

Materials and methods

Based on the exploratory nature of this review and our objective to describe and map the literature guided by our aim outlined above, a systematic review approach was selected. The strength of the systematic review methodology is that it provides a rigorous and transparent approach of mapping the literature to ensure reliable and meaningful results for end-users [14]. Study selection and screening process was performed using the Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) methodological frameworks [15]. The PROSPERO registration number is CRD42022306179. Additionally, we engaged a patient research partner with experience accessing virtual care during the COVID-19 pandemic. Our patient research partner was engaged in the conduct of this review (reviewing the study protocol, search strategy, assisting in title and abstract screening, data abstraction, reviewing this manuscript and is a co-author).

Search strategy and information sources

The preliminary search strategy was developed in collaboration with a research librarian at our University, who also has research expertise in systematic reviews. The search strategy and keywords are presented in Additional file 1. The search strategy combined structure language, keywords, and relevant synonyms. The search terms for each concept were connected through Boolean Operators ‘AND’, while search terms within each concept were combined using ‘OR’. The comprehensive search terms were tailored to each data sources, including MEDLINE, CINAHL, EMBASE, and APA PsycInfo, published from January 2020 until January 2022. To minimize publication bias and missing any relevant literature, we performed an additional search from reference lists of the included studies and grey literature sources, including google scholar and conference proceedings.

Inclusion and exclusion criteria

Inclusion criteria: (1) Population: patient, caregiver, and family member; (2) Intervention: delivery of virtual care during COVID-19; (3) Outcome: virtual care experiences, and outcomes reported by patient/caregiver/family, as well as health utilization outcomes; (4) Study Design: any studies (qualitative, quantitative, and mixed methods); (5) Peer-reviewed studies published only in English language; and (6) Only studies that used validated measures (PROMs and PREMs), as reported by the authors of the included studies.

Exclusion criteria: (1) Provider's experience; (2) Use of unvalidated patient-reported measures; (3) Clinical trials (RCT), research protocols, discussion summaries, abstracts and conference posters, systematic reviews, editorials, and letters; (4) Studies that were not in the English language.

Selection of sources of evidence

References for all included studies were uploaded and managed through Covidence. Titles and abstracts were screened for eligibility by two independent reviewers. Our team of reviewers initially screened 50 references together to ensure consistency between reviewers. For the full-text articles inclusion screening, the first five articles were reviewed by all the reviewers to ensure consistency. After that, each full-text article was reviewed by two independent reviewers. Differences between reviewers were resolved through detailed discussion and consensus or consulting a third reviewer. Differences between reviewers were resolved through detailed discussion and consensus or consulting a third reviewer.

Data charting process and data items

Two independent reviewers abstracted all relevant data following the full-text screening process for eligibility. A standardized data abstraction form was created to process all data. This form was first piloted by trained reviewers for at least two studies and revised until the authors were satisfied that all relevant data was captured accurately and comprehensively. The following information was extracted from each study for collective evaluation: author, year of publication, country, objectives, study design, patient population, virtual care delivery methods, patient reported experiences, patient reported outcomes, and health utilization outcomes. One reviewer abstracted the data, and the second reviewer checked/verified the abstracted data. Any disagreement in the abstracted data was resolved through discussion and consensus between the two reviewers, or a third reviewer was consulted, if need be. The data items abstracted from each study are presented in Tables 1, 2, and 3.

Table 1 Descriptive characteristics of all included articles
Full size table
Table 2 PREMs listed in the included articles [17, 20, 21, 24, 25, 27,28,29,30,31,32, 38, 40, 44,45,46, 49,50,51, 54, 60,61,62,63,64, 66,67,68,69,70,71,72,73,74,75,76,77,78,79]
Full size table
Table 3 PROMs listed in the included articles [63, 69, 90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109]
Full size table

Quality assessment

The Mixed Methods Appraisal Tool (MMAT) Version 2018 was used to assess the quality of all included studies [16]. Using the Mixed-Methods Assessment Tool (MMAT), the included articles demonstrated a relatively high degree of transparency in the presentation of their methods and results. Study quality did not dictate article exclusion from the review and, as such, these articles were still included in data extraction and analysis. Please note as well that using the MMAT questions to generate an overall score or rating of the articles is discouraged by the creators of the tool. For a more thorough overview of included study quality, please refer to the Additional file 2.

Data analysis

A careful assessment of data and analysis from all included studies was performed to establish and validate any conclusions regarding virtual care experiences during COVID-19. Bibliographic data, the population, and the setting for included studies are summarized using descriptive statistics in Table 1. The patient reported experiences, outcomes, and utilizations are synthesized in Tables 2 and 3. We summarized the results of the qualitative data by patient experience domains, guided by the qualitative software analysis, NVivo.

Results

After excluding duplicates, our search captured 6048 records for the title and abstract review. After title and abstract screening, a total of 644 peer-reviewed articles were assessed for full text review. Then, after full text review, we included 102 articles in this review (Fig. 1). Reasons for exclusion included articles reporting wrong outcomes, assessing satisfaction, not validated PROMs/PREMs, full-text unavailable, and wrong timing—not during COVID-19.

Fig. 1
figure 1

PRISMA for flow chart of the literature review and article identification process

Full size image

Table 1 depicts the descriptive summary of included articles. Most articles (39.2%) were published in the USA, followed by Canada (12.7%), United Kingdom (11.8%), Australia (7.8%), Italy (5.9%), and India (3.9%) (Fig. 2). Studies were of cross-sectional design (33.3%), cohort (30.4%), qualitative (12.7%), mixed methods research (11.8%), case series (2.9%), non-randomized experimental (2%), and other study designs (6.86%). In regard to the quality of the included studies, most articles addressed all (n = 41) or four out of five (n = 42) of the quality criteria listed in the assessment tool. While 19 articles fell below this quality threshold.

Fig. 2
figure 2

World map displaying the geographic origin of the articles included in the systematic review

Full size image

Most studies reported virtual care delivered in the specialized outpatient setting (78.4%) during COVID-19. Some studies reported virtual care delivery in acute hospital care settings (7.8%), primary care (4.9%), and in rehabilitation centers (3.9%). Studies reported delivery of virtual care via telephone and video (34.3%), video only (33.3%), telephone only (21.6%), remote monitoring (2%), and other (8.2%). Most studies explored virtual care delivery for adults (71.6%), some reporting both adult and pediatric populations (14.7%), and few articles that reported delivery of virtual care for pediatric populations (10.8%). Most virtual care delivery was in the year 2020 (90.2%).

Identification of Patient-Reported Experiences Measures (PREMs)

Table 2 highlights the 29 validated Patient-Reported Experience Measures (PREMs) identified in the review from 47 articles that evaluated the patient experience of receiving virtual care during COVID-19. Some articles used more than one PREM. Most of these measures were completed by adult patients. Common measures included The Telehealth Usability Questionnaire (reported by 11 studies), Telemedicine Satisfaction Questionnaire (reported by 6 studies), Telehealth Satisfaction Scale (TeSS) (reported by 5 studies), and Consumer Assessment of Healthcare Providers and Systems—Clinician and Group Survey (CG-CAHPS) (reported by 5 studies). Common domains associated with these measures included access to care, communication, and domains specific to the clinician's attitude and behavior.

Benefits of virtual care delivery

Findings from patient experience measures (PREMs) highlight positive responses on virtual care from patients in various domains. Some examples include feeling comfortable in receiving care virtually (e.g. due to privacy) (n = 8, 17%) [17,18,19,20,21,22,23,24,25], feeling safe against COVID-19 (n = 5, 11%) [18, 21, 26,27,28,29], communication with healthcare providers (n = 31, 66%) [17, [17, 20,21,22,23,24, 30,31,32,33,34,35,36,37,38,39,40,41,42,43], the convenience of virtual care and saving time (n = 24, 51%) (e.g. minimizing barriers such as transportation, traffic, cost of gas and parking, and associated anxiety) [20,21,22,23, 26, 28,29,30,31,32,33, 35, 37,38,39, 43,44,45,46,47,48,49,50,51,52,53,54,55], access to care (n = 9, 19%) [19, 22, 24, 28, 35, 38, 43,44,45, 51, 56, 57], patient engagement in care (n = 4, 9%) [36, 37, 44], comfort in the technology/telehealth system (n = 17, 36%) [27, 28, 35, 51], and not experiencing wait time delays in seeing their healthcare providers (n = 8, 17%) [22, 28, 46, 50, 54].

In the studies that included qualitative findings, we get an in-depth understanding of the experiences of patients and caregivers with virtual care delivery during COVID-19. For instance, in the study by Al-Sharif et al.[58], they found convenience and safety to be two major advantages to virtual care delivery, especially with the high risk of getting infected with COVID-19. Juarez-Reyes et al.[59] found patient participants expressed gratitude for continued mental health support, and being able to still be a part of virtual group sessions.

Barriers with virtual care

Some studies that used PREMs and also qualitative interviews reported challenges patients and caregivers faced with virtual care, such as feeling rushed during the virtual appointment (2% of PREMs articles) [26, 38, 48], lack of physical contact with the healthcare provider for physical examinations (15% of PREMs articles) [21, 23, 26, 35, 58, 87, 88], technical challenges (2% of PREMs articles) [26, 29, 35, 37, 43, 48, 56, 77], a preference for in-person care delivery (8.5% of PREMs articles) (e.g. due to the lack of personal connection with healthcare provider online) [21, 23, 28, 38, 41, 54, 88, 89], and difficulty with communicating symptoms or asking all of their questions (6% of PREMs articles) [29, 87, 90]. For instance, the study by Gibbs et al. [48] found that adult clients undergoing assessment for autism were concerned about communication difficulties in the online environment, especially with using and reading body language and feeling self-conscious about seeing themselves on screen. Adult clients and parents/caregivers were also concerned with clinicians possibly missing certain subtle behaviors not apparent on screen [47]. The study by Stirling Cameron et al. [42] found telehealth appointments to be challenging for Syrian refugee women who used interpreters for their appointments. The women expressed disappointment with back-and-forth telephone calls, and not being able to effectively communicate with their primary care providers [42].

Identification of Patient-Reported Outcome Measures (PROMs)

We identified 43 validated Patient-Reported Outcome Measures (PROMs) that assessed patient health status during the COVID-19 pandemic (Table 3). The Generalized Anxiety Disorder (GAD-7) (used in 4 studies), Eating Disorder Examination Question Version 4.0 (EDE-Q-IV) (used in 3 studies), and Patient Health Questionnaire (PHQ-9) (used in 3 studies) were the only measures reported in more than one study. All other studies utilized various PROMs. Specific PROMs were grouped by disease/condition such as PROMs for Mental Health (n = 12, 28%), Gastrointestinal (n = 2, 7%), Musculoskeletal (n = 10, 23%), Early Childhood Development (n = 1, 2%), Neurological (n = 2, 5%), and Respiratory (n = 1, 2%). Additionally, fourteen generic PROMs were identified such as the Short Form (SF-12) Quality of Life Questionnaire, Patient Health Questionnaire (PHQ-9), and the Pediatric Quality of Life Inventory (PedsQL). These studies administered PROMs following virtual care delivery. Some studies found that patients had improvements in quality of life and reduction in symptoms [91,92,93,94,95,96,97,98], such as improvements in mental health and wellbeing [63, 92, 96, 99,100,101]. However, some studies also reported no significant differences in PROMs scores before and after virtual care or between different treatment groups (in-person care vs virtual care) [90, 91, 96, 102,103,104,105,106,107,108]. A summary of the findings from the studies is included in Table 3.

Impact of virtual care delivery on healthcare use

Eleven studies evaluated the impact of virtual care delivery on healthcare use [105, 108, 110,111,112,113,114,115,116,117,118]. Nascimento et al. [110] evaluated the impact of telemedicine on visits to emergency departments and hospital admissions during the pandemic in Brazil. They found rates of ED visits and hospital admissions were respectively, 17.3% and 2.3% for patients who attended at least one teleconsultation. Kesavadev et al. [111] reported successful prevention of hospitalization for nearly all patients in a virtual in-patient program. In the study by Thesenvitz et al.[105], patients reported less use of services such as Alberta’s Health Link advice line, emergency department visits, and visits with family physicians.

Discussion

During the COVID-19 pandemic, infection control efforts have necessitated the reduction of in-person clinical visits and routine procedures leading to provider- and system-level changes in the delivery of PCC. This change might have altered patient experiences with their care, and measuring patient experiences becomes increasingly significant toward a strong person-centered healthcare system. In this review, we provide an overview of the PREMs and PROMs that have been utilized to assess patient experiences with virtual care and patient-reported outcomes during the COVID-19 pandemic. Following initial screening and full-text review, 102 articles were included in this study. These studies demonstrated large heterogeneity in study design, population of interest, and virtual care modality. Most articles targeted the delivery of virtual care in specialized outpatient settings (78.4%), including fields such as oncology, dementia, neurology, urology, dermatology, and psychiatry. Studies also primarily assessed adult responses to virtual care delivery (71.6%), with far fewer studies exclusively assessing the perspectives of pediatric patients (10.8%). We also found that a relatively even number of studies assessed patient experiences with virtual care delivery via videoconferencing (33.3%), telephone calls (21.6%), or a combination of both (34.3%). While prior systematic reviews have examined PREM and PROM utilization in various in-person care settings, this study is distinct in its focus on studies that used patient-reported measures to gauge patient experiences to virtual care during the pandemic.

The sheer number of articles (N = 102) included in this review highlight the breadth of information available on patient-reported measures that were used during the virtual care provision, as well as the adaptability of international health systems. This also provides evidence of the importance healthcare professionals ascribe to amplifying the patient voice. Despite this, review findings also show increased investment in specific patient populations, leading to the potential absence of other patient groups.

One specific group that was underrepresented in this review was pediatric patients, as we found a limited number of studies conducted in this population (10.8% of studies were pediatric focused). The lack of research into the experiences and outcomes of pediatric patients receiving virtual care signifies a gap in knowledge that could provide incredibly useful insight into pediatric care provision. Santoro et al., [50] discuss the foreseeable benefits of virtual care for pediatric patients, highlighting the involvement of one or more caregivers in the transportation and supervision of pediatric patients during in-person visits in 2021. From the pediatric studies in this review, patient caregivers discussed the convenience and cost-benefits of virtual care [18, 50, 107].

A second underrepresented patient population in this review are primary (i.e., general) care recipients. Even though primary care serves as the first interaction many patients have with the healthcare system, patient experience in primary care was only assessed in 4.9% of the articles pulled. Not capturing patient perspectives on virtual primary care delivery could significantly impact other healthcare areas by restricting the ability of general practitioners to communicate, treat, and refer patients to specialists effectively.

Another concern with virtual care provision, irrespective of the patient population being researched, is acknowledging the patients who were unable to access virtual care. Virtual care has been shown to exacerbate health inequities, creating what has been termed the “digital divide” whereby health information technology and virtual care disproportionately exclude already marginalized populations from accessing care [53, 119]. This is of particular concern during the COVID-19 pandemic, as rapid transitions from in-person to virtual care have primarily been implemented using a health systems perspective with limited consideration for diverse patient partnerships [120, 121]. The potential bottlenecking of the types of patients included in this review should therefore be needed, with greater efforts placed on broadening and adapting virtual care efforts to better suit the care needs of all patients in future research.

The patient-reported measures identified in the review often addressed care accessibility, patient-care team communication, and clinician attitudes and behavior with patients during virtual care. Several advantages to virtual care were identified, with patients citing greater convenience and increased protection from viral spread. Other literature supports these findings, explaining the potential of virtual care to alleviate barriers to care in rural and geographically isolated communities [122, 123]. Buyting et al. [124], discusses the benefits of virtual care in rural settings when a priori work is done to ensure all interventions are appropriate to the population of interest. Greater ease of access to care was also evaluated by Darr et al. [25], who identified a correlation between virtual care provision and a reduction in non-attendance rates. This also highlights the potential economic benefits of virtual care, as non-attendance rates are closely linked to increased healthcare utilization [25, 125]. In addition to virtual care's advantages, patients also mentioned various challenges. Barriers to virtual care included difficulty navigating online platforms, a need for greater technical support or educational materials, and the lack of physical interactions with healthcare providers. Edge et al. [28], reported that some patients felt they received worsened psychological support through virtual care and experienced greater difficulty understanding the clinical information shared by their healthcare provider. In response to this, 1 in 5 patients were hesitant to use virtual care in the future [28].

Virtual care is associated with various benefits and challenges, offering increased access to care during times of public isolation but also restricting care to populations experiencing social marginalization or with limited technological infrastructure [126]. The polarity of responses to virtual care raises the question of whether this mode of care will continue beyond the pandemic or if care will largely return to in-person once safe to do so. While a large proportion of patients included in the review mainly expressed positive reactions to virtual care, with some stating a preference for virtual care over in-person visits, consideration should also be applied to those not heard in these studies. Therefore, if this were to continue long-term, greater attention needs to be directed toward making technology a facilitator instead of a barrier to care access. Perhaps the most promising approach to virtual care in the future is implementing specialty-specific triage practices to provide patients with the most appropriate care. Other researchers have proposed this, promoting the benefits of triaging for better allocation of resources, assessment of disease acuity, and accommodation to various social factors [127, 128].

Irrespective of the degree to which virtual care is used in the future, this research provides a comprehensive overview of what patient-reported measures can be used by healthcare professionals to evaluate virtual care quality. As virtual care represents a burgeoning approach to care provision, utilizing these measures (PROMs/PREMs) can be crucial to ensuring that the services provided are grounded in patient-centeredness [129]. This study has implications on all conceivable aspects of virtual clinical practice, by equipping healthcare professionals with the means to respond to the needs of their specific patient population.

Strengths and limitations

One key strength of this study was the patient-oriented approach. We engaged a patient research partner in our team who was involved in reviewing the study protocol, title and abstract screening, data abstraction, reviewing the results and is a co-author in this manuscript. Additionally, we enlisted the support of a research librarian to ensure our search strategy was comprehensive.

Despite the methodological rigor applied in this review, this study was not immune to limitations. One potential issue with this study is that, while the review included studies published between January 2020 and January 2022, the vast majority (90.2%) of included articles detailed work conducted in 2020. Limited information on patient experiences further into the pandemic restricted our ability to assess the effects of patient and family burnout from continued virtual care use. Another possible limitation of this study relates to our focus on patient and family responses to virtual care, exclusively. While this does exclude the perspectives of healthcare providers and administrators, our emphasis on the patient voice was also a deliberate choice to display the experiences of virtual care recipients. Another limitation in this study was our inability to perform a meta-analysis due to the inclusion of studies that differed across statistical and methodological characteristics. Lastly, though not a limitation of this review, a recurrent issue experienced in studies examining the use of PREMs, is the common, yet misguided practice, of using “experience” and “satisfaction” as interchangeable terms. These terms, while seemingly similar, do have distinct qualities with “satisfaction” associated with greater subjectivity and potentially reflecting patient expectations more so than “experience” which describes objective aspects of patient care [130, 131].

Conclusions

In future studies, it would be efficacious to explore more recent patient experiences with virtual care as well as the experiences of other key stakeholders. Improved patient receptivity to care at the onset of the pandemic has been previously documented, however, patient experiences further into the pandemic is lacking. Due to widespread burnout within the healthcare system, assessing more recently completed patient-reported measures may paint a different picture of the benefits of virtual care [132]. Additionally, further research into healthcare professionals' perspectives (I.e., healthcare providers and administrators) would offer an alternative lens on the practicality and feasibility of long-term virtual care.