Effect of Personalized Messages Sent by a Health System’s Patient Portal on Influenza Vaccination Rates: a Randomized Clinical Trial

Background Adult influenza vaccination rates are low. Tailored patient reminders might raise rates. Objective Evaluate impact of a health system’s patient portal reminders: (1) tailored to patient characteristics and (2) incorporating behavioral science strategies, on influenza vaccination rates among adults. Design Pragmatic 6-arm randomized trial across a health system during the 2019–2020 influenza vaccination season. The setting was one large health system—53 adult primary care practices. Participants All adult patients who used the patient portal within 12 months, stratified by the following: young adults (18–64 years, without diabetes), older adults (≥65 years, without diabetes), and those with diabetes (≥18 years). Interventions Patients were randomized within strata to either (1) pre-commitment reminder alone (1 message, mid-October), (2) pre-commitment + loss frame messages, (3) pre-commitment + gain frame messages, (4) loss frame messages alone, (5) gain frame messages alone, or (6) standard of care control. Patients in the pre-commitment group were sent a message in mid-October, asking if they planned on getting an influenza vaccination. Patients in loss or gain frame groups were sent up to 3 portal reminders (late October, November, and December, if no documented influenza vaccination in the EHR) about importance and safety of influenza vaccine. Main Measures Receipt of 1 influenza vaccine from 10/01/2019 to 03/31/2020. Key Results 196,486 patients (145,166 young adults, 29,795 older adults, 21,525 adults with diabetes) were randomized. Influenza vaccination rates were as follows: for young adults 36.8%, for older adults 55.6%, and for diabetics 60.6%. On unadjusted and adjusted (for age, gender, insurance, race, ethnicity, and prior influenza vaccine history) analyses, influenza vaccination rates were not statistically different for any study group versus control. Conclusions Patient reminders sent by a health system’s patient portal that were tailored to patient demographics (young adults, older adults, diabetes) and that incorporated two behavioral economic messaging strategies (pre-commitment and loss/gain framing) were not effective in raising influenza vaccination rates. Trial Registration This trial was registered with ClinicalTrials.gov (NCT04110314). Supplementary Information The online version contains supplementary material available at 10.1007/s11606-021-07023-w.


INTRODUCTION
Influenza causes substantial morbidity and mortality among adults. 1,2 The Advisory Committee on Immunization Practices (ACIP) recommends annual influenza vaccination for all US adults, 1,2 and the US Healthy People 2030 goal is >70% influenza vaccination rates. 3 However, in 2019-2020 (before the pandemic), US vaccination coverage for the adults was as follows: 18-49 years (38.4%), 50-64 years (50.6%), and 65+ years (69.8%). 4 One strategy to raise influenza vaccination rates is vaccination reminders to patients-usually via telephone, autodialer, or mail; this is supported by a recent Cochrane review. 5 Although recommended by the Task Force on Community Preventive Services for vaccinations, 6 few primary care practices send reminders. 7,8 Experts recommend centralized reminders sent by health systems. Two studies testing reminders from state immunization information systems for children found variable impact; 9,10 authors speculated that reminders from patients' primary care providers might have greater impact. Little is known about vaccination reminders for adults, with few recent studies 5 in today's era of vaccine hesitancy. [11][12][13] Many electronic health records (EHRs) have patient portals: secure, Internet sites, and mobile applications allowing patients and healthcare providers to communicate. 14 Portals are used widely, 14 particularly with telehealth. 15 A 2014-2015 randomized trial evaluating patient portal plus interactive voice response reminders found only 1% improvement in influenza vaccination rates in a population with extremely low baseline coverage. 16 We evaluated the impact of one, two, or three generic portal reminders and found only minimal impact on influenza vaccination rates among adults and children for the 2018-2019 vaccination season. 17 It is unclear whether the negative findings were due to suboptimal messaging or to limited impact of portal reminders themselves.
Tailored messages targeting health beliefs can improve some health behaviors. 18 Concerns about influenza vaccine include vaccine safety and efficacy. 13,19 Also, healthy young adults might feel less vulnerable to influenza disease and those with chronic conditions more vulnerable. 13 Thus, reminder messages tailored by age (i.e., young adults, older adults) and chronic conditions (e.g., diabetes) might improve their effectiveness.
For this study, we expected to increase vaccination uptake by utilizing principles from social psychology 20 and behavioral economics 21,22 by testing two strategies: precommitment and gain loss framing. Pre-commitment is a strategy in which people are asked to commit today to engage in a future target behavior, harnessing people's desire to act consistently with public statements and their prior active choices. [23][24][25][26] We expected that having patients pre-commit to their doctor that they will obtain the influenza vaccine should increase their follow-through when later encouraged to do so.
Gain loss framing is a strategy in which a message is described as what a person has to gain versus to lose by taking a particular action. A cornerstone of behavioral economics is the observation that decisions are influenced by whether outcomes are framed as losses or gains, even when options are objectively equivalent. 21 Several studies have suggested that framing influences decisions by doctors and patients. [27][28][29] Reversing losses was found to provide stronger motivation than comparable gains for HPV vaccination in one study 28 (although not in another). 30 A review of loss/gain framing for vaccinations noted studies on HPV vaccine with a suggestion that parents might be more persuaded by loss framed messaging. 31 Based on studies on HPV vaccination, we expected that presenting messages highlighting costs of not getting an influenza vaccine would lead to higher vaccination rates than messages highlighting the benefits of vaccination.
We performed a 6-arm randomized clinical trial comparing the impact of (1) pre-commitment versus no pre-commitment and (2) negative or positive gain frame messaging, on receipt of influenza vaccination for three patient groups within a large health system: young adults (18-64 years), older adults (65+ years), and patients with diabetes (18+ years).

Study Design
The University of California, Los Angeles (UCLA) IRB approved the study with waiver of patient consent. Between 10/ 1/2019 and 3/31/2020 (pre-pandemic), we conducted a 6-arm RCT (Figure 1), randomizing patients to: control (no messages), pre-commitment letter only, pre-commitment letter plus loss framed reminders, pre-commitment letter plus gain framed reminders, loss framed reminders only, and gain framed reminders only. Pre-commitment groups received 1 pre-commitment message in mid-October. Loss or gain frame groups received up to 3 portal reminders-emphasizing the importance and safety of influenza vaccine-in late October, November, and December if no documented influenza vaccination. We conducted the trial across all 53 internal medicine, medicine-pediatric, and family medicine primary care practices at UCLA Health.

Study Participants
All practices had the same Epic™ EHR and portal. First, we identified all primary care patients ≥18 years ( Figure 1). The health system defines primary care patients by the following: ≥2 primary care provider (PCP) visits (by Evaluation and Management office codes) within 3 years or ≥1 PCP visit with preventive service codes within one year, or managed care patient assigned to UCLA Health (irrespective of visits). Second, we identified the primary care practice most recently visited within three years. Third, we grouped patients into family units with algorithms matching patient's phone number, address, insurance member number, or patient guarantor ID. Fourth, we identified active portal users as patients or portal proxies (for elderly or disabled persons) who logged into the portal at least once in 12 months, not including the initial portal login ( 67% of primary care patients). Fifth, we stratified all patients from the same family by patient groups (young adults 18-64 years without diabetes, older adults ≥65 years without diabetes, adults ≥18 years with diabetes [including diabetes type 2] per SUPREME criteria 32 ). Sixth, statisticians randomly selected one active portal-using index patient per family within each stratum, generating the denominator of potential subjects; other study personnel and healthcare providers were blinded to study allocation. We excluded patients who were not active portal users and family members of index subjects (for consistency with Cochrane criteria for metaanalyses). 5 Study statisticians randomized index subjects to one of six study arms. Family members of the index subjects were sent identical portal reminders to prevent confusion; we analyzed data for index subjects.

Portal Message Development
We framed portal messages following the Health Belief Model, 33 our prior portal study, 17 principles of health literacy, 34 and behavioral economics. 20,35,22 We first pre-tested several psychological and behavioral economic principles on Amazon's Mechanical Turk platform (MTurk) 36 (we could not test pre-commitment). We collected 3,896 US subjects, introduced the task, and randomly assigned them to one of 22 experimental arms-a control and a treatment message for 11 different psychological principles (Appendix 1 in the Supplementary Information). We asked five questions about the influence of particular messages on likelihood of influenza vaccination, constructed a composite measure of these five items, and asked how a message would change intention to vaccinate. We analyzed the impact of each method on vaccination intent, hypothetical change, and the composite measure. We found statistically significant and positive results for four psychological principles: gains framing (failure to vaccinate as a foregone gain rather than a loss), scarcity appeals (time is short to vaccinate), commission framing (choosing to vaccinate), and authoritative messenger (message from an authority).
Based upon these findings, we formally tested the impact of loss/gain messaging in the field because this principal is debated in the vaccination literature 31

Intervention
Study statisticians sent files to the health system's EHR team, defining which portal message to send per round. Patients were sent by system-generated notification via email or text (per patients' portal preference settings) that "A message from your doctor" was on the portal. Patients logged into the portal to read the message. System-generated messages were in English, were at <7th grade reading level per Flesch-Kincaid analysis, and included PCPs' names. All messages (Appendix 2 in the Supplementary Information) contained the following: (a) Dear "First Name"; (b) sentences following three behavioral principles, i.e., 20,35 scarcity ("time is running out to maximize the benefit of your flu vaccine"), appeal to authority (i.e.,"UCLA doctors … strongly recommend the flu vaccine each year for persons 65 years and older"), and commission (i.e., "Choosing to get vaccinated this season…"); (c) a link (and phone number) with "Call us to make an appointment or click here to request an appointment online"; and (d) a link to a website with information about influenza vaccine and video testimonials.
Loss framed messages stressed adverse consequences of choosing not to be vaccinated. Gain framed messages stressed positive consequences of choosing to get vaccinated.
Pre-commitment messages were sent in mid-October, asking patients if they planned to receive an influenza vaccine this season (Yes/No/Not sure). Identical portal reminders were sent in late October, November, and December (if loss or gain framed groups and if no vaccination was found by EHR 1 week before portal reminders).

Measures
Patient Characteristics. Patient characteristics from the EHR (Table 1) are as follows: age, sex, insurance at latest primary care visit, race, ethnicity, and influenza vaccination within two years.
Influenza Vaccination Data. We obtained influenza vaccination date and location from the EHR if administered at any UCLA H ealth site. UCLA practitioners can enter vaccination records manually for outside vaccinations. This information was merged into the EHR along with influenza vaccination data from (1) SureScripts (pharmacy benefits manager), (2) California Immunization Registry, and (3) CareEverywhere (Epic's information exchange application). Patients or proxies can also enter vaccination data via the portal. We integrated all data sources prior to analyses.
Since patients receiving our portal messages could enter vaccinations received elsewhere via a link; we included this information in a secondary outcome measure.
Outcome Measures. Primary Outcome. The primary study outcome was influenza vaccination between 10/01/2019 and 03/31/2020 (by EHR, after merging above sources). The primary analysis included all vaccinations except those selfreported by patients specifically in response to portal reminders as the control group did not have this opportunity for self-report, eliminating differential ascertainment. This analysis created a conservative bias since portal reminders may encourage patients to seek influenza vaccination at outside locations (e.g., workplace, pharmacies) not merged as above.

Statistical Analysis
We report descriptive statistics for patient characteristics. Primary analyses compared vaccination rates between study arms using mixed effects Poisson regression with robust standard errors, stratifying patients into young adults, older adults, and patients with diabetes. Models included a fixed effect for reminder arm (loss frame versus gain frame versus no message), a fixed effect for pre-commitment arm (message versus no message), random practice effects, and adjustment for patient characteristics (age, gender, insurance, race, ethnicity, and prior vaccination). Secondary subgroup analyses were performed by fitting separate models for each subgroup. For the primary analysis, a significance level of 0.017 was used. In all other analyses, we considered p-values below 0.05 as statistically significant.
As a secondary analysis, we used an instrumental variables approach to evaluate the effect of pre-commitment messages and loss/gain frame on subgroups who opened a portal message.

Practice and Patient Characteristics
We randomized 196,486 patients including young adults (N=145,166), older adults (N=29,795), and patients with diabetes (N=21,525) to one of six study groups (Figure 1). Most had private or Medicare insurance and were White (5-13% were Hispanic), and half to three-quarters had an influenza vaccination within 2 years (Table 1).

Primary Outcome: Influenza Vaccination
Influenza vaccination rates were low-37% young adults, 55% older adults, 60% patients with diabetes. There were no substantive differences by either pre-commitment (Table 2) or message framing (loss vs gain, Table 3) within any of the three strata.

Secondary Analyses
Subgroups. There were no substantive differences in influenza vaccination rates by either pre-commitment (Table 2) or message framing (loss vs gain, Table 3) within any of the pre-defined demographic subgroups or by prior influenza vaccination. Table 4 shows risk ratios (95% confidence intervals) for both unadjusted and adjusted analyses, comparing the effect of precommitment and framing on influenza vaccination rates for each stratum. There was no statistically or clinically significant impact of either pre-commitment or loss/gain framing on influenza vaccination rates. Table 4 also shows risk ratios for influenza vaccination by demographic characteristics. Among young adults and patients with diabetes, each added year of age was associated with a 1 percentage point improvement in vaccination, with opposite age effects for vaccination among older adults. For all 3 patient strata (young adults, older adults, patients with diabetes), vaccination rates this season were substantially higher if patients were vaccinated in prior seasons. Vaccination rates were higher among Asians, but lower among Black patients.

Vaccination Rates Including Patient Self-Reported Data in Response to Portal Reminders
One-third (33%) of influenza vaccines were from outside UCLA Health (n=27,157). If including community-based influenza vaccinations self-reported in response to the patient portal reminders (Appendix 3a-c in the Supplementary Information), patients sent pre-commitment reminders did not have higher vaccination rates than controls, but patients sent either loss and gain frame reminders had higher rates than controls by 1-3 percentage points (adjusted risks 1.02 to 1.07). The largest effects were for young adults.

Secondary Analyses
One possible reason for lack of impact is that only 12% of patients opened the pre-commitment portal messages sent in mid-October and 79%, 85%, and 62% opened the first, second, or third portal messages sent late October-December. Using the instrumental variable approach to evaluate effects on subgroups who opened the pre-commitment message, we estimate that the pre-commitment message raised vaccination rates by 4.5 percentage points (95% CI: −0.7 to +9.7) in the young adult group who viewed it, with no effect for older adults or diabetic patients. Using a similar instrumental variables approach, we did not find effects of loss/gain framing on patients who opened a portal reminder. Thus, even among patients who opened the portal reminders, their effect was minimal.
We evaluated whether the effects of the reminders or precommitment had differential effects within the 18-64-year age stratum. Tests for linear age interaction were not significant (p=0.35 for reminders, and p=0.46 for pre-commitment). We evaluated whether pre-commitment or portal reminders (loss/gain framing) increased influenza vaccination by 12/15/2019 (i.e., earlier than our primary end date of 03/31/2020); results were similar to those for the primary end date. Finally, we  evaluated the interventions' impact on patients who logged onto the portal more often than the median user in the prior year. The modest impact of pre-commitment reminders on young adults was limited to high portal users, but other findings did not change.

DISCUSSION
In this randomized trial across a large health system in the year prior to the pandemic, patient portal reminders for influenza vaccinations that were tailored to patient characteristics (young adults, older adults, and diabetic patients) and incorporated two behavioral science strategies (pre-commitment and loss/gain framing) had no effect on raising influenza vaccination rates. When we included additional self-reported vaccinations received outside the health system, both loss and gain framed messages among young adults and older adults had a modest impact with adjusted relative risks of vaccination (1.02 to 1.07) similar to other reminder/recall studies. 5 It is possible that portal reminders stimulated vaccination at external sites (e.g., workplace) that did not merge with the EHR, possibly because such sites were convenient. However, we would have expected a dampening but not elimination of the intervention effects for the main analysis that excluded these self-reported vaccinations.
This intervention was not designed to address vaccine hesitancy, which we suspect is the major reason for the intervention's small impact . 11,13,[37][38][39][40] Notably, during the same influenza season, our group found that for children, portal reminder messages had virtually no impact on first-dose influenza vaccination but did have a strong impact on receipt of second doses for which vaccine hesitancy is not an issue. 41 Interventions to directly address hesitancy are needed. We shaped the portal messages around findings from our MTurk survey in which both pre-commitment and message framing had positive effects on intent to vaccinate, yet these findings did not hold up in our pragmatic trial. More studies are needed to compare patients' hypothetical versus real world responses.
Study strengths include a large, pragmatic clinical trial, randomization to account for many unmeasured provider or practice-level factors, and capture of influenza vaccinations without ascertainment bias. One limitation is potential lack of generalizability from one health system, particularly if vaccine hesitancy is particularly high in this health system. Also, we used broad age categories; messages tailored to more granular age categories might have larger impact. We selected patients with diabetes because they are at higher risk and are identifiable using SUPREME criteria; messages tailored to other chronic diseases might have an impact. Further to this, since we already had multiple study arms, we did not formally evaluate other psychological principles. Other limitations include inability to generalize to non-portal users and inability to identify all vaccinations received outside the health system despite very strong data linkages between our EHR and pharmacy and other databases.
We conclude that patient reminders sent by a health system's patient portal that were tailored to patient characteristics (young adults, older adults, diabetes), and reminders that incorporated the behavioral science messaging strategies of pre-commitment and loss/gain framing, were not effective in raising pre-COVID influenza vaccination rates. Further studies to optimize patient reminders, including studies using alternative modalities such as text message reminders, are needed.
Acknowledgements: The following patients work in the UCLA Health System and provided assistance with obtaining data from the electronic medical records and sending the patient portal messages. They did not receive compensation.  Conflict of Interest: Dr. Humiston is a consultant to Sanofi Pasteur. Dr. Humiston's employer gets grant money for her work sponsored by the Pediatric Infectious Diseases Society Foundation, a not-for-profit organization. PIDS is funded for the project through unrestricted joint educational grants from Sanofi Pasteur US, Merck & Co., Inc., Pfizer, Inc., GlaxoSmithKline, and Seqirus USA, Inc. All other authors have no conflicts of interest to report.
Disclaimer: The funding agency had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons. org/licenses/by/4.0/.