Using Nudges to Reduce Missed Appointments in Primary Care and Mental Health: a Pragmatic Trial

Background Missed appointments (“no-shows”) are a persistent and costly problem in healthcare. Appointment reminders are widely used but usually do not include messages specifically designed to nudge patients to attend appointments. Objective To determine the effect of incorporating nudges into appointment reminder letters on measures of appointment attendance. Design Cluster randomized controlled pragmatic trial. Patients There were 27,540 patients with 49,598 primary care appointments, and 9420 patients with 38,945 mental health appointments, between October 15, 2020, and October 14, 2021, at one VA medical center and its satellite clinics that were eligible for analysis. Interventions Primary care (n = 231) and mental health (n = 215) providers were randomized to one of five study arms (four nudge arms and usual care as a control) using equal allocation. The nudge arms included varying combinations of brief messages developed with veteran input and based on concepts in behavioral science, including social norms, specific behavioral instructions, and consequences of missing appointments. Main Measures Primary and secondary outcomes were missed appointments and canceled appointments, respectively. Statistical Analysis Results are based on logistic regression models adjusting for demographic and clinical characteristics, and clustering for clinics and patients. Key Results Missed appointment rates in study arms ranged from 10.5 to 12.1% in primary care clinics and 18.0 to 21.9% in mental health clinics. There was no effect of nudges on missed appointment rate in primary care (OR = 1.14, 95%CI = 0.96–1.36, p = 0.15) or mental health (OR = 1.20, 95%CI = 0.90–1.60, p = 0.21) clinics, when comparing the nudge arms to the control arm. When comparing individual nudge arms, no differences in missed appointment rates nor cancellation rates were observed. Conclusions Appointment reminder letters incorporating brief behavioral nudges were ineffective in improving appointment attendance in VA primary care or mental health clinics. More complex or intensive interventions may be necessary to significantly reduce missed appointments below their current rates. Trial Number ClinicalTrials.gov, Trial number NCT03850431. Supplementary Information The online version contains supplementary material available at 10.1007/s11606-023-08131-5.


BACKGROUND
Missed appointments, frequently referred to as "no-shows," are a longstanding challenge to providers and administrators in every healthcare system. In the Department of Veterans Affairs (VA), the largest integrated healthcare system in the USA, the national missed appointment rate was 14.9% from October 2020 to September 2021, resulting in 10,109,148 missed appointments. 1 Patients prone to miss appointments are a vulnerable population, known to have higher rates of preventable hospitalizations, emergency department visits, and all-cause mortality. 2,3 Like many behaviors, the potential reasons behind a missed appointment are myriad, but broadly speaking they can include patient-related characteristics (e.g., history of psychiatric disorders 4 ), features of the healthcare system (e.g., scheduling systems 5 ), and factors that lie in between the two (e.g., how long prior to the scheduled date the appointment was scheduled). 6 Nonetheless, simply forgetting the appointment is perhaps the single most common reason for a missed appointment. 7 Given this, it is not surprising that appointment reminders are employed more or less universally across healthcare systems-a straightforward, first line of defense against missed appointments. Traditionally, appointment reminders provide a "bare bones" message about the time, date, and place of an appointment, which is more or less how they have been for at least the last 40 years. [8][9][10] One of the core ideas of behavioral economics is the idea of supporting behavior change through the use of nudges: small alterations in how options are presented that are designed to influence a person's decision making without restricting their choices. 11,12 Several studies have found promising impacts on appointment attendance through application of nudges that leveraged principles such as people's general desire to avoid physical or emotional losses ("loss aversion") and adhere to prevalent social norms. [13][14][15][16] Specifically, a randomized controlled trial in outpatient specialty clinics in the UK's National Health Service found that specifying the cost of a missed appointment to the healthcare system significantly reduced the rate of missed appointments, and including the descriptive social norm that most patients attended their appointment significantly increased the rate of canceling appointments in advance, thus potentially averting missed appointments. 17 An Israeli study found several messages effective at reducing missed appointments, the most effective being a text message appointment reminder designed to evoke emotional guilt ("Not showing up to your appointment without canceling in advance delays hospital treatment for those who need medical aid. 18 ) Finally, in the USA, a randomized trial of 360 veterans at a single VA medical center who were referred to specialty mental health for depression found that patients who received an appointment reminder letter containing a bolded, gain-framed message ("If you go to your appointment, you will learn ways to improve your mood and emotional well-being. Also, if you see your provider, he/she will be able to work with you to help you get the most out of your treatment.") were more likely to attend their appointments, compared to those who received a routine letter without the additional message. 19 To date, there have been no large-scale trials of behavioral economics-informed appointment reminders in the USA. Moreover, the effect of other types of nudges, or combining nudges, is not known. The aim of this study was to conduct a large-scale pragmatic trial within the VA to evaluate the effect of a series of nudges on missed and canceled appointments. We hypothesized that the intervention, as compared with usual care, would have a lower missed appointment rate and higher canceled appointment rate. We also hypothesized that effect size of nudges would vary by intervention arm such that some types of nudges would have a larger proportional effect than others.

Design and Setting
We conducted a cluster randomized controlled pragmatic trial at the VA Portland Health Care System, which cares for approximately 85,000 patients across Oregon and southwest Washington, including two medical centers (Portland and Vancouver) and six satellite clinics (Hillsboro, Fairview, West Linn, Salem, Bend, and North Coast).
Pragmatic trial features included broad eligibility criteria, intervention implementation integrated with usual care, usual care as the comparison condition, and outcome assessment using electronic health record data. 20,21 The trial began December 2019 but was halted in March 2020 due to the COVID-19 pandemic. It resumed October 2020 and lasted 1 year until October 2021. Due to cohort effects and a small sample size, pre-COVID trial data were not included. The VA Portland Healthcare System Institutional Review Board and Research and Development Service approved the study, which was registered with ClinicalTrials.gov (Trial number NCT03850431) prior to initiation.
For inclusion, outpatient appointments had to be as follows: (1) in either primary care or mental health; and (2) scheduled at least 2 weeks before the appointment date (the minimum length of time used to determine whether an appointment reminder is mailed). We used "stop codes," 3-digit identifiers used within VA, 22 to identify appointments as primary care (codes: 322, 323, and 338) or mental health (codes: 502, 513, 527, 534, 542, 545, and 562). Patients were included if they had at least one appointment meeting the above criteria.

Randomization and Masking
We conducted randomization at the provider (not patient) level by allocating to one of four nudge (intervention) arms or control arm (1:1:1:1:1), and stratifying by setting (primary care or mental health) and location (Portland, Vancouver, or other). A statistician (MN) masked to intervention information used block randomization with random block sizes of 5, 10, or 15, assigning providers to one of five study arms using the blockrand (v 1.5) R package 23 .

Interventions
We developed four nudge arms using a combination of theory, empirical data, and veteran input. We began with candidate messages organized around broad concepts in behavioral science that seemed plausible motivators for a patient to attend and/or cancel their appointment: social norms, behavioral instructions and intentions, consequences for self, and consequences for others (Table 1). We then employed a user-centered design (UCD) process as a strategy to make our interventions patient-centered and minimize potential risks and unintended consequences as reported previously. 24 During the UCD process, we conducted iterative waves of interviews with veterans, eliciting feedback on nudge language and the draft interventions. Interviews influenced key aspects of intervention content, ultimately resulting in removal or revision of several candidate nudge messages, generation of new messages, and selection of specific combinations of messages for each intervention. The four finalized interventions are summarized in Table 1. To illustrate how the intervention appeared in practice, Fig. 1 presents the letter for arm D, which contained a combination of all the nudges, as seen by a veteran with a primary care appointment.
Consistent with our pragmatic trial approach, interventions were delivered as they would be in routine clinical practice. This meant scheduling staff, who were unaware of study hypotheses or procedures, were responsible for preparation and mailing of templated appointment reminders. For mental health clinics, routine practice included not only an appointment reminder but also a letter after a missed appointment; therefore, nudge messages for each intervention were incorporated into these letters too. One unmasked research team member (EM) worked with clinic staff to modify, maintain, and audit letter templates.

Control Arm
The control arm consisted of routine appointment reminder letters providing the clinic name, location, provider, date and time of appointment, and contact information. Since reminder letters were the focus of this trial, patients in all study arms continued to receive automated phone (audio), text, and postcard reminders (primary care only) without the nudge messages.

Main Measures
We used the VA Corporate Data Warehouse to identify eligible appointments and coded as completed, canceled (by patient), or missed ("no-show"). Appointments that could not be classified due to incomplete data were excluded (0.05%). The primary outcome was whether a patient missed an appointment. In accordance with VA policy, an appointment was considered missed if marked by clinic Caring [48][49][50] We're here for you Consequences for Self 13,14 If you miss your appointment, you may have to wait a while to be seen Attending appointments lowers your chances of being hospitalized Behavioral Instructions and Intentions 51,52 Need to change or cancel your appointment? Take these 2 simple steps today: 1) Call the clinic at 503-555-5555 2) Give your name, last 4, and appointment information. It's fine to leave a message Consequences for Others (Arm C) Caring 48-50 1) We're here for you Behavioral Instructions and Intentions 51,52 + Consequences for Others 53,54 If you need to change or cancel your appointment, call now so we can help another Veteran in need Take these 2 simple steps today: 1) Call the clinic at 503-555-5555 2) Give your name, last 4, and appointment information. It's fine to leave a message." Social Norms (Arm E) Social Norms 15,16 Most Veterans make a point to attend their VA appointments. If they can't make their appointments, most Veterans also make an effort to let us know Behavioral Instructions and Intentions 51,52 Need to change or cancel your appointment? Take these 2 simple steps today: 1) Call the clinic at 503-555-5555 2) Give your name, last 4, and appointment information. It's fine to leave a message Combination of All Nudges (Arm D) Caring [48][49][50] We're here for you Social Norms 15,16 Most Veterans make a point to attend their VA appointments. If they can't make their appointment, most Veterans also make an effort to let us know Behavioral Instructions and Intentions 51,52 + Consequences for Others 53,54 If you need to change or cancel your appointment, call now so we can help another Veteran in need Take these 2 simple steps today: 1) Call the clinic at 503-555-5555 2) Give your name, last 4, and appointment information. It's fine to leave a message Consequences for Self 13,14 If you miss your appointment, you may have to wait a while to be seen Attending appointments lowers your chances of being hospitalized staff as a "no-show" or if canceled by the patient or clinic after the appointment time. The missed appointment rate equals the number of missed appointments divided by the number of completed and missed appointments. Our secondary outcome was whether a patient canceled their appointment. As clinics sometimes cancel appointments for administrative reasons and the focus of our trial was patient behavior, we examined only appointments canceled by patients with a timestamp prior to the scheduled appointment time and calculated a cancelation rate accordingly. The canceled appointment rate equals the number of canceled appointments divided by the number of completed, canceled, and missed appointments.

Statistical Analyses
We first compared patient demographics and clinical characteristics between the control and intervention groups (both combined and individually), using counts and percentages for categorical variables and means and standard deviations (SD) for quantitative variables. We used standardized mean differences (SMD) 25 to compare the effect size between the control and combined intervention groups. For patients with multiple appointments during the study period, we used data from their first appointment in primary care and mental health, respectively, to summarize baseline patient-level characteristics.
The primary analyses compared the odds of a missed appointment between the control and the combined intervention groups using appointment-level logistic regression models with non-nested clustering for repeated appointments among providers and patients 26 . Primary care and mental health appointments were analyzed separately. We ran unadjusted and adjusted models, which included age, gender (missing values for self-identified gender were imputed with sex), race (collapsed as white, non-white, and unknown), ethnicity, rurality, VA disability rating, 27 mental health diagnosis in the prior 2 years (i.e., depression, PTSD, substance use disorder), Elixhauser comorbidity score, 28 Care Assessment Need (CAN) score 29 (estimated probability of readmission or death within 90 days in primary care sample only), provider type, appointment age, appointment modality (virtual vs. in-person), appointment location (Portland, Vancouver, or other 6 sites combined), and the number of prior visits for the patient at the provider's clinic location in the past 2 years. We selected covariates based on significance in our models and prior literature demonstrating associations with missed appointments (e.g., demographic characteristics, rurality, disability rating as a proxy for socioeconomic and health insurance status, mental health history, appointment age). 4,[30][31][32] Variance inflation factors were checked for multicollinearity, and all were less than 2.15. Patient characteristics were non-varying, using the values from the first appointment if they had repeated appointments in either setting. We also ran separate models with intervention × gender and intervention × modality interactions. In addition to odds ratios, we calculated mean predicted probabilities and 95% confidence intervals for each intervention via 10,000 Monte Carlo simulations, using reference levels for categorical variables and mean values for continuous covariates.
We conducted two sensitivity analyses. To address the possibility of diminished effect after repeated exposure to the intervention, the first sensitivity analysis was restricted to each patient's first appointment (separately for primary care and mental health). To provide a more accurate estimate of the standard errors, we used logistic generalized estimating equation models with clustering for providers using an exchangeable covariance structure. 33,34 The second sensitivity analysis was restricted to primary care providers (in primary care) and therapists and prescribers (in mental health); this was done to avoid including appointments with other providers (e.g., nurses, pharmacists) who frequently have "group appointments" that did not receive appointment reminder letters. All analyses were repeated using all five study arms with the control group as the reference level, and again for the cancelation outcome. Analyses were run in R (v. 4.1.2) 35 using the smd (v. 0.6.6), 36 lmtest (v. 0.9-40), 37 sandwich (v. 3.0.2) 26,38 (vcovCL for two-way clustering), and geepack (v. 1.3.9) 39-41 packages. All tests were two-sided with a significance threshold of 5%.

Patients and Clinics
Two hundred thirty-one primary care providers had 27,540 unique patients with a total of 49,598 appointments in the trial. Likewise, 215 mental health providers had 9420 unique patients with a total of 38,945 appointments. Thirtysix percent of primary care (70% of mental health) patients had more than one appointment included in the study. Of these, 13% (29%) had appointments with multiple providers. Because study arm randomization was at the provider level, this led to 11% (25%) of patients having appointments in more than one study arm within primary care and/or mental health. Consort flowcharts are presented in Fig. 2. Tables 2 and 3, most patients were white, male, and not Hispanic or Latino, and resided in urban areas; mean age was 63.7 years (primary care) and 50.0 (mental health).

Primary Outcome
In both primary care and mental health, there was no significant effect of nudges on missed appointment rate (Table 4).

Secondary Outcome
In both primary care and mental health, there was no effect of nudges on canceled appointment rate (Table 4).
In primary care, the cancelation rates for the intervention arms (all four nudge arms combined) and control arm, respectively, were 7.3% and 6.0%, again with a "reversed" OR of 0.92 (95%CI = 0.83-1.03, p = 0.15) when comparing the intervention to the control in the adjusted logistic regression model. When comparing nudge arms individually to the control arm the OR's ranged from 0.89 to 0.95, all nonsignificant.
In mental health, cancelation rates for the intervention arms (all four nudge arms combined) and control arm, respectively, were 8.8% and 10.2%, with an OR of 0.83 (95%CI = 0.65-1.06, p = 0.14) when comparing the intervention to the control group in the adjusted logistic regression model. When comparing nudge arms individually to the control arm the OR's ranged from 0.74 to 0.93, all nonsignificant.

Additional Analyses
In primary care, modality of appointment (OR = 0.74, 95%CI = 0.57-0.98, p = 0.03) and gender (OR = 0.71, 95%CI = 0.53-0.95, p = 0.02) both moderated the missed appointment rate, such that missed visits were more likely in the intervention group compared to the control group for virtual appointments and men, while less likely for in-person appointments and women. For the cancelation rate in primary care, only modality of appointment was a significant moderator (OR = 1.34, 95%CI = 1.06-1.69, p = 0.02), where virtual appointments in the nudge arms were less likely to be canceled than in the control group, and more likely for in-person appointments. In mental health, the only significant moderator was gender for canceled appointment rate (OR = 0.73, 95%CI = 0.53-0.99, p = 0.04), where both men and women in the intervention group were less likely to cancel appointments than in the control group, but men more so than women. For sensitivity analyses, results were generally very similar with two minor exceptions. First, in primary care, when restricting to the patients' first appointment, the canceled appointment rate was lower for the intervention arms compared to the control arm (OR = 0.88, 95%CI = 0.78-1.00, p = 0.04) in the adjusted model. Second, in mental health, when restricting to the patients' first appointment, the missed appointment rate for arm C (Consequences for Others) was lower compared to that of the control arm (OR = 0.67, 95%CI = 0.45-0.98, p = 0.04).

Table 2 Baseline Characteristics at the Patient Level for Primary Carê
SMD, standardized mean differences * Mean (SD); n (%) † Self-identified gender as indicated in the VA electronic health record, with missing values imputed with sex ‡ Care Assessment Need score reflecting estimated probability of readmission or death at 90 days § Number of prior visits for the patient at the provider's clinic location in the past 2 years

DISCUSSION
In this yearlong pragmatic trial composed of tens of thousands of VA patients and their appointments in primary care and mental health, we found no effect of incorporating nudges in appointment reminder letters on measures relevant to outpatient appointment attendance, specifically the rate of missed appointments and rate of canceled appointments. The lack of effectiveness of incorporating nudges was seen consistently in comparing all four intervention arms to the control, in comparing each individual intervention arm to the control arm, and in both primary care and mental health, the two largest clinic types in VA. While this was a negative trial, we believe our findings are revealing and important to disseminate, particularly in light of recent analyses suggesting substantial publication bias in the realm of nudge interventions, and perhaps even null effects after accounting for this bias. 42,43 As a pragmatic trial, our findings may have been impacted by several real-world issues within the VA healthcare system. First, a number of veterans were in more than one study arm, particularly among patients in mental health. Psychiatric comorbidity among VA-using veterans is common, and this likely contributed to patients being seen by multiple mental health providers. 44 Second, patients' exposure to the intervention was limited compared to what they would have received in a more traditional efficacy trial. For instance, formatting restrictions in the systems used to create reminder letters prevented us from doing more to highlight messages. Furthermore, our trial was only able to incorporate nudges into appointment reminder letters. This left other reminders (text, audio, postcard, and-for virtual appointments that became commonplace during the trial-email) untouched by the intervention. As a consequence, patients may have experienced a diluted effect of the nudge due to varying messages being delivered for the same appointment, and they may also have experienced reminder fatigue, 45 leading some to effectively ignore reminders altogether.
Our trial was also susceptible to a variety of "on-theground" changes in healthcare operations that occurred over the course of the trial. COVID-related impacts included a large-scale switch to virtual appointments, and processes related to this change may have diminished relevance and impact of the trial's different patterns of outreach to patients. For instance, during the trial it became standard practice for providers to call a patient if they did Table 4 Odds Ratios for Missed Appointments and Canceled Appointments in Primary Care and Mental Health * N, number of appointments. For calculation of missed appointments, canceled appointments are not included † Adjusted models include covariates for age, gender, race, ethnicity, rurality, VA service connection, depression diagnosis in prior 2 years, PTSD diagnosis in prior 2 years, substance use disorder diagnosis in prior 2 years, Elixhauser comorbidity score, CAN 90 score (PC models only), number of prior visits, type of provider, appointment age, appointment modality (in-person vs. virtual), and appointment location (Portland, Vancouver, other) not log into a virtual appointment, thereby potentially creating a new intervention and obviating a missed appointment. We were also unable to quantify the fidelity of the intervention and cannot guarantee that the offices preparing and sending the appointment reminder letters consistently followed standard procedures. While our pragmatic trial was large, it was nonetheless limited to a single VA healthcare system among patients who were primarily White, male, and older adults. Findings could differ in other patient populations.
It is also plausible that no intervention focused simply on appointment reminders may be able to create significant change in appointment attendance. Financial incentives-whether framed as a gain or loss-can be used to influence health behaviors, though their cost-effectiveness would need to be considered. 46 A recent systematic review of missed appointments in diabetes clinic appointments found that qualitative studies suggest psychosocial factors playing a prominent role. 47 It may be, then, that even in the best of circumstances, nudges embedded into appointment reminders are inadequate to resolve the mixture of emotional, cognitive, and behavioral factors involved in triggering missed appointments. In that case, it may be up to health systems and their researchers to determine whether to deploy and test interventions that are more costly or higher intensity, or both. Perhaps a 10% missed appointment rate in primary care or a 20% missed appointment rate in mental health clinics is as good as can be practically achieved without addressing the larger social and contextual issues underling missed appointments.
In conclusion, in this large pragmatic trial in primary care and mental health clinics, we found appointment reminder letters incorporating brief behavioral nudges did not affect appointment attendance. Attempts to reduce missed appointments or increase cancelation of appointments that are no longer necessary likely require more complex or intensive interventions.