BACKGROUND

Missed appointments, frequently referred to as “no-shows,” are a longstanding challenge to providers and administrators in every healthcare system. In the Department of Veterans Affairs (VA), the largest integrated healthcare system in the USA, the national missed appointment rate was 14.9% from October 2020 to September 2021, resulting in 10,109,148 missed appointments.1 Patients prone to miss appointments are a vulnerable population, known to have higher rates of preventable hospitalizations, emergency department visits, and all-cause mortality.2,3

Like many behaviors, the potential reasons behind a missed appointment are myriad, but broadly speaking they can include patient-related characteristics (e.g., history of psychiatric disorders4), features of the healthcare system (e.g., scheduling systems5), and factors that lie in between the two (e.g., how long prior to the scheduled date the appointment was scheduled).6 Nonetheless, simply forgetting the appointment is perhaps the single most common reason for a missed appointment.7 Given this, it is not surprising that appointment reminders are employed more or less universally across healthcare systems—a straightforward, first line of defense against missed appointments. Traditionally, appointment reminders provide a “bare bones” message about the time, date, and place of an appointment, which is more or less how they have been for at least the last 40 years.8,9,10

One of the core ideas of behavioral economics is the idea of supporting behavior change through the use of nudges: small alterations in how options are presented that are designed to influence a person’s decision making without restricting their choices.11,12 Several studies have found promising impacts on appointment attendance through application of nudges that leveraged principles such as people’s general desire to avoid physical or emotional losses (“loss aversion”) and adhere to prevalent social norms.13,14,15,16 Specifically, a randomized controlled trial in outpatient specialty clinics in the UK’s National Health Service found that specifying the cost of a missed appointment to the healthcare system significantly reduced the rate of missed appointments, and including the descriptive social norm that most patients attended their appointment significantly increased the rate of canceling appointments in advance, thus potentially averting missed appointments.17 An Israeli study found several messages effective at reducing missed appointments, the most effective being a text message appointment reminder designed to evoke emotional guilt (“Not showing up to your appointment without canceling in advance delays hospital treatment for those who need medical aid.18) Finally, in the USA, a randomized trial of 360 veterans at a single VA medical center who were referred to specialty mental health for depression found that patients who received an appointment reminder letter containing a bolded, gain-framed message (“If you go to your appointment, you will learn ways to improve your mood and emotional well-being. Also, if you see your provider, he/she will be able to work with you to help you get the most out of your treatment.”) were more likely to attend their appointments, compared to those who received a routine letter without the additional message.19

To date, there have been no large-scale trials of behavioral economics-informed appointment reminders in the USA. Moreover, the effect of other types of nudges, or combining nudges, is not known. The aim of this study was to conduct a large-scale pragmatic trial within the VA to evaluate the effect of a series of nudges on missed and canceled appointments. We hypothesized that the intervention, as compared with usual care, would have a lower missed appointment rate and higher canceled appointment rate. We also hypothesized that effect size of nudges would vary by intervention arm such that some types of nudges would have a larger proportional effect than others.

METHODS

Design and Setting

We conducted a cluster randomized controlled pragmatic trial at the VA Portland Health Care System, which cares for approximately 85,000 patients across Oregon and southwest Washington, including two medical centers (Portland and Vancouver) and six satellite clinics (Hillsboro, Fairview, West Linn, Salem, Bend, and North Coast).

Pragmatic trial features included broad eligibility criteria, intervention implementation integrated with usual care, usual care as the comparison condition, and outcome assessment using electronic health record data.20,21 The trial began December 2019 but was halted in March 2020 due to the COVID-19 pandemic. It resumed October 2020 and lasted 1 year until October 2021. Due to cohort effects and a small sample size, pre-COVID trial data were not included. The VA Portland Healthcare System Institutional Review Board and Research and Development Service approved the study, which was registered with ClinicalTrials.gov (Trial number NCT03850431) prior to initiation.

For inclusion, outpatient appointments had to be as follows: (1) in either primary care or mental health; and (2) scheduled at least 2 weeks before the appointment date (the minimum length of time used to determine whether an appointment reminder is mailed). We used “stop codes,” 3-digit identifiers used within VA,22 to identify appointments as primary care (codes: 322, 323, and 338) or mental health (codes: 502, 513, 527, 534, 542, 545, and 562). Patients were included if they had at least one appointment meeting the above criteria.

Randomization and Masking

We conducted randomization at the provider (not patient) level by allocating to one of four nudge (intervention) arms or control arm (1:1:1:1:1), and stratifying by setting (primary care or mental health) and location (Portland, Vancouver, or other). A statistician (MN) masked to intervention information used block randomization with random block sizes of 5, 10, or 15, assigning providers to one of five study arms using the blockrand (v 1.5) R package23.

Interventions

We developed four nudge arms using a combination of theory, empirical data, and veteran input. We began with candidate messages organized around broad concepts in behavioral science that seemed plausible motivators for a patient to attend and/or cancel their appointment: social norms, behavioral instructions and intentions, consequences for self, and consequences for others (Table 1). We then employed a user-centered design (UCD) process as a strategy to make our interventions patient-centered and minimize potential risks and unintended consequences as reported previously.24 During the UCD process, we conducted iterative waves of interviews with veterans, eliciting feedback on nudge language and the draft interventions. Interviews influenced key aspects of intervention content, ultimately resulting in removal or revision of several candidate nudge messages, generation of new messages, and selection of specific combinations of messages for each intervention. The four finalized interventions are summarized in Table 1. To illustrate how the intervention appeared in practice, Fig. 1 presents the letter for arm D, which contained a combination of all the nudges, as seen by a veteran with a primary care appointment.

Table 1 Description of Study Arms
Full size table
Figure 1
figure 1

Example intervention appointment letter (arm D)

Full size image

Consistent with our pragmatic trial approach, interventions were delivered as they would be in routine clinical practice. This meant scheduling staff, who were unaware of study hypotheses or procedures, were responsible for preparation and mailing of templated appointment reminders. For mental health clinics, routine practice included not only an appointment reminder but also a letter after a missed appointment; therefore, nudge messages for each intervention were incorporated into these letters too. One unmasked research team member (EM) worked with clinic staff to modify, maintain, and audit letter templates.

Control Arm

The control arm consisted of routine appointment reminder letters providing the clinic name, location, provider, date and time of appointment, and contact information. Since reminder letters were the focus of this trial, patients in all study arms continued to receive automated phone (audio), text, and postcard reminders (primary care only) without the nudge messages.

Main Measures

We used the VA Corporate Data Warehouse to identify eligible appointments and coded as completed, canceled (by patient), or missed (“no-show”). Appointments that could not be classified due to incomplete data were excluded (0.05%). The primary outcome was whether a patient missed an appointment. In accordance with VA policy, an appointment was considered missed if marked by clinic staff as a “no-show” or if canceled by the patient or clinic after the appointment time. The missed appointment rate equals the number of missed appointments divided by the number of completed and missed appointments. Our secondary outcome was whether a patient canceled their appointment. As clinics sometimes cancel appointments for administrative reasons and the focus of our trial was patient behavior, we examined only appointments canceled by patients with a timestamp prior to the scheduled appointment time and calculated a cancelation rate accordingly. The canceled appointment rate equals the number of canceled appointments divided by the number of completed, canceled, and missed appointments.

Statistical Analyses

We first compared patient demographics and clinical characteristics between the control and intervention groups (both combined and individually), using counts and percentages for categorical variables and means and standard deviations (SD) for quantitative variables. We used standardized mean differences (SMD)25 to compare the effect size between the control and combined intervention groups. For patients with multiple appointments during the study period, we used data from their first appointment in primary care and mental health, respectively, to summarize baseline patient-level characteristics.

The primary analyses compared the odds of a missed appointment between the control and the combined intervention groups using appointment-level logistic regression models with non-nested clustering for repeated appointments among providers and patients26. Primary care and mental health appointments were analyzed separately. We ran unadjusted and adjusted models, which included age, gender (missing values for self-identified gender were imputed with sex), race (collapsed as white, non-white, and unknown), ethnicity, rurality, VA disability rating,27 mental health diagnosis in the prior 2 years (i.e., depression, PTSD, substance use disorder), Elixhauser comorbidity score,28 Care Assessment Need (CAN) score29 (estimated probability of readmission or death within 90 days in primary care sample only), provider type, appointment age, appointment modality (virtual vs. in-person), appointment location (Portland, Vancouver, or other 6 sites combined), and the number of prior visits for the patient at the provider’s clinic location in the past 2 years. We selected covariates based on significance in our models and prior literature demonstrating associations with missed appointments (e.g., demographic characteristics, rurality, disability rating as a proxy for socioeconomic and health insurance status, mental health history, appointment age).4,30,31,32 Variance inflation factors were checked for multicollinearity, and all were less than 2.15. Patient characteristics were non-varying, using the values from the first appointment if they had repeated appointments in either setting. We also ran separate models with intervention × gender and intervention × modality interactions. In addition to odds ratios, we calculated mean predicted probabilities and 95% confidence intervals for each intervention via 10,000 Monte Carlo simulations, using reference levels for categorical variables and mean values for continuous covariates.

We conducted two sensitivity analyses. To address the possibility of diminished effect after repeated exposure to the intervention, the first sensitivity analysis was restricted to each patient’s first appointment (separately for primary care and mental health). To provide a more accurate estimate of the standard errors, we used logistic generalized estimating equation models with clustering for providers using an exchangeable covariance structure.33,34 The second sensitivity analysis was restricted to primary care providers (in primary care) and therapists and prescribers (in mental health); this was done to avoid including appointments with other providers (e.g., nurses, pharmacists) who frequently have “group appointments” that did not receive appointment reminder letters. All analyses were repeated using all five study arms with the control group as the reference level, and again for the cancelation outcome. Analyses were run in R (v. 4.1.2)35 using the smd (v. 0.6.6),36 lmtest (v. 0.9–40),37 sandwich (v. 3.0.2)26,38 (vcovCL for two-way clustering), and geepack (v. 1.3.9)39,40,41 packages. All tests were two-sided with a significance threshold of 5%.

RESULTS

Patients and Clinics

Two hundred thirty-one primary care providers had 27,540 unique patients with a total of 49,598 appointments in the trial. Likewise, 215 mental health providers had 9420 unique patients with a total of 38,945 appointments. Thirty-six percent of primary care (70% of mental health) patients had more than one appointment included in the study. Of these, 13% (29%) had appointments with multiple providers. Because study arm randomization was at the provider level, this led to 11% (25%) of patients having appointments in more than one study arm within primary care and/or mental health. Consort flowcharts are presented in Fig. 2.

Figure 2
figure 2

Consort flow diagrams for primary care (a) and mental health (b). Consort Flow Diagram for Primary Care (2a). * Some patients had appointments in multiple arms of the study, so the sum of patients in each arm will not equal the number of unique patients in the trial. † Ineligible appointments refer to appointments (1) made fewer than two weeks in advance (N = 74,148 across all arms) or (2) appointments canceled by the clinic before the appointment time (e.g., provider out sick) or created for operational, not clinical, purposes (N = 5,815 across all arms). ‡ Some providers assigned an intervention reminder letter did not have appointments that met trial criteria. There were 231 providers and 27,540 unique patients across all arms. Consort Flow Diagram for Mental Health (2b). * Some patients had appointments in multiple arms of the study, so the sum of patients in each arm will not equal the number of unique patients in the trial. † Ineligible appointments refer to appointments (1) made fewer than two weeks in advance (N = 66,821 across all arms) or (2) appointments canceled by the clinic before the appointment time (e.g., provider out sick) or created for operational, not clinical, purposes (N = 4,324 across all arms). ‡ Some providers assigned an intervention reminder letter did not have appointments that met trial criteria. There were providers clinics and 9,420 unique patients across all arms

Full size image

As shown in Tables 2 and 3, most patients were white, male, and not Hispanic or Latino, and resided in urban areas; mean age was 63.7 years (primary care) and 50.0 (mental health).

Table 2 Baseline Characteristics at the Patient Level for Primary Care
Full size table
Table 3 Baseline Characteristics at the Patient Level for Mental Health
Full size table

Primary Outcome

In both primary care and mental health, there was no significant effect of nudges on missed appointment rate (Table 4).

Table 4 Odds Ratios for Missed Appointments and Canceled Appointments in Primary Care and Mental Health
Full size table

In primary care, the missed appointment rates for the intervention arms (all four nudge arms combined) and control group, respectively, were 11.1% and 11.9%. The odds ratio (OR) from the adjusted logistic regression model comparing the intervention arms to the control arm was 1.14 (95% CI = 0.96–1.36, p = 0.15; control arm: 10.9%, 95% CI = 10.0–13.7%; all nudge arms: 12.2%, 95% = 11.2–15.2%), which was the reverse relationship of the unadjusted percentages. When comparing nudge arms individually to the control arm, the OR’s ranged from 1.11 to 1.18 and were nonsignificant (control arm: 9.3%, 95% CI = 8.4–11.9%; nudge arms varied from 10.2 to 10.8%).

In mental health, the missed appointment rates for the intervention arms (all four nudge arms combined) and control group, respectively, were 20.1% and 18.0%. The OR from the adjusted logistic regression model comparing the intervention arms to the control arm was 1.20 (95%CI = 0.90–1.60, p = 0.21; control arm: 18.9%, 95% CI = 17.1–24.4%; all nudge arms: 21.8%, 95% = 20.4–25.9%). When comparing nudge arms individually to the control arm, the OR’s ranged from 1.14 to 1.33 and were nonsignificant (control arm: 19.2%, 95% CI = 17.3–24.9%; nudge arms varied from 21.4 to 23.9%).

Secondary Outcome

In both primary care and mental health, there was no effect of nudges on canceled appointment rate (Table 4).

In primary care, the cancelation rates for the intervention arms (all four nudge arms combined) and control arm, respectively, were 7.3% and 6.0%, again with a “reversed” OR of 0.92 (95%CI = 0.83–1.03, p = 0.15) when comparing the intervention to the control in the adjusted logistic regression model. When comparing nudge arms individually to the control arm the OR’s ranged from 0.89 to 0.95, all nonsignificant.

In mental health, cancelation rates for the intervention arms (all four nudge arms combined) and control arm, respectively, were 8.8% and 10.2%, with an OR of 0.83 (95%CI = 0.65–1.06, p = 0.14) when comparing the intervention to the control group in the adjusted logistic regression model. When comparing nudge arms individually to the control arm the OR’s ranged from 0.74 to 0.93, all nonsignificant.

Additional Analyses

In primary care, modality of appointment (OR = 0.74, 95%CI = 0.57–0.98, p = 0.03) and gender (OR = 0.71, 95%CI = 0.53–0.95, p = 0.02) both moderated the missed appointment rate, such that missed visits were more likely in the intervention group compared to the control group for virtual appointments and men, while less likely for in-person appointments and women. For the cancelation rate in primary care, only modality of appointment was a significant moderator (OR = 1.34, 95%CI = 1.06–1.69, p = 0.02), where virtual appointments in the nudge arms were less likely to be canceled than in the control group, and more likely for in-person appointments. In mental health, the only significant moderator was gender for canceled appointment rate (OR = 0.73, 95%CI = 0.53–0.99, p = 0.04), where both men and women in the intervention group were less likely to cancel appointments than in the control group, but men more so than women.

For sensitivity analyses, results were generally very similar with two minor exceptions. First, in primary care, when restricting to the patients’ first appointment, the canceled appointment rate was lower for the intervention arms compared to the control arm (OR = 0.88, 95%CI = 0.78–1.00, p = 0.04) in the adjusted model. Second, in mental health, when restricting to the patients’ first appointment, the missed appointment rate for arm C (Consequences for Others) was lower compared to that of the control arm (OR = 0.67, 95%CI = 0.45–0.98, p = 0.04).

DISCUSSION

In this yearlong pragmatic trial composed of tens of thousands of VA patients and their appointments in primary care and mental health, we found no effect of incorporating nudges in appointment reminder letters on measures relevant to outpatient appointment attendance, specifically the rate of missed appointments and rate of canceled appointments. The lack of effectiveness of incorporating nudges was seen consistently in comparing all four intervention arms to the control, in comparing each individual intervention arm to the control arm, and in both primary care and mental health, the two largest clinic types in VA. While this was a negative trial, we believe our findings are revealing and important to disseminate, particularly in light of recent analyses suggesting substantial publication bias in the realm of nudge interventions, and perhaps even null effects after accounting for this bias.42,43

As a pragmatic trial, our findings may have been impacted by several real-world issues within the VA healthcare system. First, a number of veterans were in more than one study arm, particularly among patients in mental health. Psychiatric comorbidity among VA-using veterans is common, and this likely contributed to patients being seen by multiple mental health providers.44 Second, patients’ exposure to the intervention was limited compared to what they would have received in a more traditional efficacy trial. For instance, formatting restrictions in the systems used to create reminder letters prevented us from doing more to highlight messages. Furthermore, our trial was only able to incorporate nudges into appointment reminder letters. This left other reminders (text, audio, postcard, and—for virtual appointments that became commonplace during the trial—email) untouched by the intervention. As a consequence, patients may have experienced a diluted effect of the nudge due to varying messages being delivered for the same appointment, and they may also have experienced reminder fatigue,45 leading some to effectively ignore reminders altogether.

Our trial was also susceptible to a variety of “on-the-ground” changes in healthcare operations that occurred over the course of the trial. COVID-related impacts included a large-scale switch to virtual appointments, and processes related to this change may have diminished relevance and impact of the trial’s different patterns of outreach to patients. For instance, during the trial it became standard practice for providers to call a patient if they did not log into a virtual appointment, thereby potentially creating a new intervention and obviating a missed appointment. We were also unable to quantify the fidelity of the intervention and cannot guarantee that the offices preparing and sending the appointment reminder letters consistently followed standard procedures.

While our pragmatic trial was large, it was nonetheless limited to a single VA healthcare system among patients who were primarily White, male, and older adults. Findings could differ in other patient populations.

It is also plausible that no intervention focused simply on appointment reminders may be able to create significant change in appointment attendance. Financial incentives—whether framed as a gain or loss—can be used to influence health behaviors, though their cost-effectiveness would need to be considered.46 A recent systematic review of missed appointments in diabetes clinic appointments found that qualitative studies suggest psychosocial factors playing a prominent role.47 It may be, then, that even in the best of circumstances, nudges embedded into appointment reminders are inadequate to resolve the mixture of emotional, cognitive, and behavioral factors involved in triggering missed appointments. In that case, it may be up to health systems and their researchers to determine whether to deploy and test interventions that are more costly or higher intensity, or both. Perhaps a 10% missed appointment rate in primary care or a 20% missed appointment rate in mental health clinics is as good as can be practically achieved without addressing the larger social and contextual issues underling missed appointments.

In conclusion, in this large pragmatic trial in primary care and mental health clinics, we found appointment reminder letters incorporating brief behavioral nudges did not affect appointment attendance. Attempts to reduce missed appointments or increase cancelation of appointments that are no longer necessary likely require more complex or intensive interventions.