Journal of General Internal Medicine

, Volume 34, Issue 6, pp 823–824 | Cite as

Improving Longitudinal Survey Participation Among Internal Medicine Residents: Incorporating Behavioral Economic Techniques and Avoiding Friday or Saturday Invitations

  • Krisda H. ChaiyachatiEmail author
  • Jason Roy
  • David A. Asch
  • C. Jessica Dine
  • Sanjay Desai
  • Lisa M. Bellini
  • Judy A. Shea
Open Access
Concise Research Reports


Low participation rates limit the generalizability of surveys. Prior studies show participation is better with financial incentives.1, 2 Beyond financial incentives, insights from behavioral economics reveal that psychological concepts, like regret aversion,3 can increase participation. For example, subsequent medication adherence improves when patients are entered into a “regret lottery”—they can win prizes if they adhere or are told how much they could have won, had they adhered.4 Other studies of survey participation have compared response rates for some, but not all days of the week.5 Using a longitudinal survey of internal medicine residents, we examined the effect of (a) whether the prize and regret message components of a regret lottery increased subsequent participation and (b) whether response rates varied based on the day of the week survey invitations were sent.


Residents from 63 internal medicine programs were invited to complete web-based surveys for the Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) trial, a cluster-randomized trial designed to test alternative resident duty hour rules.6 Residents received one survey, with 24 h to respond, every 2 weeks for 16 consecutive cycles from September 2015 through April 2016. Before each cycle, one first-year and one junior or senior resident at each program was randomly pre-selected to win a prize (either a $25 or $100 Amazon gift card) and randomly assigned a day of the week for survey invitation. At the end of each cycle, pre-selected residents who responded were emailed the prize. Pre-selected residents who had not responded received a “regret message” indicating what they would have won, had they responded. This procedure was repeated each cycle.

We fit a logistic regression model with survey completion (“responding”) as the outcome. The model included two binary variables for the prior cycle’s (a) lottery selection and (b) survey response, and an interaction term between the two. We used generalized estimating equations to account for clustering by resident, adjusting for resident year, trial arm, training program, cycle number, and cumulative prior response rate. In a secondary analysis, we distinguished the effects of winning or regretting by dollar amounts ($25 or $100). The University of Pennsylvania institutional review board approved the protocol. Analyses were conducted using Stata v14.0 (StataCorp LLP).


In total, 97,142 survey invitations went to 6228 residents, with a 40.0% overall participation rate (Table 1). Non-responders who received a regret message improved their subsequent response probability (OR 1.37, 95% CI 1.12 to 1.68; P < 0.001), as did responders receiving a prize (OR 1.44, 95% CI 1.09 to 1.90; P = 0.003) (Table 2). Each represents approximately a 4% increase in participation. No differences were found when comparing $25 versus $100 for either regret notices or receiving a prize (P = 0.08 and P = 0.83, respectively), although power for these tests was low. No differences were observed comparing the effect of regret messages with prizes (P = 0.77). The probability of response was lower when survey invitations were sent on Fridays (OR 0.72, 95% CI 0.65 to 0.79; P < 0.0001) and Saturdays (OR 0.84, 95% CI 0.76 to 0.92; P < 0.0001), corresponding to responses required by Saturday and Sunday, respectively.
Table 1

Characteristics of Eligible Residents



Eligible residentsa, no.


Survey invitations emailed, no.


Overall response rate, %


Training level, no. (%)


2528 (41)


1896 (30)


1804 (29)

Study arm, no. (%)


3211 (52)


3017 (48)

Lottery selected, by $ amount, no. (%)


1406 (23)


464 (7.5)

PGY, post-graduate year; United States dollar

aResidents were eligible if they were emailed an invitation to complete a survey during the study period. Residents were emailed an invitation if they were on a list of emails provided by each training program. Residents were excluded if they were dropped from the email list for any of the 16 survey cycles if these individuals indicated they were included erroneously because of an incorrect email or they were no longer a trainee at their respective training program. "Study arm" refer to the two trial arms within the Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) where training programs and all their residents were randomized to the 2011 Accreditation Council for Graduate Medical Education shift-length policies (“standard”) or were not required to adhere to shift-length limits (“flexible”)

Table 2

Probability of Responding in the Subsequent Cycle After Having Received a Regret Message or Prize, and the Day of Survey Invitation


Adjusted OR of Respondinge

(95% Confidence Interval)


Effect of a regret messagea (+Selected/-Responded)

 Regret message for any $ amountb

1.37 (1.12 to 1.68)


 Regret message for $25b

1.26 (1.02 to 1.55)


Regret message for $100b

1.77 (1.28 to 2.44)


 Comparison of regret for $25 vs. $100

1.41 (0.96 to 2.07)


Effect of receiving a prize (+Selected/+Responded)

 Received prize for any $ amountc

1.44 (1.09 to 1.90)


 Received prize for $25c

1.46 (1.10 to 1.94)


 Received prize for $100c

1.38 (0.87 to 2.18)


 Comparison of receiving $25 vs. $100

1.06 (0.62 to 1.82)


Regret messaged vs. received prize (any $ amount)

1.05 (0.77 to 1.42)


Day of the week invitations were sentd


0.98 (0.89 to 1.08)



0.92 (0.83 to 1.02)



0.93 (0.84 to 1.02)



0.72 (0.65 to 0.79)



0.84 (0.76 to 0.92)



0.98 (0.89 to 1.08)



Understanding the influence of behavioral economic techniques on survey response rates is an important advancement in survey science. Our findings suggest that regret messages were as influential as receiving prizes for increasing subsequent survey participation among residents. Additionally, survey responses expected on weekends had fewer responses. Our study has several limitations. First, this is a study of internal medicine residents and may not be generalizable to other populations. Second, this is a study of participation in repeated surveys. The effects of achieved or anticipated prizes or regret may differ when a single survey is administered. Third, the influence of regret messages and days of the week may have been different had we administered paper-based or mail-in surveys. Additional evaluations are needed to validate our findings in these other settings. However, at minimum, these simple techniques are likely to improve survey participation in a longitudinal survey of internal medicine residents. If behavioral economic techniques and targeting the ideal days of the week can increase participation in other survey designs, their low-cost approach (if not free) could be an important, cost-effective mechanism for improving the generalizability of survey-based findings.



We would like to acknowledge contributions to the design of the regret lottery and survey made by the iCOMPARE Research Group, specifically: Alice L. Sternberg, ScM; David F. Dinges, PhD; Mathias Basner, MD, PhD; Jeffrey H. Silber, MD, PhD; Orit Even-Shoshan, MS; David W. Bates, MD, MS; Joel Katz, MD; Kevin G. Volpp, MD, PhD; Dylan Small, PhD; Elizabeth O-Grady; and Kelsey Gangemi, MPH.


iCOMPARE is funded by the National Heart, Lung, and Blood Institute (U01HL126088, U01HL125388) and a grant from the ACGME.

Compliance with Ethical Standards

Conflict of Interest

Dr. David Asch is a partner and part owner of VAL Health, a behavioral economics consulting firm. All remaining authors declare they do not have a conflict of interest.


  1. 1.
    Schweitzer M, Asch DA. Timing payments to subjects of mail surveys: cost-effectiveness and bias. J Clin Epidemiol. 1995;48(11):1325–1329.CrossRefGoogle Scholar
  2. 2.
    Asch DA, Christakis NA, Ubel PA. Conducting physician mail surveys on a limited budget. A randomized trial comparing $2 bill versus $5 bill incentives. Med Care. 1998;36(1):95–99.CrossRefGoogle Scholar
  3. 3.
    Kahneman D, Tversky A. Prospect theory: an analysis of decision under risk. Econometrica. 1979;47(2):263–291.CrossRefGoogle Scholar
  4. 4.
    Volpp KG, Loewenstein G, Troxel AB, et al. A test of financial incentives to improve warfarin adherence. BMC Health Serv Res. 2008;8:272.CrossRefGoogle Scholar
  5. 5.
    Pit SW, Vo T, Pyakurel S. The effectiveness of recruitment strategies on general practitioner’s survey response rates - a systematic review. BMC Med Res Methodol. 2014;14:76.CrossRefGoogle Scholar
  6. 6.
    Desai SV, Asch DA, Bellini LM, et al. Education outcomes in a duty-hour flexibility trial in internal medicine. N Engl J Med. 2018;378(16):1494–1508.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2019

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  • Krisda H. Chaiyachati
    • 1
    • 2
    Email author
  • Jason Roy
    • 3
  • David A. Asch
    • 1
    • 2
    • 4
  • C. Jessica Dine
    • 1
    • 2
  • Sanjay Desai
    • 5
  • Lisa M. Bellini
    • 1
  • Judy A. Shea
    • 1
    • 2
  1. 1.The Department of MedicineUniversity of Pennsylvania Perelman School of MedicinePhiladelphiaUSA
  2. 2.The Leonard Davis Institute of Health Economics University of PennsylvaniaPhiladelphiaUSA
  3. 3.Department of Biostatistics and EpidemiologyRutgers School of Public HealthPiscatawayUSA
  4. 4.Corporal Michael J. Crescenz VA Medical CenterPhiladelphiaUSA
  5. 5.The Department of MedicineJohns Hopkins UniversityBaltimoreUSA

Personalised recommendations