Low participation rates limit the generalizability of surveys. Prior studies show participation is better with financial incentives.1, 2 Beyond financial incentives, insights from behavioral economics reveal that psychological concepts, like regret aversion,3 can increase participation. For example, subsequent medication adherence improves when patients are entered into a “regret lottery”—they can win prizes if they adhere or are told how much they could have won, had they adhered.4 Other studies of survey participation have compared response rates for some, but not all days of the week.5 Using a longitudinal survey of internal medicine residents, we examined the effect of (a) whether the prize and regret message components of a regret lottery increased subsequent participation and (b) whether response rates varied based on the day of the week survey invitations were sent.
Residents from 63 internal medicine programs were invited to complete web-based surveys for the Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) trial, a cluster-randomized trial designed to test alternative resident duty hour rules.6 Residents received one survey, with 24 h to respond, every 2 weeks for 16 consecutive cycles from September 2015 through April 2016. Before each cycle, one first-year and one junior or senior resident at each program was randomly pre-selected to win a prize (either a $25 or $100 Amazon gift card) and randomly assigned a day of the week for survey invitation. At the end of each cycle, pre-selected residents who responded were emailed the prize. Pre-selected residents who had not responded received a “regret message” indicating what they would have won, had they responded. This procedure was repeated each cycle.
We fit a logistic regression model with survey completion (“responding”) as the outcome. The model included two binary variables for the prior cycle’s (a) lottery selection and (b) survey response, and an interaction term between the two. We used generalized estimating equations to account for clustering by resident, adjusting for resident year, trial arm, training program, cycle number, and cumulative prior response rate. In a secondary analysis, we distinguished the effects of winning or regretting by dollar amounts ($25 or $100). The University of Pennsylvania institutional review board approved the protocol. Analyses were conducted using Stata v14.0 (StataCorp LLP).
In total, 97,142 survey invitations went to 6228 residents, with a 40.0% overall participation rate (Table 1). Non-responders who received a regret message improved their subsequent response probability (OR 1.37, 95% CI 1.12 to 1.68; P < 0.001), as did responders receiving a prize (OR 1.44, 95% CI 1.09 to 1.90; P = 0.003) (Table 2). Each represents approximately a 4% increase in participation. No differences were found when comparing $25 versus $100 for either regret notices or receiving a prize (P = 0.08 and P = 0.83, respectively), although power for these tests was low. No differences were observed comparing the effect of regret messages with prizes (P = 0.77). The probability of response was lower when survey invitations were sent on Fridays (OR 0.72, 95% CI 0.65 to 0.79; P < 0.0001) and Saturdays (OR 0.84, 95% CI 0.76 to 0.92; P < 0.0001), corresponding to responses required by Saturday and Sunday, respectively.
Understanding the influence of behavioral economic techniques on survey response rates is an important advancement in survey science. Our findings suggest that regret messages were as influential as receiving prizes for increasing subsequent survey participation among residents. Additionally, survey responses expected on weekends had fewer responses. Our study has several limitations. First, this is a study of internal medicine residents and may not be generalizable to other populations. Second, this is a study of participation in repeated surveys. The effects of achieved or anticipated prizes or regret may differ when a single survey is administered. Third, the influence of regret messages and days of the week may have been different had we administered paper-based or mail-in surveys. Additional evaluations are needed to validate our findings in these other settings. However, at minimum, these simple techniques are likely to improve survey participation in a longitudinal survey of internal medicine residents. If behavioral economic techniques and targeting the ideal days of the week can increase participation in other survey designs, their low-cost approach (if not free) could be an important, cost-effective mechanism for improving the generalizability of survey-based findings.
Schweitzer M, Asch DA. Timing payments to subjects of mail surveys: cost-effectiveness and bias. J Clin Epidemiol. 1995;48(11):1325–1329.
Asch DA, Christakis NA, Ubel PA. Conducting physician mail surveys on a limited budget. A randomized trial comparing $2 bill versus $5 bill incentives. Med Care. 1998;36(1):95–99.
Kahneman D, Tversky A. Prospect theory: an analysis of decision under risk. Econometrica. 1979;47(2):263–291.
Volpp KG, Loewenstein G, Troxel AB, et al. A test of financial incentives to improve warfarin adherence. BMC Health Serv Res. 2008;8:272.
Pit SW, Vo T, Pyakurel S. The effectiveness of recruitment strategies on general practitioner’s survey response rates - a systematic review. BMC Med Res Methodol. 2014;14:76.
Desai SV, Asch DA, Bellini LM, et al. Education outcomes in a duty-hour flexibility trial in internal medicine. N Engl J Med. 2018;378(16):1494–1508.
We would like to acknowledge contributions to the design of the regret lottery and survey made by the iCOMPARE Research Group, specifically: Alice L. Sternberg, ScM; David F. Dinges, PhD; Mathias Basner, MD, PhD; Jeffrey H. Silber, MD, PhD; Orit Even-Shoshan, MS; David W. Bates, MD, MS; Joel Katz, MD; Kevin G. Volpp, MD, PhD; Dylan Small, PhD; Elizabeth O-Grady; and Kelsey Gangemi, MPH.
iCOMPARE is funded by the National Heart, Lung, and Blood Institute (U01HL126088, U01HL125388) and a grant from the ACGME.
Conflict of Interest
Dr. David Asch is a partner and part owner of VAL Health, a behavioral economics consulting firm. All remaining authors declare they do not have a conflict of interest.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Clinicaltrials.org Identifier: NCT02274818
About this article
Cite this article
Chaiyachati, K.H., Roy, J., Asch, D.A. et al. Improving Longitudinal Survey Participation Among Internal Medicine Residents: Incorporating Behavioral Economic Techniques and Avoiding Friday or Saturday Invitations. J GEN INTERN MED 34, 823–824 (2019). https://doi.org/10.1007/s11606-019-04836-8