Skip to main content

Improving Longitudinal Survey Participation Among Internal Medicine Residents: Incorporating Behavioral Economic Techniques and Avoiding Friday or Saturday Invitations

BACKGROUND

Low participation rates limit the generalizability of surveys. Prior studies show participation is better with financial incentives.1, 2 Beyond financial incentives, insights from behavioral economics reveal that psychological concepts, like regret aversion,3 can increase participation. For example, subsequent medication adherence improves when patients are entered into a “regret lottery”—they can win prizes if they adhere or are told how much they could have won, had they adhered.4 Other studies of survey participation have compared response rates for some, but not all days of the week.5 Using a longitudinal survey of internal medicine residents, we examined the effect of (a) whether the prize and regret message components of a regret lottery increased subsequent participation and (b) whether response rates varied based on the day of the week survey invitations were sent.

METHODS

Residents from 63 internal medicine programs were invited to complete web-based surveys for the Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) trial, a cluster-randomized trial designed to test alternative resident duty hour rules.6 Residents received one survey, with 24 h to respond, every 2 weeks for 16 consecutive cycles from September 2015 through April 2016. Before each cycle, one first-year and one junior or senior resident at each program was randomly pre-selected to win a prize (either a $25 or $100 Amazon gift card) and randomly assigned a day of the week for survey invitation. At the end of each cycle, pre-selected residents who responded were emailed the prize. Pre-selected residents who had not responded received a “regret message” indicating what they would have won, had they responded. This procedure was repeated each cycle.

We fit a logistic regression model with survey completion (“responding”) as the outcome. The model included two binary variables for the prior cycle’s (a) lottery selection and (b) survey response, and an interaction term between the two. We used generalized estimating equations to account for clustering by resident, adjusting for resident year, trial arm, training program, cycle number, and cumulative prior response rate. In a secondary analysis, we distinguished the effects of winning or regretting by dollar amounts ($25 or $100). The University of Pennsylvania institutional review board approved the protocol. Analyses were conducted using Stata v14.0 (StataCorp LLP).

RESULTS

In total, 97,142 survey invitations went to 6228 residents, with a 40.0% overall participation rate (Table 1). Non-responders who received a regret message improved their subsequent response probability (OR 1.37, 95% CI 1.12 to 1.68; P < 0.001), as did responders receiving a prize (OR 1.44, 95% CI 1.09 to 1.90; P = 0.003) (Table 2). Each represents approximately a 4% increase in participation. No differences were found when comparing $25 versus $100 for either regret notices or receiving a prize (P = 0.08 and P = 0.83, respectively), although power for these tests was low. No differences were observed comparing the effect of regret messages with prizes (P = 0.77). The probability of response was lower when survey invitations were sent on Fridays (OR 0.72, 95% CI 0.65 to 0.79; P < 0.0001) and Saturdays (OR 0.84, 95% CI 0.76 to 0.92; P < 0.0001), corresponding to responses required by Saturday and Sunday, respectively.

Table 1 Characteristics of Eligible Residents
Table 2 Probability of Responding in the Subsequent Cycle After Having Received a Regret Message or Prize, and the Day of Survey Invitation

DISCUSSION

Understanding the influence of behavioral economic techniques on survey response rates is an important advancement in survey science. Our findings suggest that regret messages were as influential as receiving prizes for increasing subsequent survey participation among residents. Additionally, survey responses expected on weekends had fewer responses. Our study has several limitations. First, this is a study of internal medicine residents and may not be generalizable to other populations. Second, this is a study of participation in repeated surveys. The effects of achieved or anticipated prizes or regret may differ when a single survey is administered. Third, the influence of regret messages and days of the week may have been different had we administered paper-based or mail-in surveys. Additional evaluations are needed to validate our findings in these other settings. However, at minimum, these simple techniques are likely to improve survey participation in a longitudinal survey of internal medicine residents. If behavioral economic techniques and targeting the ideal days of the week can increase participation in other survey designs, their low-cost approach (if not free) could be an important, cost-effective mechanism for improving the generalizability of survey-based findings.

References

  1. 1.

    Schweitzer M, Asch DA. Timing payments to subjects of mail surveys: cost-effectiveness and bias. J Clin Epidemiol. 1995;48(11):1325–1329.

    Article  CAS  PubMed  Google Scholar 

  2. 2.

    Asch DA, Christakis NA, Ubel PA. Conducting physician mail surveys on a limited budget. A randomized trial comparing $2 bill versus $5 bill incentives. Med Care. 1998;36(1):95–99.

    Article  CAS  PubMed  Google Scholar 

  3. 3.

    Kahneman D, Tversky A. Prospect theory: an analysis of decision under risk. Econometrica. 1979;47(2):263–291.

    Article  Google Scholar 

  4. 4.

    Volpp KG, Loewenstein G, Troxel AB, et al. A test of financial incentives to improve warfarin adherence. BMC Health Serv Res. 2008;8:272.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Pit SW, Vo T, Pyakurel S. The effectiveness of recruitment strategies on general practitioner’s survey response rates - a systematic review. BMC Med Res Methodol. 2014;14:76.

    Article  PubMed  PubMed Central  Google Scholar 

  6. 6.

    Desai SV, Asch DA, Bellini LM, et al. Education outcomes in a duty-hour flexibility trial in internal medicine. N Engl J Med. 2018;378(16):1494–1508.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgments

We would like to acknowledge contributions to the design of the regret lottery and survey made by the iCOMPARE Research Group, specifically: Alice L. Sternberg, ScM; David F. Dinges, PhD; Mathias Basner, MD, PhD; Jeffrey H. Silber, MD, PhD; Orit Even-Shoshan, MS; David W. Bates, MD, MS; Joel Katz, MD; Kevin G. Volpp, MD, PhD; Dylan Small, PhD; Elizabeth O-Grady; and Kelsey Gangemi, MPH.

Funding

iCOMPARE is funded by the National Heart, Lung, and Blood Institute (U01HL126088, U01HL125388) and a grant from the ACGME.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Krisda H. Chaiyachati MD, MPH, MSHP.

Ethics declarations

Conflict of Interest

Dr. David Asch is a partner and part owner of VAL Health, a behavioral economics consulting firm. All remaining authors declare they do not have a conflict of interest.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Registration

Clinicaltrials.org Identifier: NCT02274818

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chaiyachati, K.H., Roy, J., Asch, D.A. et al. Improving Longitudinal Survey Participation Among Internal Medicine Residents: Incorporating Behavioral Economic Techniques and Avoiding Friday or Saturday Invitations. J GEN INTERN MED 34, 823–824 (2019). https://doi.org/10.1007/s11606-019-04836-8

Download citation