Abstract
When conducting experience sampling studies, one important decision that researchers must make is the method by which surveys are administered. Dozens of reviews and recommendations cover types of response modes, including paper and pencil, online survey, and interactive voice response. However, few studies have empirically tested differences across response methods, and no studies have compared online surveys to interactive voice response surveys. Using a time-based experience sampling design, the present study investigates differences in compliance, data quality, and participant burden when using interactive voice response and online surveys. Results indicate no differences in terms of compliance rates and number of responses between the two methods. Interactive voice response produced lengthier qualitative responses, although there were no differences in the clarity of qualitative responses. Finally, online surveys may alleviate time burden, particularly as the number of items increases.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Description length models were consistent when run as a normally distributed outcome (γ01 = 18.61, p = .03). Figure results reflect these findings, rather than the results based on the negative binomial distribution, because they are most easily interpretable and not substantively different.
Estimated assuming a normal distribution for description length. Models run assuming a negative binomial distribution failed to converge.
References
American Heart Association. (2017). Blood pressure vs. heart rate (pulse). Retrieved from http://www.heart.org/HEARTORG/Conditions/HighBloodPressure/GettheFactsAboutHighBloodPressure/Blood-Pressure-vs-Heart-Rate-Pulse_UCM_301804_Article.jsp#.WuKHMsgvxPY.
Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.
Beal, D. J., & Weiss, H. M. (2003). Methods of ecological momentary assessment in organizational research. Organizational Research Methods, 6, 440–464. https://doi.org/10.1177/1094428103257361.
Bliese, P. (2016). Multilevel: multilevel functions. R package version 2.6. https://CRAN.R-project.org/package=multilevel.
Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: capturing life as it is lived. Annual Review of Psychology, 54, 579–616. https://doi.org/10.1146/annurev.psych.54.101601.145030.
Broderick, J. E., & Stone, A. A. (2006). Paper and electronic diaries: too early for conclusions on compliance rates and their effects--comment on Green, Rafaeli, Bolger, Shrout, and Reis (2006). Psychological Methods, 11, 106–111. https://doi.org/10.1037/1082-989X.11.1.106.
Butts, M. M., Becker, W. J., & Boswell, W. R. (2015). Hot buttons and time sinks: the effects of electronic communication during nonwork time on emotions and work-nonwork conflict. Academy of Management Journal, 58, 763–788.
Cohen, A. A., & Lemish, D. (2003). Real time and recall measures of mobile phone use: some methodological concerns and empirical applications. New Media & Society, 5, 167–183. https://doi.org/10.1177/1461444803005002002.
Collins, R. L., Kashdan, T. B., & Gollnisch, G. (2003). The feasibility of using cellular phones to collect ecological momentary assessment data: application to alcohol consumption. Experimental and Clinical Psychopharmacology, 11, 73–78. https://doi.org/10.1037/1064-1297.11.1.73.
Eatough, E., Shockley, K., & Yu, P. (2016). A review of ambulatory health data collection methods for employee experience sampling research. Applied Psychology. An International Review, 65, 322–354. https://doi.org/10.1111/apps.12068.
Fisher, C. D., & To, M. L. (2012). Using experience sampling methodology in organizational behavior. Journal of Organizational Behavior, 33, 865–877. https://doi.org/10.1002/job.1803.
Good, D. J., Lyddy, C. J., Glomb, T. M., Bono, J. E., Brown, K. W., Duffy, M. K., Baer, R. A., Brewer, J. A., & Lazar, S. W. (2016). Contemplating mindfulness at work: an integrative review. Journal of Management, 42, 114–142. https://doi.org/10.1177/0149206315617003.
Green, A. S., Rafaeli, E., Bolger, N., Shrout, P. E., & Reis, H. T. (2006). Paper or plastic?: data equivalence in paper and electronic diaries. Psychological Methods, 11, 87–105. https://doi.org/10.1037/1082-989X.11.1.87.
Greenhaus, J. H., & Beutell, N. J. (1985). Sources and conflict between work and family roles. The Academy of Management Review, 10, 76–88. https://doi.org/10.2307/258214.
Hamaker, E. L., & Wichers, M. (2017). No time like the present: discovering the hidden dynamics in intensive longitudinal data. Current Directions in Psychological Science, 26, 10–15.
Heron, K. E., & Smyth, J. M. (2010). Ecological momentary interventions: incorporating mobile technology into psychosocial and health behavior treatments. British Journal of Health Psychology, 15, 1–39. https://doi.org/10.1348/135910709X466063.
Hobfoll, S. E. (1989). Conservation of resources: a new attempt at conceptualizing stress. American Psychologist, 44, 513–524. https://doi.org/10.1037/0003-066X.44.3.513.
Kuntsche, E., & Labhart, F. (2013). Using personal cell phones for ecological momentary assessment: an overview of current developments. European Psychologist, 18, 3–11. https://doi.org/10.1027/1016-9040/a000127.
Lanaj, K., Johnson, R. E., & Lee, S. M. (2016). Benefits of transformational behaviors for leaders: a daily investigation of leader behaviors and need fulfillment. Journal of Applied Psychology, 101, 237–251.
Ohly, S., Sonnentag, S., Niessen, C., & Zapf, D. (2010). Diary studies in organizational research: an introduction and some practical recommendations. Journal of Personnel Psychology, 9, 79–93. https://doi.org/10.1027/1866-5888/a000009.
Olson, K. J. (2014). Development and initial validation of a measure of work, family, and school conflict. Journal of Occupational Health Psychology, 19, 46–59. https://doi.org/10.1037/a0034927.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods (2nd ed.). Thousand Oaks: Sage Publications, Inc..
Rodrigues, S., Kaiseler, M., & Queirós, C. (2015). Psychophysiological assessment of stress under ecological settings: a systematic review. European Psychologist, 20, 204–226. https://doi.org/10.1027/1016-9040/a000222.
Shiffman, S. (2007). Designing protocols for ecological momentary assessment. In A. Stone, S. Shiffman, A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: self-reports in health research (pp. 27–53). New York: Oxford University Press.
Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415.
Shockley, K. M., & Allen, T. D. (2013). Episodic work–family conflict, cardiovascular indicators, and social support: an experience sampling approach. Journal of Occupational Health Psychology, 18(3), 262–275.
Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002). Patient non-compliance with paper diaries. BMJ, 324, 1193–1194.
Tunarosa, A., & Glynn, M. A. (2017). Strategies of integration in mixed methods research: insights using relational algorithms. Organizational Research Methods, 20, 224–242.
Uy, M. A., Foo, M., & Aguinis, H. (2010). Using experience sampling methodology to advance entrepreneurship theory and research. Organizational Research Methods, 13, 31–54. https://doi.org/10.1177/1094428109334977.
Weiss, H. M., & Rupp, D. E. (2011). Experiencing work: an essay on a person-centric work psychology. Industrial and Organizational Psychology, 4, 83–97.
Acknowledgements
The authors would like to acknowledge funding sources for this work: The University of South Florida Office for Undergraduate Research’s Research Scholarship, and the University of South Florida Psychology Department’s Walvoord Verizon Wireless Work-Family Research Endowment. This work was also supported in part by the Sunshine Education and Research Center at the University of South Florida. The Center is supported by Training Grant No. T42-OH008438 from the Centers for Disease Control and Prevention/National Institute for Occupational Safety and Health (NIOSH). Additionally, we would like to acknowledge Pamela Smith for her leadership in directing recruitment and data collection, as well as Andrea Bevill, LeVonte Brooks, Jake Mathwich, Dylan Reeves, Maryam Romagosa, Neal Vogel, and Hyeyoung Yoon for their assistance with recruitment and data collection, and Ryan Nguyen for his help with coding. Finally, we thank the editor and two anonymous reviewers for their constructive suggestions.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
French, K.A., Falcon, C.N. & Allen, T.D. Experience Sampling Response Modes: Comparing Voice and Online Surveys. J Bus Psychol 34, 575–586 (2019). https://doi.org/10.1007/s10869-018-9560-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10869-018-9560-y