Journal of Business and Psychology

, Volume 34, Issue 5, pp 575–586 | Cite as

Experience Sampling Response Modes: Comparing Voice and Online Surveys

  • Kimberly A. FrenchEmail author
  • Christina N. Falcon
  • Tammy D. Allen
Original Paper


When conducting experience sampling studies, one important decision that researchers must make is the method by which surveys are administered. Dozens of reviews and recommendations cover types of response modes, including paper and pencil, online survey, and interactive voice response. However, few studies have empirically tested differences across response methods, and no studies have compared online surveys to interactive voice response surveys. Using a time-based experience sampling design, the present study investigates differences in compliance, data quality, and participant burden when using interactive voice response and online surveys. Results indicate no differences in terms of compliance rates and number of responses between the two methods. Interactive voice response produced lengthier qualitative responses, although there were no differences in the clarity of qualitative responses. Finally, online surveys may alleviate time burden, particularly as the number of items increases.


Experience sampling Ecological momentary assessment Interactive voice response Electronic communication Online survey 



The authors would like to acknowledge funding sources for this work: The University of South Florida Office for Undergraduate Research’s Research Scholarship, and the University of South Florida Psychology Department’s Walvoord Verizon Wireless Work-Family Research Endowment. This work was also supported in part by the Sunshine Education and Research Center at the University of South Florida. The Center is supported by Training Grant No. T42-OH008438 from the Centers for Disease Control and Prevention/National Institute for Occupational Safety and Health (NIOSH). Additionally, we would like to acknowledge Pamela Smith for her leadership in directing recruitment and data collection, as well as Andrea Bevill, LeVonte Brooks, Jake Mathwich, Dylan Reeves, Maryam Romagosa, Neal Vogel, and Hyeyoung Yoon for their assistance with recruitment and data collection, and Ryan Nguyen for his help with coding. Finally, we thank the editor and two anonymous reviewers for their constructive suggestions.


  1. Bates, D., Maechler, M., Bolker, B., & Walker, S. (2015). Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67, 1–48.CrossRefGoogle Scholar
  2. Beal, D. J., & Weiss, H. M. (2003). Methods of ecological momentary assessment in organizational research. Organizational Research Methods, 6, 440–464. Scholar
  3. Bliese, P. (2016). Multilevel: multilevel functions. R package version 2.6.
  4. Bolger, N., Davis, A., & Rafaeli, E. (2003). Diary methods: capturing life as it is lived. Annual Review of Psychology, 54, 579–616. Scholar
  5. Broderick, J. E., & Stone, A. A. (2006). Paper and electronic diaries: too early for conclusions on compliance rates and their effects--comment on Green, Rafaeli, Bolger, Shrout, and Reis (2006). Psychological Methods, 11, 106–111. Scholar
  6. Butts, M. M., Becker, W. J., & Boswell, W. R. (2015). Hot buttons and time sinks: the effects of electronic communication during nonwork time on emotions and work-nonwork conflict. Academy of Management Journal, 58, 763–788.CrossRefGoogle Scholar
  7. Cohen, A. A., & Lemish, D. (2003). Real time and recall measures of mobile phone use: some methodological concerns and empirical applications. New Media & Society, 5, 167–183. Scholar
  8. Collins, R. L., Kashdan, T. B., & Gollnisch, G. (2003). The feasibility of using cellular phones to collect ecological momentary assessment data: application to alcohol consumption. Experimental and Clinical Psychopharmacology, 11, 73–78. Scholar
  9. Eatough, E., Shockley, K., & Yu, P. (2016). A review of ambulatory health data collection methods for employee experience sampling research. Applied Psychology. An International Review, 65, 322–354. Scholar
  10. Fisher, C. D., & To, M. L. (2012). Using experience sampling methodology in organizational behavior. Journal of Organizational Behavior, 33, 865–877. Scholar
  11. Good, D. J., Lyddy, C. J., Glomb, T. M., Bono, J. E., Brown, K. W., Duffy, M. K., Baer, R. A., Brewer, J. A., & Lazar, S. W. (2016). Contemplating mindfulness at work: an integrative review. Journal of Management, 42, 114–142. Scholar
  12. Green, A. S., Rafaeli, E., Bolger, N., Shrout, P. E., & Reis, H. T. (2006). Paper or plastic?: data equivalence in paper and electronic diaries. Psychological Methods, 11, 87–105. Scholar
  13. Greenhaus, J. H., & Beutell, N. J. (1985). Sources and conflict between work and family roles. The Academy of Management Review, 10, 76–88. Scholar
  14. Hamaker, E. L., & Wichers, M. (2017). No time like the present: discovering the hidden dynamics in intensive longitudinal data. Current Directions in Psychological Science, 26, 10–15.CrossRefGoogle Scholar
  15. Heron, K. E., & Smyth, J. M. (2010). Ecological momentary interventions: incorporating mobile technology into psychosocial and health behavior treatments. British Journal of Health Psychology, 15, 1–39. Scholar
  16. Hobfoll, S. E. (1989). Conservation of resources: a new attempt at conceptualizing stress. American Psychologist, 44, 513–524. Scholar
  17. Kuntsche, E., & Labhart, F. (2013). Using personal cell phones for ecological momentary assessment: an overview of current developments. European Psychologist, 18, 3–11. Scholar
  18. Lanaj, K., Johnson, R. E., & Lee, S. M. (2016). Benefits of transformational behaviors for leaders: a daily investigation of leader behaviors and need fulfillment. Journal of Applied Psychology, 101, 237–251.CrossRefPubMedGoogle Scholar
  19. Ohly, S., Sonnentag, S., Niessen, C., & Zapf, D. (2010). Diary studies in organizational research: an introduction and some practical recommendations. Journal of Personnel Psychology, 9, 79–93. Scholar
  20. Olson, K. J. (2014). Development and initial validation of a measure of work, family, and school conflict. Journal of Occupational Health Psychology, 19, 46–59. Scholar
  21. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods (2nd ed.). Thousand Oaks: Sage Publications, Inc..Google Scholar
  22. Rodrigues, S., Kaiseler, M., & Queirós, C. (2015). Psychophysiological assessment of stress under ecological settings: a systematic review. European Psychologist, 20, 204–226. Scholar
  23. Shiffman, S. (2007). Designing protocols for ecological momentary assessment. In A. Stone, S. Shiffman, A. Atienza, & L. Nebeling (Eds.), The science of real-time data capture: self-reports in health research (pp. 27–53). New York: Oxford University Press.Google Scholar
  24. Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. Scholar
  25. Shockley, K. M., & Allen, T. D. (2013). Episodic work–family conflict, cardiovascular indicators, and social support: an experience sampling approach. Journal of Occupational Health Psychology, 18(3), 262–275.CrossRefPubMedGoogle Scholar
  26. Stone, A. A., Shiffman, S., Schwartz, J. E., Broderick, J. E., & Hufford, M. R. (2002). Patient non-compliance with paper diaries. BMJ, 324, 1193–1194.CrossRefPubMedPubMedCentralGoogle Scholar
  27. Tunarosa, A., & Glynn, M. A. (2017). Strategies of integration in mixed methods research: insights using relational algorithms. Organizational Research Methods, 20, 224–242.CrossRefGoogle Scholar
  28. Uy, M. A., Foo, M., & Aguinis, H. (2010). Using experience sampling methodology to advance entrepreneurship theory and research. Organizational Research Methods, 13, 31–54. Scholar
  29. Weiss, H. M., & Rupp, D. E. (2011). Experiencing work: an essay on a person-centric work psychology. Industrial and Organizational Psychology, 4, 83–97.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of PsychologyGeorgia Institute of TechnologyAtlantaUSA
  2. 2.Department of PsychologyUniversity of South FloridaTampaUSA

Personalised recommendations