A validation of well-being and happiness surveys for administration via the Internet

Abstract

Internet research is appealing because it is a cost- and time-efficient way to access a large number of participants; however, the validity of Internet research for important subjective well-being (SWB) surveys has not been adequately assessed. The goal of the present study was to validate the Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985), the Positive and Negative Affect Schedule (PANAS-X; Watson & Clark, 1994), and the Subjective Happiness Scale (SHS; Lyubomirsky & Lepper, 1999) for use on the Internet. This study compared the quality of data collected using paper-based (paper-and-pencil version in a lab setting), computer-based (Web-based version in a lab setting), and Internet (Web-based version on a computer of the participant’s choosing) surveys for these three measures of SWB. The paper-based and computer-based experiment recruited two college student samples; the Internet experiments recruited a college student sample and an adult sample responding to ads on different social-networking Web sites. This study provides support for the reliability, validity, and generalizability of the Internet format of the SWLS, PANAS-X, and SHS. Across the three experiments, the results indicate that the computer-based and Internet surveys had means, standard deviations, reliabilities, and factor structures that were similar to those of the paper-based versions. The discussion examines the difficulty of higher attrition for the Internet version, the need to examine reverse-coded items in the future, and the possibility that unhappy individuals are more likely to participate in Internet surveys of SWB.

References

  1. Bech, M., & Kristensen, M. B. (2009). Differential response rates in postal and Web-based surveys in older respondents. Survey Research Methods, 3, 1–6.

    Google Scholar 

  2. Birnbaum, M. H. (2004). Human research and data collection via the Internet. Annual Review of Psychology, 55, 803–832.

    PubMed  Article  Google Scholar 

  3. Bliven, B. D., Kaufman, S. E., & Spertus, J. A. (2001). Electronic collection of health-related quality of life data: Validity, time, benefits, and patient preference. Quality of Life Research, 10, 15–21.

    PubMed  Article  Google Scholar 

  4. Brecko, B. N., & Carstens, R. (2006). Online data collection in SITES 2006. Paper versus Web survey—Do they provide comparable results? The Second IEA International Research Conference: Proceedings of the IRC-2006, 2, 261–269.

    Google Scholar 

  5. Buchanan, T. (2002). Online assessment: Desirable or dangerous? Professional Psychology: Research & Practice, 33, 148–154.

    Article  Google Scholar 

  6. Buchanan, T., Johnson, J. A., & Goldberg, L. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115–127.

    Article  Google Scholar 

  7. Buchanan, T., & Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World Wide Web. British Journal of Psychology, 90, 125–144.

    PubMed  Article  Google Scholar 

  8. Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Ost, L.-G., & Andersson, G. (2007). Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Computers in Human Behavior, 23, 1421–1434.

    Article  Google Scholar 

  9. Chuah, S. C., Drasgow, F., & Roberts, B. W. (2006). Personality measurement: Does the medium matter? No. Journal of Research in Personality, 40, 359–376.

    Article  Google Scholar 

  10. Cole, M. S., Bedeian, A. G., & Field, H. S. (2006). The measurement equivalence of Web-based and paper-and-pencil measures of transformational leadership. Organizational Research Methods, 9, 339–368.

    Article  Google Scholar 

  11. Davis, R. N. (1999). Web-based administration of a personality questionnaire: Comparison with traditional methods. Behavior Research Methods, Instruments, & Computers, 31, 572–577.

    Article  Google Scholar 

  12. DeNeve, K. M., & Cooper, H. (1998). The happy personality: A metaanalysis of 137 personality traits and subjective well-being. Psychological Bulletin, 124, 197–229.

    PubMed  Article  Google Scholar 

  13. Denscombe, M. (2008). The length of responses to open-ended questions: A comparison of online and paper questionnaires in terms of a mode effect. Social Science Computer Review, 26, 359–368.

    Article  Google Scholar 

  14. Diener, E., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The Satisfaction With Life Scale. Journal of Personality Assessment, 49, 71–75.

    PubMed  Article  Google Scholar 

  15. Diener, E., & Seligman, M. E. P. (2004). Beyond money. Toward an economy of well-being. Psychological Science in the Public Interest, 5, 1–31.

    Article  Google Scholar 

  16. Feldt, L. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363–373.

    Article  Google Scholar 

  17. Fortson, B. L., Scotti, J. R., Del Ben, K. S., & Chen, Y.-C. (2006). Reliability and validity of an Internet traumatic stress survey with a college student sample. Journal of Traumatic Stress, 19, 709–720.

    PubMed  Article  Google Scholar 

  18. George, C. E., Lankford, J. S., & Wilson, S. E. (1992). The effects of computerized versus paper-and-pencil administration on measures of negative affect. Computers in Human Behavior, 8, 203–209.

    Article  Google Scholar 

  19. Gosling, S. D., Vazire, S., Srivastava, S., & John, O. P. (2004). Should we trust Web-based studies: A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93–104.

    PubMed  Article  Google Scholar 

  20. Herrero, J., & Meneses, J. (2006). Short Web-based versions of the perceived stress (PSS) and Center for Epidemiological Studies-Depression (CESD) Scales: A comparison to pencil and paper responses among Internet users. Computers in Human Behavior, 22, 830–846.

    Article  Google Scholar 

  21. Hewson, C., & Charlton, J. P. (2005). Measuring health beliefs on the Internet: A comparison of paper and Internet administrations of the Multidimensional Health Locus of Control Scale. Behavior Research Methods, 37, 691–702.

    PubMed  Article  Google Scholar 

  22. Im, E.-O., Chee, W., Bender, M., Cheng, C.-Y., Tsai, H.-M., Kang, N. M., & Lee, H. (2005). The psychometric properties of pen-andpencil and Internet versions of the Midlife Women’s Symptom Index (MSI). International Journal of Nursing Studies, 42, 167–177.

    PubMed  Article  Google Scholar 

  23. John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five inventory—Versions 4a and 54. Berkeley: University of California, Berkeley, Institute of Personality and Social Research.

    Google Scholar 

  24. Joinson, A. (1999). Social desirability, anonymity, and Internet-based questionnaires. Behavior Research Methods, Instruments, & Computers, 31, 433–438.

    Article  Google Scholar 

  25. Kim, S., & Feldt, L. S. (2008). A comparison of tests for equality of two or more independent alpha coefficients. Journal of Educational Measurement, 45, 179–193.

    Article  Google Scholar 

  26. Kveton, P., Jelínek, M., Klimusová, H., & Vobořil, D. (2007). Data collection on the Internet: Evaluation of Web-based questionnaires. Studia Psychologica, 49, 81–88.

    Google Scholar 

  27. Lonsdale, C., Hodge, K., & Rose, E. A. (2006). Pixels vs. paper: Comparing online and traditional survey methods in sport psychology. Journal of Sport & Exercise Psychology, 28, 100–108.

    Google Scholar 

  28. Luthans, F., Avey, J. B., & Patera, J. L. (2008). Experimental analysis of a Web-based training intervention on the learning and development of positive psychological states. Academy of Management Learning & Education, 7, 209–221.

    Google Scholar 

  29. Lyubomirsky, S., & Lepper, H. S. (1999). A measure of subjective happiness: Preliminary reliability and construct validation. Social Indicators Research, 46, 137–155.

    Article  Google Scholar 

  30. McGraw, K. O., Tew, M. D., & Williams, J. E. (2000). The integrity of Web-delivered experiments: Can you trust the data? Psychological Science, 11, 502–506.

    PubMed  Article  Google Scholar 

  31. McNeil, J. M., & Fleeson, W. (2006). The causal effect of extraversion on positive affect and neuroticism on negative affect: Manipulating state extraversion and state neuroticism in an experimental approach. Journal of Research in Personality, 40, 529–550.

    Article  Google Scholar 

  32. Nosen, E., & Woody, S. R. (2008). Online surveys: Effect of research design decision on rates of invalid participation and data credibility. Graduate Student Journal of Psychology, 10, 3–14.

    Google Scholar 

  33. Pavot, W., & Diener, E. (1993). Review of the Satisfaction With Life Scale. Psychological Assessment, 5, 164–172.

    Article  Google Scholar 

  34. Reimers, S., & Stewart, N. (2007). Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities. Behavior Research Methods, 39, 365–370.

    PubMed  Article  Google Scholar 

  35. Riva, G., Teruzzi, T., & Anolli, L. (2003). The use of the Internet in psychological research: Comparison of online and offline questionnaires. CyberPsychology & Behavior, 6, 73–80.

    Article  Google Scholar 

  36. Rosenfeld, P., Giacalone, R. A., Knouse, S. B., Doherty, L. M., Vicino, S. M., Kantor, J., & Greaves, J. (1991). Impression management, candor, and microcomputer-based organizational surveys: An individual differences approach. Computers in Human Behavior, 7, 23–32.

    Article  Google Scholar 

  37. Ryan, R. M., & Deci, E. L. (2001). On happiness and human potentials: A review of research on hedonic and eudaimonic well-being. Annual Review of Psychology, 52, 141–166.

    PubMed  Article  Google Scholar 

  38. Saucier, G. (1994). Mini-Markers: A brief version of Goldberg’s unipolar Big-Five markers. Journal of Personality Assessment, 63, 506–516.

    PubMed  Article  Google Scholar 

  39. Schimmack, U., Oishi, S., Furr, R. M., & Funder, D. C. (2004). Personality and life satisfaction: A facet-level analysis. Personality & Social Psychology Bulletin, 30, 1062–1075.

    Article  Google Scholar 

  40. Shih, T.-H., & Fan, X. (2008). Comparing response rates from Web and mail surveys: A meta-analysis. Field Methods, 20, 249–271.

    Article  Google Scholar 

  41. Steel, P., Schmidt, J., & Shultz, J. (2008). Refining the relationship between personality and subjective well-being. Psychological Bulletin, 134, 138–161.

    PubMed  Article  Google Scholar 

  42. Tabachnick, B. G., & Fidell, L. S. (1989). Using multivariate statistics (2nd ed.). New York: Harper & Row.

    Google Scholar 

  43. Tseng, H.-M., Macleod, H. A., & Wright, P. (1997). Computer anxiety and measurement of mood change. Computers in Human Behavior, 13, 305–316.

    Article  Google Scholar 

  44. Vadillo, M. A., Bárcena, R., & Matute, H. (2006). The Internet as a research tool in the study of associative learning: An example from overshadowing. Behavioural Processes, 73, 36–40.

    PubMed  Article  Google Scholar 

  45. Vallejo, M. A., Mañanes, G., Comeche, M. I., & Díaz, M. I. (2007). Comparison between administration via Internet and paper-andpencil administration of two clinical instruments: SCL-90-R and GHQ-28. Journal of Behavior Therapy & Experimental Psychiatry, 39, 201–208.

    Article  Google Scholar 

  46. Vereecken, C. A., & Maes, L. (2006). Comparison of a computer-administered and paper-and-pencil-administered questionnaire on health and lifestyle behaviors. Journal of Adolescent Health, 38, 426–432.

    PubMed  Article  Google Scholar 

  47. Vittersø, J., & Nilsen, F. (2002). The conceptual and relational structure of subjective well-being, neuroticism, and extraversion: Once again, neuroticism is the important predictor of happiness. Social Indicators Research, 57, 89–118.

    Article  Google Scholar 

  48. Watson, D., & Clark, L. A. (1994). The PANAS-X: Manual for the Positive and Negative Affect Schedule—Expanded Form. Iowa City: University of Iowa, Department of Psychology.

    Google Scholar 

  49. Williams, J. E., McGraw, K. O., & Tew, M. D. (1999). Undergraduate labs and computers: The case for PsychExps. Behavior Research Methods, Instruments, & Computers, 31, 287–291.

    Article  Google Scholar 

  50. Yu, S.-C., & Yu, M.-N. (2007). Comparison of Internet-based and paper-based questionnaires in Taiwan using multisample invariance approach. CyberPsychology & Behavior, 10, 501–507.

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ryan T. Howell.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Howell, R.T., Rodzon, K.S., Kurai, M. et al. A validation of well-being and happiness surveys for administration via the Internet. Behavior Research Methods 42, 775–784 (2010). https://doi.org/10.3758/BRM.42.3.775

Download citation

Keywords

  • Life Satisfaction
  • Negative Affect
  • Internet Survey
  • College Student Sample
  • Subjective Happiness Scale