Psychometric Properties and Administration Measurement Invariance of Social Phobia Symptom Measures: Paper-Pencil vs. Internet Administrations

  • Michiyo Hirai
  • Laura L. Vernon
  • George A. Clum
  • Susan T. Skidmore


The psychometric properties of the online versions of assessment instruments for phobias and related symptoms were compared to those of paper-pencil versions. The examined measures were the Social Interaction Anxiety Scale (SIAS) and Social Phobia Scale (SPS). Students (n = 514) completed the instruments either in a paper-pencil in lab condition or an online at home condition. Means of the measures were largely equivalent between the two groups. Factor structures of the measures support construct validity of the two versions of the questionnaires. The coefficient alphas for the measures were reasonably high, ranging between .91 and .93 for both groups. Correlations between the measures were similar across groups. Overall, adequate psychometric properties of the online versions of the instruments were demonstrated, however, neither of the social phobia measures demonstrated measurement invariance between the two assessment modalities.


Internet Social phobia Assessment Psychometrics Measurement invariance 


  1. Andersson, E., Ljótsson, B., Smit, F., Paxling, B., Hedman, E., Lindefors, N., et al. (2011). Cost-effectiveness of internet-based cognitive behavior therapy for irritable bowel syndrome: results from a randomized controlled trial. BioMed Central Public Health, 11, 1–7.Google Scholar
  2. Austin, D. W., Carlbring, P., Richards, J. C., & Andersson, G. (2006). Internet administration of three commonly used questionnaires in panic research: equivalence to paper administration in Australian and Swedish samples of people with panic disorder. International Journal of Testing, 6, 25–39.CrossRefGoogle Scholar
  3. Baker, S. L., Heinrichs, N., Kim, H. J., & Hofmann, S. G. (2002). The Liebowitz social anxiety scale as a self-report instrument: a preliminary psychometric analysis. Behaviour Research and Therapy, 40, 701–715.PubMedCrossRefGoogle Scholar
  4. Buchanan, T. (2003). Internet-based questionnaire assessment: appropriate use in clinical contexts. Cognitive Behaviour Therapy, 32, 100–109.PubMedCrossRefGoogle Scholar
  5. Carlbring, P., Brunt, S., Bohman, S., Austin, D., Richards, J., Öst, L., et al. (2007). Internet vs. paper and pencil administration of questionnaires commonly used in panic/agoraphobia research. Computers in Human Behavior, 23, 1421–1434.CrossRefGoogle Scholar
  6. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304–1312.CrossRefGoogle Scholar
  7. Cole, M. S., Bedeian, A. G., & Feild, H. S. (2006). The measurement equivalence of web-based and paper-and-pencil measures of transformational leadership: a multinational test. Organizational Research Methods, 9, 339–368.CrossRefGoogle Scholar
  8. Coles, M. E., Cook, L. M., & Blake, T. R. (2007). Assessing obsessive compulsive symptoms and cognitions on the internet: evidence for the comparability of paper and internet administration. Behaviour Research and Therapy, 45, 2232–2240.PubMedCrossRefGoogle Scholar
  9. Curran, P. J., West, S. G., & Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1, 16–29.CrossRefGoogle Scholar
  10. Emmelkamp, P. M. G. (2005). Technological innovations in clinical assessment and psychotherapy. Psychotherapy and Psychosomatics, 74, 336–343.PubMedCrossRefGoogle Scholar
  11. Feldt, L. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34, 363–373.CrossRefGoogle Scholar
  12. Fortson, B. L., Scotti, J. R., Del Ben, K. S., & Chen, Y. (2006). Reliability and validity of an internet traumatic stress survey with a college student sample. Journal of Traumatic Stress. Special Issue: Dissemination: Transforming Lives through Transforming Care, 19, 709–720.Google Scholar
  13. Habke, A. M., Hewitt, P. L., Norton, G. R., & Asmundson, G. (1997). The social phobia and social interaction anxiety scales: an exploration of the dimensions of social anxiety and sex differences in structure and relations with pathology. Journal of Psychopathology and Behavioral Assessment, 19, 21–39.CrossRefGoogle Scholar
  14. Hedman, E., Ljótsson, B., Rück, C., Furmark, T., Carlbring, P., Lindefors, N., et al. (2010). Internet administration of self-report measures commonly used in research on social anxiety disorder: a psychometric evaluation. Computers in Human Behavior, 26, 736–740.CrossRefGoogle Scholar
  15. Heimberg, R. G., Mueller, G. P., Holt, C. S., Hope, D. A., & Liebowitz, M. R. (1992). Assessment of anxiety in social interaction and being observed by others: the social interaction anxiety scale and the social phobia scale. Behavior Therapy, 23, 53–73.CrossRefGoogle Scholar
  16. Henson, R. K. (2001). Understanding internal consistency reliability estimates: a conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177–189.Google Scholar
  17. Hirai, M., & Clum, G. A. (2008). Self-help therapies for anxiety disorders. In P. L. Watkins & G. A. Clum (Eds.), Handbook of self-help therapies (pp. 77–107). New York: Routledge/Taylor & Francis Group; Mahwah, NJ, US: Lawrence Erlbaum Associates Publishers.Google Scholar
  18. Horn, J. L., & McArdle, J. J. (1992). A practical and theoretical guide to measurement invariance in aging research. Experimental Aging Research. Special Issue: Quantitative Topics in Research on Aging, 18, 117–144.Google Scholar
  19. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: coventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.CrossRefGoogle Scholar
  20. Kenny, D. A., & McCoach, D. (2003). Effect of the number of variables on measures of fit in structural equation modeling. Structural Equation Modeling, 10, 333–351.CrossRefGoogle Scholar
  21. Liebowitz, M. R. (1987). Social phobia. Modern Problems of Pharmacopsychiatry, 22, 141–173.PubMedGoogle Scholar
  22. Marks, I. M., & Mathews, A. M. (1979). Brief standard self-rating for phobic patients. Behavior Research and Therapy, 17, 263–267.CrossRefGoogle Scholar
  23. Mattick, R. P., & Clarke, J. C. (1998). Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behaviour Research and Therapy, 36, 455–470.PubMedCrossRefGoogle Scholar
  24. McCabe, R. E., & Antony, M. M. (2002). Specific and social phobia. In M. Antony & D. Barlow (Eds.), Handbook of assessment and treatment planning for psychological disorders. New York: The Guilford Press.Google Scholar
  25. McCrae, R. R., Zonderman, A. B., Costa, P. T., Jr., Bond, M. H., & Paunonen, S. V. (1996). Evaluating replicability of factors in the revised NEO personality inventory: confirmatory factor analysis versus procrustes rotation. Journal of Personality and Social Psychology, 70, 552–566.CrossRefGoogle Scholar
  26. Meade, A. W., Michels, L. C., & Lautenschlager, G. J. (2007). Are internet and paper-and-pencil personality tests truly comparable?: An experimental design measurement invariance study. Organizational Research Methods, 10, 322–345.CrossRefGoogle Scholar
  27. Meredith, W. (1993). Measurement invariance, factor analysis and factorial invariance. Psychometrika, 58, 525–543.CrossRefGoogle Scholar
  28. Meredith, W., & Teresi, J. A. (2006). An essay on measurement and factorial invariance. Medical Care, 44, S69–S77.PubMedCrossRefGoogle Scholar
  29. Naglieri, J. A., Drasgow, F., Schmit, M., Handler, L., Prifitera, A., Margolis, A., et al. (2004). Psychological testing on the internet: new problems, old issues. American Psychologist, 59, 150–162.PubMedCrossRefGoogle Scholar
  30. Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5, 241–301.PubMedCrossRefGoogle Scholar
  31. Nunnally, J. C., & Bernstein, I. (1994). Psychometric theory. New York: McGraw-Hill.Google Scholar
  32. Osman, A., Gutierrez, P. M., Barrios, F. X., Kopper, B. A., & Chiros, C. E. (1998). The social phobia and social interaction anxiety scales: evaluation of psychometric properties. Journal of Psychopathology and Behavioral Assessment, 20, 249–264.CrossRefGoogle Scholar
  33. Raykov, T. (2004). Behavioral scale reliability and measurement invariance evaluation using latent variable modeling. Behavior Therapy, 35, 299–331.CrossRefGoogle Scholar
  34. Read, J. P., Farrow, S. M., Jaanimägi, U., & Ouimette, P. (2009). Assessing trauma and traumatic stress via the internet: measurement equivalence and participant reactions. Traumatology. Special Issue: Culture and International Contributions, 15, 94–102.Google Scholar
  35. Tate, D. F., & Zabinski, M. F. (2004). Computer and internet applications for psychological treatment: update for clinicians. Journal of Clinical Psychology, 60, 209–220.PubMedCrossRefGoogle Scholar
  36. Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. New York: Guilford.Google Scholar
  37. Turner, S. M., Beidel, D. C., Dancu, C. V., & Stanley, M. A. (1989). An empirically derived inventory to measure social fears and anxiety: the social phobia and anxiety inventory. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 1, 35–40.CrossRefGoogle Scholar
  38. Watson, D., & Friend, R. (1969). Measurement of social-evaluative anxiety. Journal of Consulting and Clinical Psychology, 33, 448–457.PubMedCrossRefGoogle Scholar
  39. Zlomke, K. R. (2009). Psychometric properties of internet administered versions of Penn State Worry Questionnaire (PSWQ) and Depression, Anxiety, and Stress Scale (DASS). Computers in Human Behavior, 25, 841–843.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Michiyo Hirai
    • 1
  • Laura L. Vernon
    • 2
  • George A. Clum
    • 3
  • Susan T. Skidmore
    • 4
  1. 1.Department of Psychology and AnthropologyUniversity of Texas- Pan AmericanEdinburgUSA
  2. 2.Florida Atlantic UniversityJupiterUSA
  3. 3.Virginia Polytechnic Institute and State UniversityBlacksburgUSA
  4. 4.Sam Houston State UniversityHuntsvilleUSA

Personalised recommendations