Abstract
To date, many studies have assessed the measurement invariance of a wide variety of measures across Internet and paper-and-pencil conditions; however a relative dearth exists in the literature investigating measurement invariance across administration modes for differing subgroups of respondents. Using MIMIC modeling, this study assessed whether gender and age of the respondent systematically influenced responding according to administration mode above and beyond measurement invariance. Consistent with past research, this study demonstrated that job satisfaction ratings were indeed measurement invariant across Internet and paper-and-pencil conditions, however, older respondents tend to differentially rate job satisfaction according to administration mode. Implications are discussed.
Article PDF
Similar content being viewed by others
References
Ayersman, D., &Reed, W. (1996). Effects of learning styles, programming and gender on computer anxiety.Journal of Educational Computing Research,28, 148–161.
Barak, A., &Cohen, L. (2002). Empirical examination of an online version of the Self Directed Search.Journal of Career Assessment,10, 387–400.
Barak, A., &English, N. (2002). Prospects and limitations of psychological testing on the Internet.Journal of Technology in Human Services,19, 65–89.
Bedeian, A. G., Ferris, G. R., &Kacmar, K. M. (1992). Age, tenure, and job satisfaction: A tale of two perspectives.Journal of Vocational Behavior,40, 33–48.
Bentler, P. M, &Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures.Psychological Bulletin,88, 588–606.
Boomsma, A. (1983).On the robustness of LISREL (maximum likelihood estimation) against small sample size and non-normality. Amsterdam: Sociometric Research Foundation.
Brosnan, M. J. (1998). The impact of psychological gender, gender-related perceptions, significant others and the introducer of technology upon computer anxiety in students.Journal of Educational Computing Research,18, 63–78.
Buchanan, T. (2001). Online personality assessment: Equivalence of traditional and WWW personality measures. In U.-D. Reips & M. Bosnjak (Eds.),Dimensions of Internet Science (pp. 57–74), Lengerich, Germany: Pabst.
Buchanan, T., &Smith, J. L. (1999). Using the Internet for psychological research: Personality testing on the World-Wide Web.British Journal of Psychology,90, 125–144.
Burnkrant, S. R., & Taylor, C. D. (2001).Equivalence of traditional and Internet-based data collection: Three multigroup analyses. Paper presented at the 16th Annual Conference of the Society of Industrial and Organizational Psychology, San Diego, April, 2001.
Carmines, E., &McIver, J. (1981).Analyzing models with unobservable variables. In G.W. Bohrnstedt & E. F. Borgotta (Eds.), Social measurement: Current issues (pp. 65–115). Beverly Hills, CA: Sage.
Cheung, G. W. (1999). Multifaceted conceptions of self-other ratings disagreement.Personnel Psychology,52, 1–36.
Cheung, G. W., &Rensvold, R. B. (2001). The effects of model parsimony and sampling error on the fit of structural equation models.Organizational Research Methods,4, 236–264. 962.
Church, A. H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results.Personnel Psychology,54, 937–962.
Conoley, J. C., &Impara, J. C. (Eds.), (1995). Thetwelfth mental measurements yearbook. Lincoln: University of Nebraska, Buros Institute of Mental Measurements.
Dillman, D. A., &Bowker, D. K. (2001). The Web questionnaire challenge to survey methodologists. In U.-D. Reips & M. Bosnjak (Eds.),Dimensions of Internet Science (pp. 159–178), Lengerich, Germany: Pabst.
Ding, L, Velicer, W., &Harlow, L. (1995). Effects of estimation methods, number of indicators per factor, and improper solution on structural equation modeling fit indices.Structural Equation Modeling,2, 119–144.
Dupin-Bryant, P. (2002). Reducing computer anxiety in adults learning to use microcomputers.Journal of Extension,40.
Durndell, A., &Haag, Z. (2001). Computer self efficacy, computer anxiety, and attitudes towards the Internet and reported experience with the Internet, by gender, in an East European sample.Computers in Human Behavior,18, 521–535.
Etchegaray, J. M., Sexton, J. B., Helmreich, R. L., & Thomas, E. J. (2003).Job satisfaction ratings: Measurement equivalence across physicians and nurses. Paper presented at the 18th Annual Conference of the Society of Industrial and Organizational Psychology, Orlando, April, 2003.
Fenlason, K. J. (2000).Multiple data-collection methods in a 360 feedback program: Implications for use and interpretation. Paper presented at the 15th Annual Conference of the Society of Industrial and Organizational Psychology, New Orleans, April, 2000.
Frericks, L., Ehrhart, K., and O’Connell, M. (2003).Computer anxiety and test performance: Comparing selection test formats. Paper presented at the 18th Annual Conference of the Society of Industrial and Organizational Psychology, Orlando, April, 2003.
Gibson, J. L., &Klein, S. M. (1970). Employee attitudes as a function of age and length of service: A re-conceptualization.Academy of Management Journal,13, 411–425.
Glass, C. andKnight, L. (1988). Cognitive factors in computer anxiety.Cognitive Therapy & Research,12, 351–366.
Glenn, W. D., Taylor, P. A., &Weaver, C. N. (1977). Age and job satisfaction among males and females: A multi-variate, multi-survey study.Journal of Applied Psychology,62, 189–193.
Gregson, T. (1987). Factor analysis of a multiple-choice format for job satisfaction.Psychological Reports,61, 747–750.
Guadagnoli, E., &Velicer, W. (1988). Relation of sample size to the stability of component patterns.Psychological Bulletin,103, 265–275.
Guion, R. M. (1983). Comments on Hunter. In F. Landy, S. Zedeck, & J. Cleveland (Eds.),Performance measurement and theory (pp. 267–275). Hillsdale, NJ: Erlbaum.
Hodson, R. (1989). Gender differences in job satisfaction: Why aren’t women more dissatisfied.Sociological Quarterly,30, 385–399.
Honeyman, D. S., & White, W. J. (1987). Computer anxiety in educators learning to use The computer: A preliminary report.Journal of Research on Computing in Education,129–138.
Hu, L., &Bentler, P. M. (1999). Cut off criteria for fit indexes in covariance structural analysis: Conventional versus new alternatives.Structural Equation Modeling,6, 1–55.
Hu, L., Bentler, P. M., &Kano, Y. (1992). Can test statistics in covariance structure analysis be trusted?Psychological Bulletin,112, 351–362
Johnson, N.D. (1987). Effects of inservice training on writing apprehension and computer anxiety in elementary school teachers.Dissertation Abstracts International,48, 904–909.
Joinson, A. N. (2001). Self-disclosure in computer-mediated communication: The role of self-awareness and visual anonymity.European Journal of Social Psychology,31, 177–192.
Joreskog, K. G., &Goldberger, A. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable,Journal of the American Statistical Association,10, 631–639.
Judge, T. A., Locke, E. A., Durham, C. C., &Kluger A. N. (1998). Dispositional effects on job and life satisfaction: The role of core evaluations.Journal of Applied Psychology,83, 17–34.
Judge, T. A., &Watanabe, S. (1998). Another look at the job satisfaction-life satisfaction relationship.Journal of Applied Psychology,78, 939–948.
Kacmar, E.M., &Ferris, G.R. (1989). Theoretical and methodological considerations in the age-job satisfaction relationship.Journal of Applied Psychology,74, 201–207.
Keeler, C.M., &Anson, R. (1995). An assessment of cooperative learning used for basic computer skills instruction in the college classroom.Journal of Educational Computing Research,12, 379–393.
Krantz, J. H., &Dalal, R. (2000). Validity of Web-Based Psychological Research. In M. H. Birnbaum (Ed.),Psychological experiments on the Internet (pp. 35–60). New York: Academic Press.
Lam, S. S. K., Hui, C, Law, K. S. (1999). Organizational citizenship behavior: Comparing perspectives of supervisors and subordinates across four international samples.Journal of Applied Psychology,84, 594–601.
Liu, M., Reed, W.M., &Phillips, P.D. (1992). Teacher education students and computers: major, prior computer experience, occurrence, and anxiety.Journal of Research on Computing in Education,24, 457–467.
Mahar, D., Henderson, R., andDeane, F. (1997). The effects of computer anxiety, state anxiety, and computer experience on user’s performance on computer-based tasks.Personality & Individual Differences,22, 683–692.
Maurer, T. J., Raju, N. S., &Collins, W. C. (1998). Peer and subordinate performance appraisal measurement equivalence.Journal of Applied Psychology,83, 693–702. Meier, S. (1985). Computer aversion.Computers in Human Behavior,1, 171–179.
Muthen, L., &Muthen B. (2001).Mplus User’s Guide. Los Angeles: Author.
Meyerson, P. andTryon, W. W. (2003). Validating Internet research: A test of the psychometric equivalence of Internet and in-person samples.Behavior Research Methods, Instruments, & Computers,35, 614–620.
Nunnally, J. C. (1978).Psychometric theory (2nd ed.). New York: McGraw-Hill.
Pasveer, K. A., &Ellard, J. H. (1998). The making of a personality inventory: Help from the WWW.Behavior Research Methods, Instruments and Computers,30, 309–313.
Paxton, A., &Turner, E. (1984). The application of human factors to the needs of the novice computer user.International Journal of Man-Machine Studies,20, 137–156.
Ployhart, R. E., Weekley, J., Holtz, B., &Kemp, C. (2003). Web-based vs. pencil testing: Are test scores comparable across applicants and incumbents?Personnel Psychology,56, 733–752.
Preckel, F., &Thieman, H. (2003). Online- versus paper-pencil-version of a high potential intelligence test.Swiss Journal of Psychology,62,131–138
Reips, U.-D. (2002). Standards for Internet-based experimenting.Experimental Psychology,49, 243–256.
Reips, U.-D., &Franek, L. (2004). Mitarbeiterbefragungen per Internet oder Papier? Der Einfluss von Anonymität, Freiwilligkeit und Alter auf das Antwortverhalten [Employee surveys via Internet or paper? The influence of anonymity, voluntariness and age on answering behavior].Wirtschaftspsychologie,6, 67–83.
Robinson, P. W. (Ed.) (1981).Fundamentals of experimental psychology. Englewood Cliffs, MJ: Prentice-Hall.
Rosen, L.D., &Weil, M.M. (1994). Computer availability, computer experience and technophobia among public school teachers.Computers in Human Behavior,11, 9–31.
Rosen, L. D., &Weil, M. (1995). Adult and teenager consumer users of technology: Potholes on the information superhighway?Journal of Consumer Affairs,29, 55–84.
Rosznowski, M. (1989). Examination of the measurement properties of the Job Descriptive Index with experimental items.Journal of Applied Psychology,74, 805–814.
Smith, P.C., Kendall, L. M., &Hulin, C. L. (1969).The measurement of satisfaction in work and retirement. Chicago: Rand McNally.
Spera, T. J., & Moye, B. G. (2001).Measurement equivalence between paper and Web survey methods in a multinational company. Paper presented at the 16th Annual Conference of the Society of Industrial and Organizational Psychology, San Diego, April, 2001.
Stanton, J. M. (1998). An empirical assessment of data collection using the Internet.Personnel Psychology,51, 709–724.
Sterns, H. L., &Doverspike, D. (1989). Aging and the training and learning process. In L. Goldstein,Training and development in organization. San Francisco: Jossey-Bass.
Thompson, L. F., Surface, E. A., Martin, D. L., &Sanders, M. G. (2003). From paper to pixels: Moving personnel surveys to the Web.Personnel Psychology,56, 197–227.
Vandenberg, R. J., &Lance, C. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research.Organizational Research Methods,4, 4–70.
Varca, P. E., Shaffer, G. S., &McCauley, C. D. (1983). Gender differences in job satisfaction revisited.Academy of Management Journal,26, 384–353.
Whitaker, B. (April, 2004).The Impact of Data Collection Methodology on Popular Data-Reporting Indices. Poster presented at the 19th Annual Society for Industrial and Organizational Psychology Conference, Chicago.
Whitaker, B. (July, 2004).Internet-based attitude assessment: Does gender affect measurement equivalence? Poster presented at the 112th Annual American Psychological Association Convention, Honolulu.
Williams, S., Ogletree, S., Woodburn, W., andRaffeld, P. (1993). Gender roles, computer attitudes, and dyadic interaction performance in college students.Sex Roles,29, 515–525.
Woehr, D. J., Sheehan, M. K., & Bennett, W. Jr. (1999).Understanding disagreement across rating sources: An assessment of the measurement equivalence of raters in 360 degree feedback systems. Poster presented at the Fourteenth Annual Conference of the Society for Industrial and Organizational Psychology, Atlanta.
Yun, G. W., &Trumbo, C. W. (2000). Comparative response to a survey executed by post, email, & Web form.Journal of Computer-Mediated Communication,6, 1–26.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Whitaker, B.G., McKinney, J.L. Assessing the measurement invariance of latent job satisfaction ratings across survey administration modes for respondent subgroups: A MIMIC modeling approach. Behavior Research Methods 39, 502–509 (2007). https://doi.org/10.3758/BF03193019
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.3758/BF03193019