Given the common practice of using employee attitude surveys as a group-level intervention, this study used a group-level approach to examine the relationship between group satisfaction and group nonresponse.
Samples from four large organizations enabled job satisfaction scores to be aggregated to the work group level and correlated with group-level response rates. Additional regression analysis was conducted to control for a number of confounding variables at the group level.
Aggregate job satisfaction showed significant associations with group-level response rates across each of the samples examined. Work groups with higher aggregate job satisfaction had significantly higher response rates. Regression analyses showed that, in addition to job satisfaction, work group size, heterogeneity in tenure, and heterogeneity in gender composition all had significant effects on response rates.
Social influence processes may operate at the group level to increase homogeneity of job-relevant attitudes and similarity in survey response behavior. Future research should be designed to investigate the effects of group-level variables on nonresponse.
The current study adds to the literature by demonstrating that work group variables may play an important role in explaining nonresponse in employee attitude surveys. Because the processes underlying survey response are likely to be different at different levels of analysis, the investigation of nonresponse as a group-level phenomenon creates new opportunities for research and practice.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
We conducted hierarchical MGCFA for heterogeneous groups to evaluate the psychometric equivalence of the measure across companies, allowing for different items in the different companies (Hattrup et al. 2007). In a first step, each item was allowed to load on its respective factor in each group. In a second step, we constrained the loadings of the shared items to equality across companies. The overall model across all four samples showed adequate fit with the data (CFI = 0.912, IFI = 0.912, RMSEA = 0.036), and the restriction on item loadings did not result in a significant decrease in model fit (ΔCFI = 0.003) in accordance with Cheung and Rensvold (2002). Thus, results of the MGCFA analyses support the overall equivalence of scales across companies.
We chose to compute heterogeneity indices as indicators for group composition instead of group mean levels, given the potential theoretical importance of group heterogeneity. We also, however, repeated the analysis with the group mean descriptors (mean levels for age and tenure and proportions for gender and employment status) instead of the heterogeneity indexes. That is, we conducted a two-step hierarchical linear regression analysis for sample 4, using aggregate job satisfaction, group function and the group means as independent variables. In a first step we entered the means for gender, age, tenure, employment status, and group size. Again, all variables were centered at their grand means. Group size and tenure were found to be significantly related to group response rates. Entering aggregate job satisfaction in a second step resulted in a significant increase in R 2 (ΔR 2 = 0.02, ΔF = 4.41, p = 0.04). Once aggregate job satisfaction was entered in the regression, however, the only significant covariate was group size.
Anderson, N. R., & West, M. A. (1998). Measuring climate for work group innovation: Development and validation of the team climate inventory. Journal of Organizational Behavior, 19, 235–258.
Anseel, F., Lievens, F., Schollaert, E., & Choragwicka, B. (2010). Response rates in organizational science, 1995–2008: A meta-analytic review and guidelines for survey researchers. Journal of Business and Psychology, 25, 335–349.
Armstrong, J. S., & Lusk, E. J. (1987). Return postage in mail surveys: A meta-analysis. Public Opinion Quarterly, 51, 233–248.
Armstrong, J. S., & Overton, T. S. (1977). Estimating nonresponse bias in mail surveys. Journal of Marketing Research, 16, 396–402.
Barr, C., Spitzmueller, C., & Stuebing, K. (2008). Too stressed out to participate? Examining the relation between stressors and survey response behavior. Journal of Occupational Health Psychology, 13, 232–243.
Bartko, J. J. (1976). On various intraclass correlation reliability coefficients. Psychological Bulletin, 83, 762–765.
Björklund, C., Grahn, A., Jensen, I., & Bergstrom, G. (2007). Does survey feedback enhance the psychosocial work environment and decrease sick leave? European Journal of Work and Organizational Psychology, 16, 76–93.
Blau, P. M. (1977). Inequality and heterogeneity: A primitive theory of social structure. New York: Free Press.
Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1, 355–373.
Bliese, P. D. (2000). Within-group agreement, non-independence, and reliability: Implications for data aggregation and analysis. In K. J. Klein & S. W. J. Kozlowski (Eds.), Multilevel theory, research, and methods in organizations: Foundations, extensions, and new directions (pp. 349–381). San Francisco: Jossey-Bass.
Bliese, P. D., & Halverson, R. R. (1996). Individual and nomothetic models of job stress: An examination of work hours, cohesion, and well-being. Journal of Applied Social Psychology, 26, 1171–1189.
Bliese, P. D., & Halverson, R. R. (1998). Group size and measures of group-level properties: An examination of eta-squared and ICC values. Journal of Management, 24, 157–172.
Born, D. H., & Mathieu, J. E. (1996). Differential effects of survey-guided feedback. Group & Organization Management, 21, 388–403.
Bosnjak, M., & Batinic, B. (2002). Understanding the willingness to participate in online surveys—The case of e-mail questionnaires. In B. Batinic, U. Reips, & M. Bosnjak (Eds.), Online social sciences (pp. 81–92). Seattle, WA: Hogrefe & Huber.
Bosnjak, M., Tuten, T. L., & Wittmann, W. W. (2005). Unit (non)response in web-based access panel surveys: An extended planned-behavior approach. Psychology and Marketing, 22, 489–505.
Burke, W. W. (2006). Organizational survey as leverage for organization development and change. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 131–149). San Francisco: Jossey-Bass.
Burke, W. W., Coruzzi, C. A., & Church, A. H. (1996). The organizational survey as an intervention for change. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 41–66). San Francisco: Jossey-Bass.
Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.
Chan, D. (1998). Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology, 83, 234–246.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233–255.
Chiaburu, D. S., & Harrison, D. A. (2008). Do peers make the place? Conceptual synthesis and meta-analysis of coworker effects on perceptions, attitudes, OCBs, and performance. Journal of Applied Psychology, 93, 1082–1103.
Church, A. H. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57, 62–79.
Church, A. H., & Oliver, D. H. (2006). The importance of taking action, not just sharing survey feedback. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 33–52). San Francisco: Jossey-Bass.
Church, A. H., & Waclawski, J. (1998). Designing and using organizational surveys. Aldershot: Gower.
Cialdini, R. B., & Trost, M. R. (1998). Social influence: Social norms, conformity, and compliance. In D. Gilbert, S. Fiske, & G. Lindzey (Eds.), The handbook of social psychology (4th ed., Vol. 2, pp. 151–192). New York: McGraw-Hill.
Clausen, J. A., & Ford, R. N. (1947). Controlling bias in mail questionnaires. Journal of the American Statistical Association, 42, 497–511.
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. Mahwah, NJ: Lawrence Erlbaum.
Conway, J. M., & Lance, C. E. (2010). What reviewers should expect from authors regarding common method bias in organizational research. Journal of Business and Psychology, 25(3), 325–334.
Cook, C., Heath, F., & Thompson, R. L. (2000). A meta-analysis of response rates in web- or internet based surveys. Educational and Psychological Measurement, 60, 821–836.
Cropanzano, R., & Mitchel, S. M. (2005). Social exchange theory: An interdisciplinary review. Journal of Management, 31, 874–900.
Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. New York: Wiley.
Edwards, J. E., Thomas, M. D., Rosenfeld, O., & Booth-Kewley, S. (1997). How to conduct organizational surveys. A step-by-step guide. London: Sage.
Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of unit-level organizational citizenship behavior. Personnel Psychology, 57, 61–94.
Ehrhart, M. G., & Naumann, S. E. (2004). Organizational citizenship behavior in work groups: A group norms approach. Journal of Applied Psychology, 89(6), 960–974.
Falletta, S. V., & Combs, W. (2002). Surveys as a tool for organization development and change. In J. Waclawski & A. H. Church (Eds.), Organization development: A data-driven approach to organizational change (pp. 78–102). San Francisco: Jossey-Bass.
Festinger, L., & Thibaut, J. (1951). Interpersonal communication in small groups. Journal of Abnormal and Social Psychology, 46, 92–99.
Folkman, J. (1998). Employee surveys that make a difference. Provo, UT: Executive Excellence.
Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail survey response rate: A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52, 467–491.
French, W., & Bell, C. (1999). Organization development: Behavioural science interventions for organization improvement. New Jersey: Prentice Hall.
Glick, W. H. (1985). Conceptualizing and measuring organizational and psychological climate: Pitfalls in multilevel research. Academy of Management Review, 10, 601–616.
Goodman, J., & Blum, T. (1996). Assessing the non-random sampling effects of subject attrition in longitudinal research. Journal of Management, 22, 627–652.
Green, K. E. (1996). Sociodemographic factors and mail survey response. Psychology and Marketing, 13, 171–184.
Harter, J. K., & Schmidt, F. L. (2006). Connecting employee satisfaction to business unit performance. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 102–130). San Francisco: Jossey-Bass.
Hartman, R. L., & Johnson, J. D. (1989). Social contagion and multiplexity communication networks as predictors of commitment and role ambiguity. Human Communication Research, 15, 523–548.
Hattrup, K., Mueller, K., & Joens, I. (2007). The effects of nations and organizations on work value importance: A cross-cultural investigation. Applied psychology: An International Review, 56(3), 479–499.
Heberlein, T. A., & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43, 447–462.
Helgeson, J. G., Voss, K. E., & Terpening, W. D. (2002). Determinants of mail-survey response: Survey design factors and respondent factors. Psychology and Marketing, 19, 303–328.
Hinrichs, J. R. (1996). Feedback, action planning, and follow-through. In A. I. Kraut (Ed.), Organizational surveys: Tools for assessment and change (pp. 255–278). San Francisco: Jossey-Bass.
Hogg, M. A., & Abrams, D. (1988). Social identifications: A social psychology of intergroup relations and group processes. London: Routledge.
Hogg, M. A., & Grieve, P. (1999). Social identity theory and the crisis of confidence in social psychology: A commentary, and some research on uncertainty reduction. Asian Journal of Social Psychology, 2, 79–93.
Hutchison, J., Tollefson, N., & Wigington, H. (1987). Response bias in college freshmen’s responses to mail surveys. Research in Higher Education, 26, 99–106.
Jackson, S. E. (1992). Team composition in organizational settings: Issues in managing an increasingly diverse work force. In S. Worchel, W. Wood, & J. A. Simpson (Eds.), Group process and productivity (pp. 138–173). Newbury Park: Sage.
Jackson, S., Brett, J., Sessa, V., Cooper, D., Julin, J., & Peyronnin, K. (1991). Some differences make a difference: Individual dissimilarity and group heterogeneity as correlates of recruitment, promotions, and turnover. Journal of Applied Psychology, 76, 675–689.
James, L. R. (1982). Aggregation bias in estimates of perceptual agreement. Journal of Applied Psychology, 67, 219–229.
James, J. M., & Bolstein, R. (1990). The effect of material incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54, 346–361.
James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.
Johnson, L. C., Beaton, R., Murphy, S., & Pike, K. (2000). Sampling bias and other methodological threats to the validity of health survey research. International Journal of Stress Management, 7, 247–268.
Kantor, J. (1991). The effects of computer administration and identification on the Job Descriptive Index. Journal of Business and Psychology, 75, 309–323.
Kirkman, B. L., & Rosen, B. (1999). Beyond self-management: Antecedents and consequences of team empowerment. Academy of Management Journal, 42, 58–74.
Klein, K. J., Dansereau, F., & Hall, R. I. (1994). Levels issues in theory development, data collection, and analysis. Academy of Management Review, 19, 195–229.
Klein, K. J., & Kozlowski, S. W. J. (2000). From micro to meso: Critical steps in conceptualizing and conducting multilevel research. Organizational Research Methods, 3, 211–236.
Kozlowski, S. W. J., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161–167.
Kraut, A. (2006). Getting action from organizational surveys: New concepts, technologies and applications. San Francisco: Jossey Bass.
LeBreton, J. M., Burgess, J. R. D., Kaiser, R. B., Atchley, E. K., & James, L. R. (2003). The restriction of variance hypothesis and interrater reliability and agreement: Are ratings from multiple sources really dissimilar? Organizational Research Methods, 6, 80–128.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Lubin, B., Levitt, E., & Zukerman, M. S. (1962). Some personality differences between respondents and nonrespondents to a survey questionnaire. Journal of Consulting Psychology, 26, 192.
Mason, C. M., & Griffin, M. A. (2002). Group task satisfaction: Applying the construct of job satisfaction to groups. Small Group Research, 33, 271–312.
Mason, C. M., & Griffin, M. A. (2003a). Identifying group task satisfaction at work. Small Group Research, 34, 413–442.
Mason, C. M., & Griffin, M. A. (2003b). Group absenteeism and positive affective tone: A longitudinal study. Journal of Organizational Behavior, 24, 667–687.
Mayer, C., & Pratt, R. (1966). A note on nonresponse in a mail survey. Public Opinion Quarterly, 30, 646–667.
McConnell, J. H. (2003). How to design, implement, and interpret an employee survey. New York: Amacom.
McDaniel, S., Madden, C. S., & Verille, P. (1987). Do topic differences affect survey nonresponse? Journal of the Market Research Society, 29, 55–67.
Moliner, C., Martinez-Tur, V., Peiro, J. M., Ramos, J., & Cropanzano, R. (2005). Relationships between organizational justice and burnout at the work-unit level. International Journal of Stress Management, 12, 99–116.
Moore, S. (1999). Understanding and managing diversity among groups at work: Key issues for organizational training and development. Journal of. European Industrial Training, 23(4/5), 208–217.
Mueller, K., Liebig, C., & Hattrup, K. (2007). Computerizing organizational attitudes surveys: An investigation of the measurement equivalence of a multifaceted job satisfaction measure. Journal of Educational and Psychological Measurement, 67, 658–678.
Mueller, K., Hattrup, K., & Straatmann, T. (2011). Globally surveying in English: Investigation of the measurement equivalence of a job satisfaction measure across bilingual and native English speakers. Journal of Occupational and Organizational Psychology, 84, 618–624.
Nadler, D. A. (1977). Feedback and organization development: Using data-based methods in organization development. Reading, MA: Addison-Wesley.
Nesterkin, D. A., & Ganster, D. C. (2012). The effects of nonresponse rates on group-level correlations. Journal of Management. doi:10.1177/0149206311433853.
Newman, J. E. (1975). Understanding the organizational structure—Job attitude relationship through perceptions of the work environment. Organizational Behavior and Human Decision Processes, 14, 371–397.
Newman, D. A., & Sin, H.-P. (2009). How do missing data bias estimates of within-group agreement? Sensitivity of SD WG, CVWG, rWG(J), rWG(J) * and ICC to systematic nonresponse. Organizational Research Methods, 12, 113–147.
Organ, D. W., Podsakoff, P. M., & MacKenzie, S. B. (2006). Organizational citizenship behavior: Its nature, antecedents and consequences. Thousand Oaks, CA: Sage.
Ostroff, C. (1992). The relationship between satisfaction, attitudes, and performance: An organizational level analysis. Journal of Applied Psychology, 77, 963–974.
Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.
Podsakoff, N. P., Whiting, S. W., Podsakoff, P. M., & Blume, B. D. (2009). Individual- and organizational-level consequences of organizational citizenship behaviors: A meta-analysis. Journal of Applied Psychology, 94(1), 122–141.
Pritchard, R. D., & Karasick, B. W. (1973). The effects of organizational climate on managerial job performance and job satisfaction. Organizational Behavior and Human Performance, 9, 126–146.
Rogelberg, S. G. (Ed.). (2002). Handbook of research methods in industrial and organizational psychology. London: Blackwell.
Rogelberg, S. G. (2006). Understanding nonresponse and facilitating response to organizational surveys. In A. I. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 312–325). San Francisco: Jossey-Bass.
Rogelberg, S. G., Conway, J. M., Sederburg, M. E., Spitzmueller, C., Aziz, S., & Knight, W. E. (2003). Profiling active and passive-non-respondents to an organizational survey. Journal of Applied Psychology, 88, 1104–1114.
Rogelberg, S. G., Fisher, G. G., Maynard, D. C., Hakel, M. D., & Horvath, M. (2001). Attitudes toward surveys: Development of a measure and its relationship to respondent behavior. Organizational Research Methods, 4, 3–25.
Rogelberg, S. G., & Luong, A. (1998). Nonresponse to mailed surveys: A review and guide. Current Direction in Psychological Science, 7, 60–65.
Rogelberg, S. G., Luong, A., Sederburg, M. E., & Cristol, D. S. (2000). Employee attitude surveys: Examining the attitudes of noncompliant employees. Journal of Applied Psychology, 85, 284–293.
Rogelberg, S. G., Spitzmueller, C., Little, I. S., & Reeve, C. L. (2006). Understanding response behavior to an online special topics organizational satisfaction survey. Personnel Psychology, 59, 903–923.
Rogelberg, S. G., & Stanton, J. M. (2007). Understanding and dealing with organizational survey nonresponse. Organizational Research Methods, 10, 195–209.
Rousseau, D. M. (1978). Characteristics of departments, positions, and individuals: Contexts for attitudes and behavior. Administrative Science Quarterly, 23, 521–540.
Ryan, A. M., Schmit, M. J., & Johnson, R. (1996). Attitudes and effectiveness: Examining relations at an organizational level. Personnel Psychology, 49, 853–882.
Salancik, G. R., & Pfeffer, J. (1978). A social information processing approach to job attitudes and task design. Administrative Science Quarterly, 23, 224–253.
Sax, L., Gilmartin, S., & Bryant, A. (2002). Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education, 44, 409–432.
Schiemann, W., & Morgan, B. (2006). Strategic surveys: Linking people to business strategy. In A. Kraut (Ed.), Getting action from organizational surveys: New concepts, technologies, and applications (pp. 76–100). San Francisco: Jossey-Bass.
Schneider, B., Hanges, P. J., Smith, D. B., & Salvaggio, A. N. (2003). Which comes first: Employee attitudes or organizational financial and market performance? Journal of Applied Psychology, 88, 836–851.
Schyns, B., Paul, T., Mohr, G., & Blank, H. (2005). Comparing antecedents and consequences of leader-member-exchange in a German working context to findings in the US. European Journal of Work and Organizational Psychology, 14(1), 1–22.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
Smith, P. B. (2003). Meeting the challenge of cultural difference. In D. Tjosvold & K. Leung (Eds.), Cross-cultural management: Foundation and future (pp. 59–73). Aldershot: Ashgate.
Smith, P. C., Kendall, L. M., & Hulin, C. L. (1985). The job descriptive index (rev. ed.). Bowling Green, OH: Department of Psychology, Bowling Green State University.
Song, L. J., Tsui, A. S., & Law, K. S. (2009). Unpacking employee responses to organizational exchange mechanisms: The role of social and economic exchange perceptions. Journal of Management, 35, 56–93.
Sosdian, C. P., & Sharp, L. M. (1980). Nonresponse in mail surveys: Access failure or respondent resistance. Public Opinion Quarterly, 44, 396–402.
Spitzmueller, C., Barr, C., Glenn, D., Rogelberg, S., & Daniel, P. (2006). “If you treat me right, I reciprocate”: Examining the role of exchange in organizational survey nonresponse. Journal of Organizational Behavior, 27, 19–35.
Spitzmueller, C., & Glenn, D. M. (2006). Nonresponse in employee surveys—A review and directions for future work. In M. Braun (Ed.), Survey methodology (pp. 139–162). Mannheim: ZUMA Press.
Spitzmueller, C., Glenn, D., Sutton, M. M., Barr, C. D., & Rogelberg, S. G. (2007). Are survey nonrespondents bad organizational citizens? International Journal of Selection and Assessment, 15, 449–459.
Stinchcombe, A. L., Jones, C., & Sheatsley, P. (1981). Nonresponse bias for attitude questions. Public Opinion Quarterly, 45, 359–375.
Tajfel, H., & Turner, J. C. (1979). An integrative theory of intergroup conflict. In W. G. Austin & S. Worchel (Eds.), The social psychology of intergroup relations (pp. 33–47). Monterey, CA: Brooks/Cole.
Taris, T. W., & Schreurs, P. J. (2007). How may nonresponse affect findings in organizational surveys? The tendency-to-the-positive-effect. International Journal of Stress Management, 14(3), 249–259.
Thompson, L. F., & Surface, E. A. (2007). Employee surveys administered online: Attitudes toward the medium, nonresponse, and data representativeness. Organizational Research Methods, 10, 241–261.
Turner, J. C. (1982). Towards a cognitive redefinition of the social group. In H. Tajfel (Ed.), Social identity and intergroup relations (pp. 15–40). Cambridge: Cambridge University Press.
Vincent, C. (1964). Socioeconomic status and familial variables in mail questionnaire responses. American Journal of Sociology, 69, 647–653.
Wallace, D. (1954). A case for-and-against mail questionnaires. Public Opinion Quarterly, 18, 40–52.
Whelan, T. J. (2008). Antecedents of anonymity perceptions in web-based surveys. Paper presented at the 23rd Annual Meeting of the Society for Industrial and Organizational Psychology, San Francisco, CA.
Williams, R. (2007). Achieving leadership results through employee surveys. Delta, BC, Canada: TWI Surveys.
Wolf, T. R., Hattrup, K., & Mueller, K. (2011). A cross-national investigation of the measurement equivalence of computerized organizational attitude surveys: A two study design in multiple nations. Journal of Organizational Computing and Electronic Commerce, 21(3), 246–263.
Yammarino, F. J., & Markham, S. E. (1992). On the application of within and between analysis: Are absence and affect really group-based phenomena? Journal of Applied Psychology, 77, 168–176.
About this article
Cite this article
Fauth, T., Hattrup, K., Mueller, K. et al. Nonresponse in Employee Attitude Surveys: A Group-Level Analysis. J Bus Psychol 28, 1–16 (2013). https://doi.org/10.1007/s10869-012-9260-y