Skip to main content
SpringerLink
Log in

Rater Personality and Performance Dimension Weighting in Making Overall Performance Judgments

  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Purpose

The present study examined the effects of rater personality on the performance appraisal process. Specifically, we determined the relative weights that raters place on different performance dimensions when making overall performance evaluations, and examined whether rater personality influenced this weighting process. The literatures on social/political values and mate/friend selection were used as guiding frameworks in developing specific hypotheses.

Design/Methodology/Approach

A policy capturing method was used to address the research question in a sample of 192 Canadian undergraduate students. Students were asked to read a number of vignettes describing the teaching behaviours of hypothetical professors, and made overall performance judgments thereafter. Hierarchical Linear Modeling (HLM) was used to test the hypotheses.

Findings

Results indicated that when making overall performance ratings, raters with high levels of Openness to Experience place greater weight on adaptive performance (e.g., handling changing and uncertain work environments effectively), while raters with high levels of Modesty (a facet of Honesty-Humility) place greater weight on maintaining personal discipline (e.g., lack of deviant or condescending behaviours).

Implications

The finding that individuals vary systematically in their performance dimension weightings adds to a growing body of literature indicating that raters have unique implicit theories regarding performance. As such, there is a real need for organizations to impart a standard theory of performance to their employees.

Originality/Value

This study was the first to implicate the personality dimensions of Honesty-Humility and Openness to Experience in the performance weighting process, and as such, adds to our understanding of the nature of rater implicit theories.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Some have suggested the use of an alternate index of rater agreement, the Average Deviation index, which puts inter-rater agreement into the context of raw scale points (Burke et al. 1999). For a 7-point scale such as the one above, the authors suggest values lower than 1 indicate acceptable inter-rater agreement. The mean average deviation for items retained for the final scales was .85, which is below this level of 1. The average deviation for items retained ranged from .46 to 1.21, indicating that some items were above this threshold, although not substantially. Generally, the retained items demonstrated acceptable levels of inter-rater agreement according to this index.

  2. It should be noted that the slightly larger value obtained for maintaining personal discipline, compared to the other performance dimensions, is likely due to the fact that the average performance rating for behaviours in the maintaining personal discipline scale was lower than the other scale (3.78 vs. 4.07 and above). This, combined with the fact that negative information tends to be more salient, likely has combined to influence the slightly larger value obtained for the maintaining personal discipline dimension.

  3. When an exploratory intercept-as-outcomes model was specified to determine the direct effects of the level-2 personality predictors on the level-1 outcome, conscientiousness was found to negatively relate to overall performance ratings (γ = −.161, p < .05), incremental to the predictability afforded by the five performance dimensions. This implies that raters who are more conscientious tend to be less lenient in evaluating overall job performance. Although the effect of personality on rating leniency (or severity) was not a direct focus of the present research, this result is consistent with a finding reported by Bernadin et al. (2000).

References

  • Ashton, M. C., & Lee, K. (2005). Honesty-Humility, the Big Five, and the Five-Factor model. Journal of Personality, 73, 1321–1353.

    Article  PubMed  Google Scholar 

  • Ashton, M. C., & Lee, K. (2007). Empirical, theoretical, and practical advantages of the HEXACO model of personality structure. Personality and Social Psychology Review, 11, 150–166.

    Article  PubMed  Google Scholar 

  • Ashton, M. C., Lee, K., Perugini, M., Szarota, P., de Vries, R. E., Di Blas, L., et al. (2004). A six-factor structure of personality-descriptive adjectives: Solutions from psycholexical studies in seven languages. Journal of Personality and Social Psychology, 86, 356–366.

    Article  PubMed  Google Scholar 

  • Bernadin, H. J., Cooke, D. K., & Villanova, P. (2000). Conscientiousness and agreeableness as predictors of rating leniency. Journal of Applied Psychology, 85, 232–234.

    Article  Google Scholar 

  • Borman, W. C., Penner, L. A., Allen, T. D., & Motowidlo, S. J. (2001). Personality predictors of citizenship performance. International Journal of Selection and Assessment, 9, 52–69.

    Article  Google Scholar 

  • Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.

    Article  Google Scholar 

  • Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial, organizational psychology (Vol. 1, 2nd ed.). Palo Alto, CA: Consulting Psychologists Press.

    Google Scholar 

  • Campbell, J. P., McCloy, R. A., Oppler, S. H., & Sager, C. E. (1993). A theory of performance. In N. Schmitt & W. C. Borman (Eds.), Personnel selection in organizations (pp. 35–70). San Francisco: Jossey-Bass.

    Google Scholar 

  • Cheng, C., Bond, M. H., & Chan, S. C. (1995). The perception of ideal best friends by Chinese adolescents. International Journal of Psychology, 30, 91–108.

    Article  Google Scholar 

  • Comber, C., Colley, A., Hargreaves, D. J., & Dorn, L. (1997). The effects of age, gender, and computer experience upon computer attitudes. Educational Research, 39, 123–133.

    Google Scholar 

  • Conway, J. M. (1999). Distinguishing contextual performance from task performance for managerial jobs. Journal of Applied Psychology, 84(1), 3–13.

    Article  Google Scholar 

  • Duckitt, J. (2001). A dual-process cognitive-motivational theory of ideology and prejudice. In M. P. Zanna (Ed.), Advances in Experimental Social Psychology, 33, 41–113. San Diego: Academic Press.

  • Hofmann, D. A. (1997). An overview of the logic and rationale of hierarchical linear models. Journal of Management, 23, 723–744.

    Article  Google Scholar 

  • Jost, J., Nosek, B. A., & Gosling, S. D. (2008). Ideology: Its resurgence in social, personality, and political psychology. Perspectives on Psychological Science, 3, 126–136.

    Article  Google Scholar 

  • Kiker, D. S., & Motowidlo, S. J. (1999). Main and interaction effects of task and contextual performance on supervisory reward decisions. Journal of Applied Psychology, 84, 602–609.

    Article  Google Scholar 

  • Lee, K., & Ashton, M. C. (2004). Psychometric properties of the HEXACO Personality Inventory. Multivariate Behavioral Research, 39, 329–358.

    Article  Google Scholar 

  • Lee, K., & Ashton, M. C. (2006). Further assessment of the HEXACO Personality Inventory: Two new facet scales and an observer report form. Psychological Assessment, 18, 182–191.

    Article  PubMed  Google Scholar 

  • Lee, K., Ashton, M. C., & de Vries, R. E. (2005a). Predicting workplace delinquency and integrity with the HEXACO and Five-Factor models of personality structure. Human Performance, 18(2), 179–197.

    Article  Google Scholar 

  • Lee, K., Ashton, M. C., Ogunfowora, B., Bourdage, J. S., & Shin, K. (in press). The personality bases of socio-political attitudes: The role of honesty-humility and openness to experience. Journal of Research in Personality.

  • Lee, K., Ashton, M. C., Pozzebon, J. A., Visser, B. A., Bourdage, J. S., & Ogunfowora, B. (2009). Similarity and assumed similarity in personality reports of well-acquainted persons. Journal of Personality and Social Psychology, 96, 460–472.

    Article  PubMed  Google Scholar 

  • Lee, K., Ogunfowora, B., & Ashton, M. C. (2005b). Personality traits beyond the Big Five: Are they within the HEXACO space? Journal of Personality, 73, 1437–1463.

    Article  PubMed  Google Scholar 

  • LePine, J. A., & Van Dyne, L. (2001). Voice and cooperative behavior as contrasting forms of contextual performance: Evidence of differential relationships with big five personality characteristics and cognitive ability. Journal of Applied Psychology, 86, 326–336.

    Article  PubMed  Google Scholar 

  • Marcus, B., Lee, K., & Ashton, M. C. (2007). Personality dimensions explaining relationships between integrity tests and counterproductive behaviour: Big Five, or one in addition? Personnel Psychology, 60, 1–34.

    Article  Google Scholar 

  • Mount, M. K., Judge, T. A., Scullen, S. E., Sytsma, M. R., & Hezlett, S. A. (1998). Trait, rater, and level effects in 360-degree performance ratings. Personnel Psychology, 51, 557–576.

    Article  Google Scholar 

  • Murphy, K. R., & Cleveland, J. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Murphy, K. R., & DeShon, R. (2000). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873–900.

    Article  Google Scholar 

  • Organ, D. W., & Ryan, K. (1995). A meta-analytic review of attitudinal and dispositional predictors of organizational citizenship behaviour. Personnel Psychology, 48, 775–802.

    Article  Google Scholar 

  • Paunonen, S. V. (2002). Design and construction of the Supernumerary Personality Inventory (Research Bulletin 763). London, Ontario: University of Western Ontario.

    Google Scholar 

  • Pulakos, E. D., Arad, S., Donovan, M. A., & Plamondon, K. E. (2000). Adaptability in the workplace: Development of a taxonomy of adaptive performance. Journal of Applied Psychology, 85(4), 612–624.

    Article  PubMed  Google Scholar 

  • Pulakos, E. D., Schmitt, N., Dorsey, D. W., Arad, S., Hedge, J. W., & Borman, W. C. (2002). Predicting adaptive performance: Further tests of a model of adaptability. Human Performance, 15, 299–323.

    Article  Google Scholar 

  • Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach. Journal of Applied Psychology, 87, 66–80.

    Article  PubMed  Google Scholar 

  • Rotundo, M., & Xie, J. L. (2006). Rater personality and ability in the performance appraisal process. In Annual conference of the Society for Industrial and Organizational Psychologists, Dallas, TX.

  • Rytting, M., Ware, R., & Hopkins, P. (1992). Type and ideal mate: Romantic attraction or type bias? Journal of Psychological Type, 24, 3–12.

    Google Scholar 

  • Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901–912.

    Article  Google Scholar 

  • Schneider, B., Hanges, P. J., Goldstein, H. W., & Braverman, E. P. (1994). Do customer service perceptions generalize? The case of student and chair ratings of faculty effectiveness. Journal of Applied Psychology, 79, 685–690.

    Article  Google Scholar 

  • Steel, P. D. G., Huffcutt, A. I., & Kammeyer-Mueller, J. (2006). From the work one knows the worker: A systematic review of challenges, solutions, and steps to creating synthetic validity. International Journal of Selection and Assessment, 14, 16–36.

    Article  Google Scholar 

  • Uggerslev, K. L., & Sulsky, L. M. (2008). Using Frame-of-Reference training to understand the implications of rater idiosyncrasy for rating accuracy. Journal of Applied Psychology, 93, 711–719.

    Article  PubMed  Google Scholar 

  • Viswesvaran, D., Ones, D. S., & Schmidt, F. L. (1996). Comparative analysis of the reliability of job performance ratings. Journal of Applied Psychology, 81, 557–574.

    Article  Google Scholar 

  • Viswesvaran, D., Schmidt, F. L., & Ones, D. S. (2002). The moderating influence of job performance dimensions on convergence of supervisory and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty. Journal of Applied Psychology, 87, 345–354.

    Article  PubMed  Google Scholar 

  • Wherry, R. J., & Bartlett, C. J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521–555.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, University of Calgary, 2500 University Drive N.W., Calgary, AB, T2N 1N4, Canada

    Babatunde Ogunfowora, Joshua Bourdage & Kibeom Lee

Authors
  1. Babatunde Ogunfowora
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joshua Bourdage
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kibeom Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Babatunde Ogunfowora.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ogunfowora, B., Bourdage, J. & Lee, K. Rater Personality and Performance Dimension Weighting in Making Overall Performance Judgments. J Bus Psychol 25, 465–476 (2010). https://doi.org/10.1007/s10869-009-9144-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-009-9144-y

Keywords

Search

Navigation