Journal of Business and Psychology

, Volume 25, Issue 4, pp 663–672 | Cite as

BARS and Those Mysterious, Missing Middle Anchors

  • Neil M. A. Hauenstein
  • Reagan D. Brown
  • Andrea L. Sinclair
Article
First Online:

Abstract

Purpose

A contributing reason for the common problem of missing middle anchors on behaviorally anchored rating scales (BARS) is the standard deviation (SD) criterion used in scaling phase. An alternative BARS scaling process is proposed based on the a wg(1) index of interrater agreement.

Design/Methodology/Approach

Algebraic principles are used to explicate that the SD criterion is analogous to using a r wg(1) interrater agreement statistic with the assumption of a uniform null distribution, and this reliance on r wg(1) decreases the likelihood of anchoring behaviors in the mid-range of the rating scale. Archival data from a law enforcement agency were used to compare the success and failure of anchoring a BARS using a SD criterion versus an a wg(1) criterion.

Findings

The a wg(1) criterion was successful at anchoring the full range of the rating scale, but only if the cut-off for anchoring behaviors required a “weak” level of interrater agreement.

Implications

The most surprising finding was that the traditional 1.5 SD criterion on a 9-point rating scale is not a particularly stringent agreement requirement for anchoring behaviors. Although we demonstrated the advantages of using a wg(1) to anchor BARS, an equally important conclusion is that incumbents need to be better trained prior to scaling behaviors.

Originality/Value

We provide a theoretically defensible approach for anchoring BARS that ameliorates the missing middle anchor problem. Further, the utility of a wg(1) in the BARS context is yet another example of the limitation of r wg(1) when assuming a uniform null distribution.

Keywords

awg(1) rwg(1) Interrater agreement Behaviorally anchored rating scales Performance appraisal 
This is a preview of subscription content, log in to check access.

References

  1. Barnes-Farrell, J. L., & Lynch, A. M. (2003). Performance appraisal and feedback programs. In J. E. Edwards, J. C. Scott, & N. S. Raju (Eds.), The Human Resources Program-Evaluation Handbook (pp. 155–176). Thousand Oaks, CA: Sage Publications.Google Scholar
  2. Bernardin, H. J., & Smith, P. C. (1981). A clarification of some issues regarding the development and use of behaviorally anchored ratings scales (BARS). Journal of Applied Psychology, 66, 458–463.CrossRefGoogle Scholar
  3. Bownas, D. A., & Bernardin, H. J. (1988). Critical incident technique. In S. Gael (Ed.), The job analysis handbook for business, industry, and government (pp. 1120–1137). New York, NY: Wiley.Google Scholar
  4. Brown, R. D., & Hauenstein, N. M. A. (2005). Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods, 8, 1–20.CrossRefGoogle Scholar
  5. Burke, M. J., & Dunlop, W. P. (2002). Estimating interrater agreement with the average deviation index: A user’s guide. Organizational Research Methods, 5, 159–172.CrossRefGoogle Scholar
  6. Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49–68.CrossRefGoogle Scholar
  7. Cardy, R. L. (1998). Performance appraisal in a quality context: A new look at an old problem. In J. Smither (Ed.), Performance appraisal: State of the art in practice (pp. 132–162). Josey-Bass: San Francisco.Google Scholar
  8. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.CrossRefGoogle Scholar
  9. Conte, J. M., Landy, F. J., & Mathieu, J. E. (1995). Time urgency: Conceptual and construct development. Journal of Applied Psychology, 80, 178–185.CrossRefGoogle Scholar
  10. DeNisi, A. S. (1996). Cognitive approach to performance appraisal: A program of research. New York: Routledge.Google Scholar
  11. Engelbrecht, A. S., & Fischer, H. (1995). The managerial performance implications of a developmental assessment center process. Human Relations, 48, 387–404.CrossRefGoogle Scholar
  12. Finn, R. H. (1970). A note on estimating the reliability of categorical data. Educational and Psychological Measurement, 30, 71–76.CrossRefGoogle Scholar
  13. George, J. M. (1990). Personality, affect and behavior in groups. Journal of Applied Psychology, 75, 107–116.CrossRefGoogle Scholar
  14. Grote, D. (1996). The complete guide to performance appraisal. New York: American Management Association.Google Scholar
  15. Harari, O., & Zedeck, S. (1973). Development of behaviorally anchored scales for the evaluation of faculty teaching. Journal of Applied Psychology, 58, 261–265.CrossRefGoogle Scholar
  16. Hauenstein, N. M. A., & Foti, R. J. (1989). From laboratory to practice: Neglected issues in implementing frame-of-reference rater training. Personnel Psychology, 42, 359–378.CrossRefGoogle Scholar
  17. James, L. R., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. Journal of Applied Psychology, 69, 85–98.CrossRefGoogle Scholar
  18. Judge, T. A., & Bono, J. E. (2000). Five factor model of personality and transformational leadership. Journal of Applied Psychology, 85, 751–765.CrossRefPubMedGoogle Scholar
  19. Kozlowski, S. W., & Hattrup, K. (1992). A disagreement about within-group agreement: Disentangling issues of consistency versus consensus. Journal of Applied Psychology, 77, 161–167.CrossRefGoogle Scholar
  20. Landy, F. J., & Guion, R. M. (1970). Development of scales for the measurement of work motivation. Organizational Behavior and Human Decision Processes, 5, 93–103.Google Scholar
  21. Latham, G. P., Saari, L. M., Pursell, E. D., & Campion, M. A. (1980). The situational interview. Journal of Applied Psychology, 65, 422–427.CrossRefGoogle Scholar
  22. London, M., Mone, E. M., & Scott, J. C. (2004). Performance management and assessment: Methods for improved rater accuracy and employee goal setting. Human Resource Management, 43, 319–336.CrossRefGoogle Scholar
  23. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
  24. Malos, S. B. (1998). Current legal issues in performance appraisal. In J. Smither (Ed.), Performance appraisal: State of the art in practice (pp. 49–94). San Francisco: Josey-Bass.Google Scholar
  25. Martin, D. D., & Bartol, K. M. (1998). Performance appraisal: Maintaining system effectiveness. Public Personnel Management, 27, 223–230.Google Scholar
  26. Maurer, S. D. (2002). A practitioner-based analysis of interviewer job expertise and scale format as contextual factors in situational interviews. Personnel Psychology, 55, 307–328.CrossRefGoogle Scholar
  27. Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal: Social, organizational, and goal-based perspectives. Thousand Oaks, CA: Sage Publications.Google Scholar
  28. Murphy, K. R., & De Shon, R. (2000a). Interrater correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873–900.CrossRefGoogle Scholar
  29. Murphy, K. R., & De Shon, R. (2000b). Progress in psychometrics: Can industrial and organizational psychology catch up? Personnel Psychology, 53, 913–924.CrossRefGoogle Scholar
  30. Ramus, C. A., & Steger, U. (2000). The roles of supervisory support behaviors and environmental policy in employee “ecoinitiatives” at leading-edge European companies. Academy of Management Journal, 43, 605–626.CrossRefGoogle Scholar
  31. Schleicher, D. J., Day, D. V., Mayes, B. T., & Riggio, R. E. (2002). A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers. Journal of Applied Psychology, 87, 735–746.CrossRefPubMedGoogle Scholar
  32. Schmidt, F. L., Viswesvaran, C., & Ones, D. S. (2000). Reliability is not validity and validity is not reliability. Personnel Psychology, 53, 901–912.CrossRefGoogle Scholar
  33. Smith, P. C., & Kendall, L. M. (1963). Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales. Journal of Applied Psychology, 47, 149–255.CrossRefGoogle Scholar
  34. Tziner, A., Joanis, C., & Murphy, K. (2000). A comparison of three methods of performance appraisal with regard to goal properties, goal perception, and ratee satisfaction. Group & Organization Management, 25, 175–190.CrossRefGoogle Scholar
  35. Woods, R. H., Sciarini, M., & Breiter, D. (1998). Performance appraisals in hotels. Cornell Hotel and Restaurant Administration Quarterly, 39, 25–29.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2010

Authors and Affiliations

  • Neil M. A. Hauenstein
    • 1
  • Reagan D. Brown
    • 2
  • Andrea L. Sinclair
    • 3
  1. 1.Virginia Polytechnic Institute and State UniversityBlacksburgUSA
  2. 2.Western Kentucky UniversityBowling GreenUSA
  3. 3.Human Resources Research OrganizationLouisvilleUSA

Personalised recommendations