Advertisement

Investigating the Construct Validity of Performance Comments: Creation of the Great Eight Narrative Dictionary

  • Andrew B. Speer
  • Michael G. Schwendeman
  • Caitlynn C. Reich
  • Andrew P. Tenbrink
  • Sydney R. Siver
Original Paper
  • 62 Downloads

Abstract

Performance narratives are qualitative text descriptions of an employee’s work performance. Despite containing rich information that can be leveraged by practitioners and researchers, few efforts have systematically examined performance narratives. This study investigated whether performance narratives can automatically and reliably be scored into meaningful performance dimensions. Using the Great Eight as a conceptual framework, a custom dictionary was developed and comments were scored via automated text mining. This dictionary, labeled the Great Eight Narrative Dictionary, was then validated against a set of convergent measures to establish construct validity evidence for the derived narrative scores. Inter-rater agreement in linking word phrases to performance dimensions was high, and the derived performance dimensions had acceptable internal consistency. Narrative scores also displayed evidence of construct validity, with an expected pattern of correlations with text scores from an alternative text mining dictionary and with developmental performance ratings made using traditional numerical formats. Collectively, findings support the use of the Great Eight Narrative Dictionary to score performance narratives, and the dictionary is provided openly to facilitate future use.

Keywords

Job performance Performance management Text mining Narrative comments 

Notes

Supplementary material

10869_2018_9599_MOESM1_ESM.docx (16 kb)
ESM 1 (DOCX 15 kb)

References

  1. Ammons-Stephens, S., Cole, H. J., Riehle, C. F., & Weare, W. H. (2009). Developing core leadership competencies for the library profession. Library Leadership & Management, 23(2), 63–74.Google Scholar
  2. Bartram, D. (2005). The Great Eight competencies: A criterion-centric approach to validation. Journal of Applied Psychology, 90(6), 1185–1203.CrossRefGoogle Scholar
  3. Brutus, S. (2010). Words versus numbers: A theoretical exploration of giving and receiving narrative comments in performance appraisal. Human Resource Management Review, 20, 144–157.CrossRefGoogle Scholar
  4. Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 687–732). Palo Alto: Consulting Psychologists Press.Google Scholar
  5. Condon, D. M., & Revelle, W. (2014). The international cognitive ability resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52–64.CrossRefGoogle Scholar
  6. Costigan, R. D., & Donahue, L. (2009). Developing the great eight competencies with leaderless group discussion. Journal of Management Education, 33(5), 596–616.CrossRefGoogle Scholar
  7. Davenport, E., & El-Sanhury, N. (1991). Phi/phimax: Review and synthesis. Educational and Psychological Measurement, 51, 821–828.CrossRefGoogle Scholar
  8. Ferstl, K. L., & Bruskiewicz, K. T. (2000). Self-other agreement and cognitive reactions to multi-rater feedback. Paper presented at 15th Annual Conference of the Society of Industrial and Organizational Psychology, New Orleans, LA.Google Scholar
  9. Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. C. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96.CrossRefGoogle Scholar
  10. Gorman, C. A., Meriac, J. P., Roch, S. G., Ray, J. L., & Gamble, J. S. (2017). An exploratory study of current performance management practices: human resource executives’ perspectives. International Journal of Selection and Assessment, 25, 193–202.CrossRefGoogle Scholar
  11. Hayes, P. A., & Omodei, M. M. (2011). Managing emergencies: Key competencies for incident management teams. The Australasian Journal of Organisational Psychology, 4, 1–10.CrossRefGoogle Scholar
  12. Hogan, J., & Holland, B. (2003). Using theory to evaluate personality and job-performance relations: A socioanalytic perspective. Journal of Applied Psychology, 88(1), 100–112.CrossRefGoogle Scholar
  13. Ignatow, G., & Mihalcea, R. (2017). Text mining: A guidebook for the social sciences. Thousand Oaks: Sage Publications.Google Scholar
  14. Klendauer, R., Berkovich, M., Gelvin, R., Leimeister, J. M., & Krcmar, H. (2012). Towards a competency model for requirements analysts. Information Systems Journal, 22(6), 475–503.CrossRefGoogle Scholar
  15. Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismihók, G., & Den Hartog, D. N. (2017a). Text classification for organizational researchers: A tutorial. Organizational Research Methods, 21(3), 766–799.CrossRefGoogle Scholar
  16. Kobayashi, V. B., Mol, S. T., Berkers, H. A., Kismihók, G., & Den Hartog, D. N. (2017b). Text mining in organizational research. Organizational Research Methods, 21(3), 733–765.CrossRefGoogle Scholar
  17. Kurz, R., & Bartram, D. (2002). Competency and individual performance: Modeling the world of work. In I. T. Robertson, M. Callinan, & D. Bartram (Eds.), Organizational effectiveness: The role of psychology (pp. 227–255). Chichester: Wiley.CrossRefGoogle Scholar
  18. Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87(1), 72–107.CrossRefGoogle Scholar
  19. Liu, B. (2012). Sentiment analysis and opinion mining. San Rafael: Morgan & Claypool Publishers.CrossRefGoogle Scholar
  20. McDowall, A., & Kurtz, R. (2007). Making the most of psychometric profiles-effective integration into the coaching process. International Coaching Psychology Review, 2(3), 299–309.Google Scholar
  21. O’Neill, T. A., Goffin, R. D., & Tett, R. P. (2009). Content validation is fundamental for optimizing the criterion validity of personality tests. Industrial and Organizational Psychology, 2(4), 509–513.CrossRefGoogle Scholar
  22. Pandey, S., & Pandey, S. K. (2017). Applying natural language processing capabilities in computerized textual analysis to measure organizational culture. Organizational Research Methods. Advanced online publication. http://journals.sagepub.com/doi/abs/10.1177/1094428117745648.
  23. Pang, B., & Lee, L. (2008). Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2, 1–135.CrossRefGoogle Scholar
  24. Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. Austin: University of Texas at Austin.Google Scholar
  25. Pulakos, E. D., Hanson, R. M., Arad, S., & Moye, N. (2015). Performance management can be fixed: An on-the-job experiential learning approach for complex behavior change. Industrial and Organizational Psychology, 8(1), 51–76.CrossRefGoogle Scholar
  26. Pulakos, E. D., & O’Leary, R. S. (2011). Why is performance management broken? Industrial and Organizational Psychology: Perspectives on Science and Practice, 4, 146–164.CrossRefGoogle Scholar
  27. R Core Development Team. (2007). R: A language and environment for statistical computing. R Vienna: Foundation for Statistical Computing.Google Scholar
  28. Rojon, C., McDowall, A., & Saunders, M. N. (2015). The relationships between traditional selection assessments and workplace performance criteria specificity: A comparative meta-analysis. Human Performance, 28(1), 1–25.CrossRefGoogle Scholar
  29. Scullen, S. E., Mount, M. K., & Judge, T. A. (2003). Evidence of the construct validity of developmental ratings of managerial performance. Journal of Applied Psychology, 88, 50–66.CrossRefGoogle Scholar
  30. Short, J. C., Broberg, J. C., Cogliser, C. C., & Brigham, K. C. (2010). Construct validation using computer-aided text analysis (CATA): An illustration using entrepreneurial orientation. Organizational Research Methods, 13, 320–347.CrossRefGoogle Scholar
  31. Sliter, K. A. (2015). Assessing 21st century skills: Competency modeling to the rescue. Industrial and Organizational Psychology, 8(2), 284–289.CrossRefGoogle Scholar
  32. Speer, A. B. (2018). Quantifying with words: An investigation of the validity of narrative-derived performance scores. Personnel Psychology. Advanced online publication, 71, 299–333.  https://doi.org/10.1111/peps.12263.CrossRefGoogle Scholar
  33. Spendlove, M. (2007). Competencies for effective leadership in higher education. International Journal of Educational Management, 21(5), 407–417.Google Scholar
  34. Stemler, S. E. (2015). Content analysis. In R. Scott & S. S. Kosslyn (Eds.), Emerging trends in the social and behavioral sciences: an interdisciplinary, searchable, and linkable resource (pp. 1–14). New York: Wiley.Google Scholar
  35. Stone, P. J., Dunphy, D. C., & Smith, M. S. (1966). The general inquirer: A computer approach to content analysis. Oxford, England: M.I.T. Press.Google Scholar
  36. Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24–54.CrossRefGoogle Scholar
  37. Tett, R. P., Guterman, H. A., Bleier, A., & Murphy, P. J. (2000). Development and content validation of a “hyperdimensional” taxonomy of managerial competence. Human Performance, 13, 205–251.CrossRefGoogle Scholar
  38. Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of job performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90(1), 108–131.CrossRefGoogle Scholar
  39. Wu, C.-H., & Wang, Y. (2011). Understanding proactive leadership. In W. H. Mobley, M. Li, & Y. Wang (Eds.), Advances in global leadership (Vol. 6, pp. 299–314). Bingley: Emerald Group.CrossRefGoogle Scholar
  40. Zou, H., & Hastie, T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B, 67, 301–320.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of PsychologyWayne State UniversityDetroitUSA

Personalised recommendations