Investigating the Construct Validity of Performance Comments: Creation of the Great Eight Narrative Dictionary
- 123 Downloads
Performance narratives are qualitative text descriptions of an employee’s work performance. Despite containing rich information that can be leveraged by practitioners and researchers, few efforts have systematically examined performance narratives. This study investigated whether performance narratives can automatically and reliably be scored into meaningful performance dimensions. Using the Great Eight as a conceptual framework, a custom dictionary was developed and comments were scored via automated text mining. This dictionary, labeled the Great Eight Narrative Dictionary, was then validated against a set of convergent measures to establish construct validity evidence for the derived narrative scores. Inter-rater agreement in linking word phrases to performance dimensions was high, and the derived performance dimensions had acceptable internal consistency. Narrative scores also displayed evidence of construct validity, with an expected pattern of correlations with text scores from an alternative text mining dictionary and with developmental performance ratings made using traditional numerical formats. Collectively, findings support the use of the Great Eight Narrative Dictionary to score performance narratives, and the dictionary is provided openly to facilitate future use.
KeywordsJob performance Performance management Text mining Narrative comments
- Ammons-Stephens, S., Cole, H. J., Riehle, C. F., & Weare, W. H. (2009). Developing core leadership competencies for the library profession. Library Leadership & Management, 23(2), 63–74.Google Scholar
- Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. D. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, pp. 687–732). Palo Alto: Consulting Psychologists Press.Google Scholar
- Ferstl, K. L., & Bruskiewicz, K. T. (2000). Self-other agreement and cognitive reactions to multi-rater feedback. Paper presented at 15th Annual Conference of the Society of Industrial and Organizational Psychology, New Orleans, LA.Google Scholar
- Ignatow, G., & Mihalcea, R. (2017). Text mining: A guidebook for the social sciences. Thousand Oaks: Sage Publications.Google Scholar
- McDowall, A., & Kurtz, R. (2007). Making the most of psychometric profiles-effective integration into the coaching process. International Coaching Psychology Review, 2(3), 299–309.Google Scholar
- Pandey, S., & Pandey, S. K. (2017). Applying natural language processing capabilities in computerized textual analysis to measure organizational culture. Organizational Research Methods. Advanced online publication. http://journals.sagepub.com/doi/abs/10.1177/1094428117745648.
- Pennebaker, J. W., Boyd, R. L., Jordan, K., & Blackburn, K. (2015). The development and psychometric properties of LIWC2015. Austin: University of Texas at Austin.Google Scholar
- R Core Development Team. (2007). R: A language and environment for statistical computing. R Vienna: Foundation for Statistical Computing.Google Scholar
- Spendlove, M. (2007). Competencies for effective leadership in higher education. International Journal of Educational Management, 21(5), 407–417.Google Scholar
- Stemler, S. E. (2015). Content analysis. In R. Scott & S. S. Kosslyn (Eds.), Emerging trends in the social and behavioral sciences: an interdisciplinary, searchable, and linkable resource (pp. 1–14). New York: Wiley.Google Scholar
- Stone, P. J., Dunphy, D. C., & Smith, M. S. (1966). The general inquirer: A computer approach to content analysis. Oxford, England: M.I.T. Press.Google Scholar