Abstract
Personality measures are popular and useful in employment selection and academic contexts; however, concerns have been voiced regarding the strength of their association with desirable criteria. Contextualization (i.e., modifying measures to reflect the desired frame of reference, like work or school) has emerged as a promising option. Research has demonstrated that contextualizing personality measures increases predictive validity and enhances participants’ perceptions of the assessments. However, few studies have compared contextualization methods to one another and, to date, only one study has compared the two most common forms of contextualization (i.e., instruction and tag contextualization), returning inconsistent findings. In a within-person, multi-wave study using a working sample (N = 399), we compared the relative efficacy of personality measures that are contextualized through manipulating the instructions and those contextualized through the addition of contextual item tags. We specifically contextualized the big five personality factors in order to predict work-related outcomes (i.e., job satisfaction, perpetrated incivility, job performance, creative job performance, and emotional exhaustion). Our study supports the use of tag-level contextualization and provides guidance on how to best implement contextual tags. Best practices, implications, and future research directions are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
In accordance with the research participants’ informed consent and, as approved by the relevant Institutional Review Board, data associated with this manuscript are not publicly available.
References
Aguinis, H., Villamor, I., & Ramani, R. S. (2021). MTurk research: Review and recommendations. Journal of Management, 47(4), 823–837. https://doi.org/10.1177/0149206320969787
Ajzen, I. (2005). Laws of human behavior: Symmetry, compatibility, and attitude-behavior correspondence. In A. Beauducel, B. Biehl, M. Bosniak, W. Conrad, G. Schonberger, & D. Wagener (Eds.), Multivariate research strategies (pp. 3–19). Shaker.
Ansell, C., Boin, A., & Keller, A. (2010). Managing transboundary crises: Identifying the building blocks of an effective response system. Journal of Contingencies and Crisis Management, 18(4), 195–207.
Baldasaro, R. E., Shanahan, M. J., & Bauer, D. J. (2013). Psychometric properties of the mini-IPIP in a large, nationally representative sample of young adults. Journal of Personality Assessment, 95(1), 74–84.
Barrick, M. R., & Mount, M. K. (1991). The big five personality dimensions and job performance: A meta-analysis. Personnel Psychology, 44(1), 1–26. https://doi.org/10.1111/j.1744-6570.1991.tb00688.x
Bing, M. N., Whanger, J. C., Davison, H. K., & VanHook, J. B. (2004). Incremental validity of the frame-of-reference effect in personality scale scores: A replication and extension. Journal of Applied Psychology, 89(1), 150–157. https://doi.org/10.1037/0021-9010.89.1.150
Bing, M. N., Davison, H. K., & Smothers, J. (2014). Item-level frame-of-reference effects in personality testing: An investigation of incremental validity in an organizational setting. International Journal of Selection and Assessment, 22(2), 165–178. https://doi.org/10.1111/ijsa.12066
Bowling, N. A., & Burns, G. N. (2010). A comparison of work-specific and general personality measures as predictors of work and non-work criteria. Personality and Individual Differences, 49(2), 95–101. https://doi.org/10.1016/j.paid.2010.03.009
Bowling, N. A., & Hammond, G. D. (2008). A meta-analytic examination of the construct validity of the Michigan Organizational Assessment Questionnaire Job Satisfaction Subscale. Journal of Vocational Behavior, 73(1), 63–77. https://doi.org/10.1016/j.jvb.2008.01.004
Cammann, C., Fichman, M., Jenkins, D., & Klesh, J. (1983). Assessing the attitudes and perceptions of organizational members. In S. E. Seashore, E. E. Lawler, P. H. Mirvis, & C. Cammann (Eds.), Assessing organizational change: A guide to methods, measures, and practices (pp. 71–138). Wiley.
Caskie, G. I., Sutton, M. C., & Eckhardt, A. G. (2014). Accuracy of self-reported college GPA: Gender-moderated differences by achievement level and academic self-efficacy. Journal of College Student Development, 55(4), 385–390. https://doi.org/10.1353/csd.2014.0038
Cassady, J. C. (2000). Self-reported GPA and SAT: A methodological note. Practical Assessment, Research, and Evaluation, 7(12), 1–4. https://doi.org/10.7275/5hym-y754
Charters, E. (2003). The use of think-aloud methods in qualitative research. An introduction to think-aloud methods. Brock Education Journal, 12(2), 68–82. https://doi.org/10.26522/brocked.v12i2.38
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2013). Applied multiple regression/correlation analysis for the behavioral sciences. Routledge. https://doi.org/10.4324/9780203774441
Cortina, L. M., Magley, V. J., Williams, J. H., & Langhout, R. D. (2001). Incivility in the workplace: Incidence and impact. Journal of Occupational Health Psychology, 6(1), 64–80. https://doi.org/10.1037/1076-8998.6.1.64
Donaldson, S. I., & Grant-Vallone, E. J. (2002). Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, 17, 245–260.
Donnellan, M. B., Oswald, F. L., Baird, B. M., & Lucas, R. E. (2006). The mini-IPIP scales: Tiny-yet-effective measures of the Big Five factors of personality. Psychological Assessment, 18(2), 192–203. https://doi.org/10.1037/1040-3590.18.2.192
Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006). A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91(1), 40–57. https://doi.org/10.1037/0021-9010.91.1.40
Eid, M., Gollwitzer, M., & Schmitt, M. (2011). Statistik und Forschungsmethoden Lehrbuch. Beltz.
Furnham, A., Monsen, J., & Ahmetoglu, G. (2009). Typical intellectual engagement, big five personality traits, approaches to learning and cognitive ability predictors of academic performance. British Journal of Educational Psychology, 79(4), 769–782. https://doi.org/10.1348/978185409X412147
Goldberg, L. R. (1992). The development of markers for the big-five factor structure. Psychological Assessment, 4(1), 26–42. https://doi.org/10.1037/1040-3590.4.1.26
Gray, E. K., & Watson, D. (2002). General and specific traits of personality and their relation to sleep and academic performance. Journal of Personality, 70(2), 177–206. https://doi.org/10.1111/1467-6494.05002
Guion, R. M., & Gottier, R. F. (1965). Validity of personality measures in personnel selection. Personnel Psychology, 18(2), 135–164. https://doi.org/10.1111/j.1744-6570.1965.tb00273.x
Hausknecht, J. P., Day, D. V., & Thomas, S. C. (2004). Applicant reactions to selection procedures: An updated model and meta-analysis. Personnel Psychology, 57(3), 639–683. https://doi.org/10.1111/j.1744-6570.2004.00003.x
Heggestad, E. D., Scheaf, D. J., Banks, G. C., Monroe Hausfeld, M., Tonidandel, S., & Williams, E. B. (2019). Scale adaptation in organizational science research: A review and best-practice recommendations. Journal of Management, 45(6), 2596–2627. https://doi.org/10.1177/0149206319850280
Heller, D., Ferris, D. L., Brown, D., & Watson, D. (2009). The influence of work personality on job satisfaction: Incremental validity and mediation effects. Journal of Personality, 77(4), 1051–1084.
Holbrook, A. L., Krosnick, J. A., Moore, D., & Tourangeau, R. (2007). Response order effects in dichotomous categorical questions presented orally: The impact of question and respondent attributes. The Public Opinion Quarterly, 71(3), 325–348. https://doi.org/10.1093/poq/nfm024
Holtrop, D., Born, M. P., de Vries, A., & de Vries, R. E. (2014). A matter of context: A comparison of two types of contextualized personality measures. Personality and Individual Differences, 68, 234–240. https://doi.org/10.1016/j.paid.2014.04.029
Holtz, B. C., Ployhart, R. E., & Dominguez, A. (2005). Testing the rules of justice: The effects of frame-of-reference and pre-test validity information on personality test responses and test perceptions. International Journal of Selection & Assessment, 13(1), 75–86. https://doi.org/10.1111/j.0965-075X.2005.00301
Hunthausen, J. M., Truxillo, D. M., Bauer, T. N., & Hammer, L. B. (2003). A field study of frame-of-reference effects on personality test validity. Journal of Applied Psychology, 88(3), 545–551. https://doi.org/10.1037/0021-9010.88.3.545
Johnson, R. E., Rosen, C. C., & Djurdjevic, E. (2011). Assessing the impact of common method variance on higher order multidimensional constructs. Journal of Applied Psychology, 96(4), 744–761. https://doi.org/10.1037/a0021504
Judge, T. A., Heller, D., & Mount, M. K. (2002). Five-factor model of personality and job satisfaction: A meta-analysis. Journal of Applied Psychology, 87(3), 530–541. https://doi.org/10.1037/0021-9010.87.3.530
Kammeyer-Mueller, J. D., Simon, L. S., & Judge, T. A. (2016). A head start or a step behind? Understanding how dispositional and motivational resources influence emotional exhaustion. Journal of Management, 42(3), 561–581. https://doi.org/10.1177/0149206313484518
Kelley, M. R., Neath, I., & Surprenant, A. M. (2013). Three more semantic serial position functions and a SIMPLE explanation. Memory & Cognition, 41(4), 600–610. https://doi.org/10.3758/s13421-012-0286-1
Krosnick, J. A., & Alwin, D. F. (1987). An evaluation of a cognitive theory of response-order effects in survey measurement. Public Opinion Quarterly, 51(2), 201–219. https://doi.org/10.1086/269029
Lievens, F., De Corte, W., & Schollaert, E. (2008). A closer look at the frame-of-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93(2), 268–279. https://doi.org/10.1037/0021-9010.93.2.268
Maslach, C., & Jackson, S. E. (1981). The measurement of experienced burnout. Journal of Organizational Behavior, 2(2), 99–113. https://doi.org/10.1002/job.4030020205
McAbee, S. T., & Oswald, F. L. (2013). The criterion-related validity of personality measures for predicting GPA: A meta-analytic validity competition. Psychological Assessment, 25(2), 532–544. https://doi.org/10.1037/a0031748
Mischel, W. (1973). Toward a cognitive social learning reconceptualization of personality. Psychological Review, 80, 252–283. https://doi.org/10.1037/h0035002
Mischel, W., & Shoda, Y. (1995). A cognitive-affective system theory of personality: Reconceptualizing situations, dispositions, dynamics, and invariance in personality structure. Psychological Review, 102(2), 246–268. https://doi.org/10.1037/0033-295X.102.2.246
Ng, M. A., Naranjo, A., Schlotzhauer, A. E., Shoss, M. K., Kartvelishvili, N., Bartek, M., … & Silva, C. (2021). Has the COVID-19 pandemic accelerated the future of work or changed its course? Implications for research and practice. International Journal of Environmental Research and Public Health, 18(19), 10199.
Noftle, E. E., & Robins, R. W. (2007). Personality predictors of academic outcomes: Big five correlates of GPA and SAT scores. Journal of Personality and Social Psychology, 93(1), 116–130. https://doi.org/10.1037/0022-3514.93.1.116
Oldham, G. R., & Cummings, A. (1996). Employee creativity: Personal and contextual factors at work. Academy of Management Journal, 39(3), 607–634. https://doi.org/10.2307/256657
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009
Pace, V. L., & Brannick, M. T. (2010). Improving prediction of work performance through frame-of-reference consistency: Empirical evidence using openness to experience. International Journal of Selection & Assessment, 18(2), 230–235. https://doi.org/10.1111/j.1468-2389.2010.00506.x
Pathki, C. S., Kluemper, D. H., Meuser, J. D., & McLarty, B. D. (2022). The org-B5: Development of a short work frame-of-reference measure of the big five. Journal of Management, 48(5), 1299–1337. https://doi.org/10.1177/01492063211002627
Pearce, J. L., & Porter, L. W. (1986). Employee responses to formal performance appraisal feedback. Journal of Applied Psychology, 71(2), 211–218. https://doi.org/10.1037/0021-9010.71.2.211
Ployhart, R. E., Ziegert, J. C., & McFarland, L. A. (2003). Understanding racial differences on cognitive ability tests in selection contexts: An integration of stereotype threat and applicant reactions research. Human Performance, 16(3), 231–259. https://doi.org/10.1207/S15327043HUP1603_4
Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879
Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63, 539–569.
Reddock, C. M., Biderman, M. D., & Nguyen, N. T. (2011). The relationship of reliability and validity of personality tests to frame-of-reference instructions and within-person inconsistency. International Journal of Selection and Assessment, 19(2), 119–131. https://doi.org/10.1111/j.1468-2389.2011.00540.x
Robie, C., & Risavy, S. D. (2016). A comparison of frame-of-reference and frequency-based personality measurement. Personality and Individual Differences, 92, 16–21. https://doi.org/10.1016/j.paid.2015.12.005
Robie, C., Schmit, M. J., Ryan, A. M., & Zickar, M. J. (2000). Effects of item context specificity on the measurement equivalence of a personality inventory. Organizational Research Methods, 3(4), 348–365. https://doi.org/10.1177/109442810034003
Robie, C., Born, M. P., & Schmit, M. J. (2001). Personal and situational determinants of personality responses: A partial reanalysis and reinterpretation of the Schmit et al. (1995) data. Journal of Business and Psychology, 16(1), 101–117. https://doi.org/10.1023/A:1007843906550
Robie, C., Risavy, S. D., Holtrop, D., & Born, M. P. (2017). Fully contextualized, frequency-based personality measurement: A replication and extension. Journal of Research in Personality, 70, 56–65. https://doi.org/10.1016/j.jrp.2017.05.005
Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2021). Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range. Journal of Applied Psychology, 107(11), 2040–2068. https://doi.org/10.1037/apl0000994
Schilpzand, M. C., Herold, D. M., & Shalley, C. E. (2011). Members’ openness to experience and teams’ creative performance. Small Group Research, 42(1), 55–76. https://doi.org/10.1177/1046496410377509
Schmit, M. J., Ryan, A. M., Stierwalt, S. L., & Powell, A. B. (1995). Frame-of-reference effects on personality scale scores and criterion-related validity. Journal of Applied Psychology, 80(5), 607–620. https://doi.org/10.1037/0021-9010.80.5.607
Schulze, J., West, S. G., Freudenstein, J. P., Schäpers, P., Mussel, P., Eid, M., & Krumm, S. (2021). Hidden framings and hidden asymmetries in the measurement of personality––A combined lens-model and frame-of-reference perspective. Journal of Personality, 89(2), 357–375. https://doi.org/10.1111/jopy.12586
Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65(3), 445–494. https://doi.org/10.1111/j.1744-6570.2012.01250.x
Shamon, H., & Berning, C. (2020). Attention check items and instructions in online surveys with incentivized and non-incentivized samples: Boon or bane for data quality? Survey Research Methods, 14(1), 55–77. https://doi.org/10.2139/ssrn.3549789
Sosnowska, J., De Fruyt, F., & Hofmans, J. (2019). Relating neuroticism to emotional exhaustion: A dynamic approach to personality. Frontiers in Psychology, 10, 2264. https://doi.org/10.3389/fpsyg.2019.02264
Spector, P. E., & Brannick, M. T. (2010). Common method issues: An introduction to the feature topic in organizational research methods. Organizational Research Methods, 13(3), 403–406. https://doi.org/10.1177/1094428110366303
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87(2), 245–251. https://doi.org/10.1037/0033-2909.87.2.245
Swift, V., & Peterson, J. B. (2019). Contextualization as a means to improve the predictive validity of personality models. Personality and Individual Differences, 144, 153–163. https://doi.org/10.1016/j.paid.2019.03.007
Taylor, S. G., & Kluemper, D. H. (2012). Linking perceptions of role stress and incivility to workplace aggression: The moderating role of personality. Journal of Occupational Health Psychology, 17(3), 316–329. https://doi.org/10.1037/a0028211
Tett, R. P., & Burnett, D. D. (2003). A personality trait-based interactionist model of job performance. Journal of Applied Psychology, 88(3), 500–517. https://doi.org/10.1037/0021-9010.88.3.500
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review. Personnel Psychology, 44(4), 703–742. https://doi.org/10.1111/j.1744-6570.1991.tb00696.x
Tonidandel, S., & LeBreton, J. M. (2015). RWA web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology, 30(2), 207–216. https://doi.org/10.1007/s10869-014-9351-z
Tonidandel, S., LeBreton, J. M., & Johnson, J. W. (2009). Determining the statistical significance of relative weights. Psychological Methods, 14(4), 387–399. https://doi.org/10.1037/a0017735
Welbourne, J. L., Miranda, G., & Gangadharan, A. (2020). Effects of employee personality on the relationships between experienced incivility, emotional exhaustion, and perpetrated incivility. International Journal of Stress Management, 27(4), 335–345. https://doi.org/10.1037/str0000160
Williams, J. R., & Levy, P. E. (1992). The effects of perceived system knowledge on the agreement between self-ratings and supervisor ratings. Personnel Psychology, 45(4), 835–847. https://doi.org/10.1111/j.1744-6570.1992.tb00970.x
Acknowledgements
The authors wish to thank Dr. Nathan Bowling for his helpful comments on an earlier version of this article.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors declare no competing interests.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional supplementary materials may be found here by searching on article title https://osf.io/collections/jbp/discover.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Schlotzhauer, A.E., Ng, M.A. & Su, S. How to Frame the Frame of Reference: A Comparison of Contextualization Methods. J Bus Psychol (2024). https://doi.org/10.1007/s10869-024-09953-8
Accepted:
Published:
DOI: https://doi.org/10.1007/s10869-024-09953-8