Advertisement

Situational Judgement Tests (SJTs)

  • Fiona PattersonEmail author
  • Rachel Driver
Chapter

Abstract

In this chapter, we provide an introduction into Situational Judgement Tests (SJTs) and their use for assessment and selection in healthcare settings. The chapter begins by defining SJTs as a measurement methodology and follows by outlining the research evidence base exploring reliability, validity, and other considerations for using SJTs to assess and select candidates into high-stakes healthcare roles. The chapter closes with an overview of key issues for consideration in the future research, development, and applications for SJTs, including understanding group differences, technological advances, and uses in training and development.

References

  1. Anderson, N., Lievens, F., van Dam, K., & Born, M. (2006). A construct-driven investigation of gender differences in a leadership-role assessment center. Journal of Applied Psychology, 91(3), 555–566.  https://doi.org/10.1037/0021-9010.91.3.555.CrossRefPubMedGoogle Scholar
  2. Banki, S., & Latham, G. P. (2010). The criterion-related validities and perceived fairness of the situational interview and the situational judgment test in an Iranian organization. Applied Psychology: An International Review, 59(1), 124–142.CrossRefGoogle Scholar
  3. British Medical Association. (2009). Equality and diversity in UK medical schools. London: BMA.Google Scholar
  4. Buyse, T., & Lievens, F. (2011). Situational judgment tests as a new tool for dental student selection. Journal of Dental Education, 75(6), 743–749. Google Scholar
  5. Campion, M. C., Ployhart, R. E., & MacKenzie, W. I., Jr. (2014). The state of research on situational judgment tests: A content analysis and directions for future research. Human Performance, 27(4), 283–310.CrossRefGoogle Scholar
  6. Catano, V., Brochu, A., & Lamerson, C. (2012). Assessing the reliability of situational judgment tests used in high-stakes situations. International Journal of Selection and Assessment, 20(3), 333–346.CrossRefGoogle Scholar
  7. Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of assessment in situational judgment tests: Subgroup differences in test performance and face validity perceptions. Journal of Applied Psychology, 82(1), 143–159.CrossRefGoogle Scholar
  8. Chan, D., & Schmitt, N. (2002). Situational judgment and job performance. Human Performance, 15(3), 233–254.CrossRefGoogle Scholar
  9. Chan, D., Schmitt, N., Sacco, J. M., & DeShon, R. P. (1998). Understanding pretest and posttest reactions to cognitive ability and personality tests. Journal of Applied Psychology, 83(3), 471–485.CrossRefGoogle Scholar
  10. Chen, L., Fan, J., Zheng, L., & Hack, E. (2016). Clearly defined constructs and specific situations are the currency of SJTs. Industrial and Organizational Psychology, 9(1), 34–38.CrossRefGoogle Scholar
  11. Christian, M. S., Edwards, B. D., & Bradley, J. C. (2010). Situational judgment tests: Constructs assessed and a meta-analysis of their criterion-related validities. Personnel Psychology, 63, 83–117.  https://doi.org/10.1111/j.1744-6570.2009.01163.x.CrossRefGoogle Scholar
  12. Cleland, J. A., Nicholson, S., Kelly, N., & Moffat, M. (2015). Taking context seriously: Explaining widening accesspolicy enactments in UK medical schools. Medical Education, 25–35. Retrieved from http://www.gmc-uk.org/Identifying_best_practice_in_the_selection_of_medical_students.pdf_51119804.pdf.CrossRefGoogle Scholar
  13. Clevenger, J., Pereira, G. M., Wiechmann, D., Schmitt, N., & Harvey, V. S. (2001). Incremental validation of situational judgment tests. The Journal of Applied Psychology, 86(3), 410–417.  https://doi.org/10.1037/0021-9010.86.3.410.CrossRefPubMedGoogle Scholar
  14. Corstjens, J., Lievens, F., & Krumm, S. (2017). Situational judgement tests for selection. In The Wiley Blackwell handbook of the psychology of recruitment, selection and employee retention, 1 (pp. 226–247). Wiley. Google Scholar
  15. Cousans, F., Patterson, F., Edwards, H., McLaughlan, J., & Good, D. (2017). Evaluating the complementary roles of a situational judgement test and academic assessment for entry into clinical practice. Advances in Health Sciences Education Special Issue, 22(2), 401–413. CrossRefGoogle Scholar
  16. Dore, K. L., Reiter, H. I., Kreuger, S., & Norman, G. R. (2017a). CASPer, an online pre-interview screen for personal/professional characteristics: Prediction of national licensure scores. Advances in Health Sciences Education, 22(2), 327–336.  https://doi.org/10.1007/s10459-016-9739-9.CrossRefPubMedPubMedCentralGoogle Scholar
  17. Dore, K. L., Reiter, H. I., Kreuger, S., & Norman, G. R. (2017b). Correction to: CASPer, an online pre-interview screen for personal/professional characteristics: Prediction of national licensure scores. Advances in Health Sciences Education, 22(5), 1321–1322.  https://doi.org/10.1007/s10459-017-9798-6.CrossRefPubMedPubMedCentralGoogle Scholar
  18. Escudier, M., Flaxman, C., Cousans, F., Woolford, M., & Patterson, F. (2015). Pilot results from a new assessment of professionalism. Proceedings of the AMEE Conference 2015; Glas. UK.Google Scholar
  19. Ferguson, E., Semper, H., Yates, J., Fitzgerald, J., Skatova, A., & James, D. (2014). The ‘dark side’ and ‘bright side’ of personality: When too much conscientiousness and too little anxiety are detrimental with respect to the acquisition of medical knowledge and skill. PLoS ONE, 9(2), 1–11.CrossRefGoogle Scholar
  20. Gaugler, B. B., Rosenthal, D. B., Thornton, G. C., & Bentson, C. (1987). Meta-analysis of assessment center validity. Journal of Applied Psychology, 72(3), 493–511.CrossRefGoogle Scholar
  21. Golubovich, J., Seybert, J., Martin-raugh, M., Naemi, B., Vega, R. P., & Roberts, R. D. (2017). Assessing perceptions of interpersonal behavior with a video-based situational judgment test. International Journal of Testing, 17(3), 191–209.  https://doi.org/10.1080/15305058.2016.1194275.CrossRefGoogle Scholar
  22. Graziano, W. G., Jensen-campbell, L. A., & Hair, E. C. (1996). Perceiving interpersonal conflict and reacting to it: The case for agreeableness. Journal of Personality and Social Psychology, 70(4), 820–835.CrossRefGoogle Scholar
  23. Guenole, N., Chernyshenko, O. S., & Weekly, J. (2017). On designing construct driven situational judgment tests: Some preliminary recommendations. International Journal of Testing, 17(3), 234–252.  https://doi.org/10.1080/15305058.2017.1297817.CrossRefGoogle Scholar
  24. Kanning, U. P., Grewe, K., Hollenberg, S., & Hadouch, M. (2006). From the subject’s point of view: Reactions to different types of situational judgment items. European Journal of Psychological Assessment, 22, 168–176.CrossRefGoogle Scholar
  25. Kasten, N., & Freund, P. A. (2016). A meta-analytical multilevel reliability generalization of Situational Judgment Tests (SJTs). European Journal of Psychological Assessment, 32(3), 230–240.CrossRefGoogle Scholar
  26. Kline, P. (2000). Handbook of psychological testing (2nd ed.). London: Routledge.Google Scholar
  27. Koczwara, A., Patterson, F., Zibarras, L. D., Kerrin, M., Irish, B., & Wilkinson, M. (2012). Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection. Medical Education, 46(4), 399–408.  https://doi.org/10.1111/j.1365-2923.2011.04195.x.CrossRefGoogle Scholar
  28. Libbrecht, N., Lievens, F., & Carette, B. (2014). Emotional intelligence predicts success in medical school. Emotion, 14(1), 64–73.  https://doi.org/10.1037/a0034392.CrossRefPubMedGoogle Scholar
  29. Lievens, F. (2013). Adjusting medical school admission: Assessing interpersonal skills using situational judgement tests. Medical Education, 47, 182–189.  https://doi.org/10.1111/medu.12089.CrossRefPubMedGoogle Scholar
  30. Lievens, F. (2017). Construct-driven SJTs: Toward an agenda for future research. International Journal of Testing, 17, 269–276.  https://doi.org/10.1080/15305058.2017.1309857.CrossRefGoogle Scholar
  31. Lievens, F., Buyse, T., & Sackett, P. R. P. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. The Journal of Applied Psychology, 90(3), 442–452.  https://doi.org/10.1037/0021-9010.90.3.442.CrossRefPubMedGoogle Scholar
  32. Lievens, F., Corstjens, J., Angel, M., Olea, J., Ponsoda, V., & Jos, F. (2015). The cross-cultural transportability of situational judgment tests: How does a US-based integrity situational judgment test fare in Spain? International Journal of Selection and Assessment, 23(4), 361–372.CrossRefGoogle Scholar
  33. Lievens, F., De Corte, W., & Westerveld, L. (2015). Understanding the building blocks of selection procedures: Effects of response fidelity on performance and validity. Journal of Management, 41(6), 1604–1627.CrossRefGoogle Scholar
  34. Lievens, F., & Motowidlo, S. J. (2016). Situational judgment tests: From measures of situational judgment to measures of general domain knowledge. Industrial and Organizational Psychology, 9(1), 3–22.  https://doi.org/10.1017/iop.2015.71.CrossRefGoogle Scholar
  35. Lievens, F., & Patterson, F. (2011). The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection. The Journal of Applied Psychology, 96(5), 927–940.CrossRefGoogle Scholar
  36. Lievens, F., Patterson, F., Corstjens, J., Martin, S., & Nicholson, S. (2016). Widening access in selection using situational judgement tests: Evidence from the UKCAT. Medical Education, 50(6), 624–636.  https://doi.org/10.1111/medu.13060.CrossRefPubMedGoogle Scholar
  37. Lievens, F., Peeters, H., & Schollaert, E. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426–441.  https://doi.org/10.1108/00483480810877598.CrossRefGoogle Scholar
  38. Lievens, F., & Sackett, P. R. (2012). The validity of interpersonal skills assessment via situational judgment tests for predicting academic success and job performance. Journal of Applied Psychology, 97(2), 460–468.  https://doi.org/10.1037/a0025741.CrossRefPubMedGoogle Scholar
  39. Luschin-Ebengreuth, M., Dimai, H. P., Ithaler, D., Neges, H. M., & Reibnegger, G. (2015). Situational judgment test as an additional tool in a medical admission test: An observational investigation. BMC Research Notes, 8(1), 81.Google Scholar
  40. Martin, M., & Motowidlo, S. (2010). A single-response SJT for measuring procedural knowledge for human factors professionals. In Poster Session Presented at 25th Annual Meeting of the Society for Industrial Organizational Psychology.Google Scholar
  41. McCrae, R. R., & Costa, P. (2008). Empirical and theoretical status of the five-factor model of personality traits. In The Sage handbook of personality theory and assessment (pp. 273–294).  https://doi.org/10.4135/9781849200462.n13.
  42. McDaniel, M., Hartman, N., Whetzel, D., & Grubb, W. (2007). Situational judgment tests, response instructions, and validity: A meta-analysis. Personnel Psychology, 60, 63–91.CrossRefGoogle Scholar
  43. McDaniel, M. A., Morgeson, F. P., Finnegan, E. B., & Campion, M. A. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86(4), 730–740.CrossRefGoogle Scholar
  44. McDaniel, M., & Nguyen, N. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9(1/2), 103–113.Google Scholar
  45. McDaniel, M., & Whetzel, D. (2007). Situational judgment tests. In D. Whetzel & G. Wheaton (Eds.), Applied measurement: Industrial psychology in human resources management (pp. 235–257). Mahwah, NJ: Lawrence Erlbaum and Associates.Google Scholar
  46. McDaniel, M. A., Whetzel, D. L., Schmidt, F. L., & Maurer, S. D. (1994). The validity of employment interviews: A comprehensive review and meta-analysis. Journal of Applied Psychology, 79(4), 599–616.CrossRefGoogle Scholar
  47. Motowidlo, S. J., & Beier, M. E. (2010). Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test. The Journal of Applied Psychology, 95(2), 321–333.  https://doi.org/10.1037/a0017975.CrossRefPubMedGoogle Scholar
  48. Motowidlo, S. J., Crook, A. E., Kell, H., & Naemi, B. (2009). Measuring procedural knowledge more simply with a single-response situational judgment test. Journal of Business and Psychology, 24(3), 281–288.CrossRefGoogle Scholar
  49. Motowidlo, S. J., Hooper, A. C., & Jackson, H. L. (2006). Implicit policies about relations between personality traits and behavioral effectiveness in situational judgment items. Journal of Applied Psychology, 91(4), 749–761.  https://doi.org/10.1037/0021-9010.91.4.749.CrossRefPubMedGoogle Scholar
  50. Mussel, P., Gatzka, T., Hewig, J., Mussel, P., Gatzka, T., & Hewig, J. (2016). Situational judgment tests as an alternative measure for personality assessment. European Journal of Psychological Assessment, 1–9.  https://doi.org/10.1027/1015-5759/a000346.CrossRefGoogle Scholar
  51. O’Connell, M. S., Hartman, N. S., McDaniel, M. A., Grubb, W. L., & Lawrence, A. (2007). Incremental validity of situational judgment tests for task and contextual job performance. International Journal of Selection and Assessment, 15(1), 19–29.  https://doi.org/10.1111/j.1468-2389.2007.00364.x.CrossRefGoogle Scholar
  52. O’Neill, L., Vonsild, M., Wallstedt, B., & Dornan, T. (2013). Admission criteria and diversity in medical school. Medical Education, 47(6), 557–561.CrossRefGoogle Scholar
  53. Oswald, F., Schmitt, N., Kim, B., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89(2), 187–207.  https://doi.org/10.1037/0021-9010.89.2.187.CrossRefPubMedGoogle Scholar
  54. Pangallo, A., Zibarras, L., & Patterson, F. (2016). Measuring resilience in palliative care workers using the situational judgement test methodology. Medical Education, 50, 1131–1142.  https://doi.org/10.1111/medu.13072.CrossRefPubMedGoogle Scholar
  55. Patterson, F., Aitkenhead, A., Edwards, H., Flaxman, C., Shaw, R., & Rosselli, A. (2015). Analysis of the situational judgement test for selection to the foundation programme 2015: Technical Report.Google Scholar
  56. Patterson, F., Ashworth, V., Kerrin, M., & O’Neill, P. (2013). Situational judgement tests represent a measurement method and can be designed to minimise coaching effects situational judgement tests represent a measurement method and can be designed to minimise coaching effects. Medical Education (February).  https://doi.org/10.1111/medu.12100.CrossRefGoogle Scholar
  57. Patterson, F., Ashworth, V., Mehra, S., & Falcon, H. (2012). Could situational judgement tests be used for selection into dental foundation training? British Dental Journal, 213(1), 23–26.CrossRefGoogle Scholar
  58. Patterson, F., Ashworth, V., Zibarras, L., Coan, P., Kerrin, M., & Neill, P. O. (2012). Evaluations of situational judgement tests to assess non-academic attributes in selection. Medical Education, 46(9), 850–868.  https://doi.org/10.1111/j.1365-2923.2012.04336.x.CrossRefGoogle Scholar
  59. Patterson, F., Baron, H., Carr, V., Plint, S., & Lane, P. (2009). Evaluation of three short-listing methodologies for selection into postgraduate training in general practice. Medical Education, 43(1), 50–57.CrossRefGoogle Scholar
  60. Patterson, F., Carr, V., Zibarras, L., Burr, B., Berkin, L., Plint, S., … Gregory, S. (2009). New machine-marked tests for selection into core medical training: Evidence from two validation studies. Clinical Medicine, 9(5), 417–420.CrossRefGoogle Scholar
  61. Patterson, F., Knight, A., Dowell, J., Nicholson, S., Cousans, F., & Cleland, J. (2016). How effective are selection methods in medical education? A systematic review. Medical Education, 50, 36–60.  https://doi.org/10.1111/medu.12817.CrossRefGoogle Scholar
  62. Patterson, F., Knight, A., Mcknight, L., & Booth, T. C. (2016). Evaluation of two selection tests for recruitment into radiology specialty training. BMC Medical Education, 16(170), 1–10.  https://doi.org/10.1186/s12909-016-0687-0.
  63. Patterson, F., Lievens, F., Kerrin, M., Munro, N., & Irish, B. (2013). The predictive validity of selection for entry into postgraduate training in general practice: Evidence from three longitudinal studies. British Journal of General Practice, 63(616), 734–741.  https://doi.org/10.3399/bjgp13X674413.CrossRefGoogle Scholar
  64. Patterson, F., Lievens, F., Kerrin, M., Zibarras, L., & Carette, B. (2012). Designing selection systems for medicine: The importance of balancing predictive and political validity in high-stakes selection contexts. International Journal of Selection and Assessment, 20(4), 486–496.  https://doi.org/10.1111/ijsa.12011.CrossRefGoogle Scholar
  65. Patterson, F., Lopes, A. S., Harding, B. S., Vaux, C. E., E, D. L. B., & F, D. B. (2017). The predictive validity of a situational judgement test, a clinical problem solving test and the core medical training selection methods for performance in specialty training (February).  https://doi.org/10.7861/clinmedicine.17-1-13.CrossRefGoogle Scholar
  66. Patterson, F., Prescott-Clements, L., Zibarras, L., Edwards, H., Kerrin, M., & Cousans, F. (2015). Recruiting for values in healthcare: A preliminary review of the evidence. Advances in Health Sciences Education, 21(4), 859–881.  https://doi.org/10.1007/s10459-014-9579-4.CrossRefGoogle Scholar
  67. Patterson, F., Rowett, E., Hale, R., Grant, M., Roberts, C., Cousans, F., et al. (2016). The predictive validity of a situational judgement test and multiple-mini interview for entry into postgraduate training in Australia. BMC Medical Education, 16(1), 87.  https://doi.org/10.1186/s12909-016-0606-4.
  68. Patterson, F., Zibarras, L., & Ashworth, V. (2015). Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100. Medical Teacher, 0(0), 1–15.  https://doi.org/10.3109/0142159X.2015.1072619.CrossRefGoogle Scholar
  69. Ployhart, R., & Ehrhart, M. G. (2003). Be careful what you ask for: Effects of response instructions on the construct validity and reliability of situational judgment tests. International Journal of Selection and Assessment, 11(1), 1–16.  https://doi.org/10.1111/1468-2389.00222.CrossRefGoogle Scholar
  70. Ployhart, R., & Weekley, J. (2006). Situational judgment tests: A research synthesis and directions for future research. In J. Weekley & R. E. Ployhart (Eds.), Situational judgment tests (pp. 345–350). Mahwah, NJ: Lawrence Earlbaum.Google Scholar
  71. Ployhart, R. E., Weekley, J. A., Holtz, B. C., & Kemp, C. (2003). Web-based and paper-and-pencil testing of applicants in a proctored setting: Are personality, biodata, and situational judgment tests comparable? Personnel Psychology, 56, 733–752.CrossRefGoogle Scholar
  72. Powis, D. (2014). Selecting medical students: An unresolved challenge*. Medical Teacher (April), 252–260.  https://doi.org/10.3109/0142159X.2014.993600.CrossRefGoogle Scholar
  73. Rust, J., & Golombok, S. (1999). Modern psychometrics: The science of psychological assessment. London: Routledge. Google Scholar
  74. Schmidt, F. (1994). The future of personnel selection in the U.S. Army. In J. Weekley & R. Ployhart (Eds.), Personnel selection and classification (pp. 333–350). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  75. Simon, E., Walsh, K., Paterson-Brown, F., & Cahill, D. (2015). Does a high ranking mean success in the Situational Judgement Test? The Clinical Teacher, 12(1), 42–45.CrossRefGoogle Scholar
  76. Stemig, M. S., Sackett, P. R., & Lievens, F. (2015). Effects of organizationally endorsed coaching on performance and validity of situational judgment tests. International Journal of Selection and Assessment, 23(2), 174–181.CrossRefGoogle Scholar
  77. Stemler, S., Aggarwal, V., & Nithyanand, S. (2016). Knowing what not to do is a critical job skill: Evidence from 10 different scoring methods. International Journal of Selection and Assessment, 24(3), 229–245.CrossRefGoogle Scholar
  78. Wakeford, R., Denney, M.-L., Ludka-Stempien, K., Dacre, J., & McManus, I. (2015). Cross-comparison of MRCGP & MRCP (UK) in a database linkage study of 2,284 candidates taking both examinations: Assessment of validity and differential performance by ethnicity. BMC Medical Education, 15(1), 1–12.CrossRefGoogle Scholar
  79. Weekley, J. A., & Jones, C. (1999). Further studies of situational tests. Personnel Psychology, 52(3), 679–700.CrossRefGoogle Scholar
  80. Wernimot, P., & Campbell, J. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 3, 188–202.Google Scholar
  81. Whetzel, D. L., & McDaniel, M. A. (2009). Situational judgment tests: An overview of current research. Human Resource Management Review, 19(3), 188–202.  https://doi.org/10.1016/j.hrmr.2009.03.007.CrossRefGoogle Scholar
  82. Whetzel, D. L., McDaniel, M. A., & Nguyen, N. T. (2008). Subgroup differences in situational judgment test performance: A meta-analysis. Human Performance, 21(3), 291–309.  https://doi.org/10.1080/08959280802137820.CrossRefGoogle Scholar
  83. Whetzel, D. L., & Reeder, M. C. (2016). Why some situational judgment tests fail to predict job performance (and others succeed). Human Resources Research Organization, 9(1), 71–77.CrossRefGoogle Scholar
  84. Woolf, K., Potts, H., & McManus, I. (2011). Ethnicity and academic performance in UK trained doctors and medical students: Systematic review and meta-analysis. BMJ, 342, d901.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Authors and Affiliations

  1. 1.Work Psychology GroupDerbyUK

Personalised recommendations