Introduction

In medical school selection, a growing emphasis is placed on the measurement of so-called ‘non-academic’ characteristics, such as professionalism, empathy, communication skills, and integrity (Salvatori 2001; Siu and Reiter 2009). One instrument measuring such ‘non-academic’ variables is the Situational Judgement Test (SJT), which is increasingly used for medical school selection purposes. In SJTs, applicants’ reactions to hypothetical scenarios they might encounter in the occupation they are applying for are assessed (Lievens and Coetsier 2002; Patterson et al. 2015). Research on SJTs in medical school selection reports good validity for future job performance: SJT performance was found to significantly predict performance in interpersonal courses, internships, OSCEs, and jobs (Lievens et al. 2005; Lievens 2013; Lievens and Sackett 2012). In this study, we examine whether different types of applicants respond differently to SJT scenarios set in certain medical contexts.

Research on SJT use in medical student selection shows that SJT scores differ between white individuals and other ethnic groups: white applicants score better than applicants from other ethnic backgrounds (Whetzel et al. 2008). Additionally, adverse impact of gender has been reported: female applicants tend to score slightly better in SJTs than male applicants (Lievens 2013; Whetzel et al. 2008). However, the influence of factors other than gender and ethnicity on SJT performance in medical school selection has not yet been studied. We wondered whether differences between applicants in terms of vocational interests, academic experience and age are related to differences in SJT performance in medical school selection.

In SJTs, applicants judge multiple responses to written or video-based scenarios in which occupation-related situations are depicted. Depending on the evaluation method, applicants rate or rank the responses for appropriateness, or name the most appropriate response(s) (Patterson et al. 2012). SJTs aim at giving an indication of so-called ‘non-academic’ skills, not at measuring clinical knowledge (Lievens et al. 2008). In medical school admissions, this means that applicants need not have experience within the medical field, and should be able to judge SJT situations regardless of their clinical knowledge. However, even if clinical knowledge is not required in an SJT, the situations depicted in the scenarios may be more familiar or attractive to one applicant than to another, depending on their experiences and personal interests. In a field as large as the medical profession, an infinite number of possible scenario settings is available. Likely, applicants differ in their interest in different fields within health care. Human Resources research shows that the ‘fit’ between vocational interests and job characteristics is predictive of job performance (Van Iddekinge et al. 2011). Job performance is better among those who have an interest in the specific content of their job, than among those whose vocational interests lie somewhere else. We wondered whether there would be a similar relation between vocational interests of medical school applicants and performance on SJT scenarios in different settings.

In addition to factors related to vocational interests, previous experience in the academic context might influence SJT performance in medical school admissions. The transition from pre-university education to medical school means a change of educational environment. Likely, this new environment requires newly admitted students to adapt to a new culture and to the norms and values in the academic community. Experience in this setting may influence performance on SJT scenarios in the admissions process because applicants with this experience have already learned what kind of behaviour is appropriate in the university context. Therefore, graduate-entry applicants might for example score higher on SJT scenarios than undergraduate-entry applicants. Additionally, there is a possibility that age plays a role in SJT performance. Older applicants have more life experience, which might lead to a better understanding of interpersonal relations and consequently, better SJT performance.

In the current study, we therefore examined relations of vocational interests, prior experience in the academic setting, age, and gender with performance on SJT scenarios in different settings. We focused on the following questions:

  1. 1.

    Is vocational interest related to SJT performance in medical school admissions?

  2. 2.

    Is prior experience in the academic setting related to SJT performance in medical school admissions?

  3. 3.

    Is age related to SJT performance in medical school admissions?

  4. 4.

    Is gender related to SJT performance in medical school admissions?

Methods

Context

This study was performed at University of Groningen, the Netherlands. Medical training at this university is divided in a three-year pre-clinical Bachelor’s degree programme followed by a three-year clinical Master’s programme. In this study, applicants for the Bachelor’s degree programme are considered undergraduate-entry applicants, whereas applicants who have already obtained a Bachelor’s degree in a field related to medicine are considered graduate-entry applicants.

In the Bachelor’s degree programme, all students acquire the same medical knowledge in the core programme, but intrinsic competencies are acquired in four different learning communities. Each learning community consists of teachers and students interested in a specific area of health care. In the admissions process, applicants have to choose one of the four learning communities based on their vocational affinities and interests: Global Health, Sustainable Care, Intramural Care, and Molecular Medicine. The different learning communities lead to the same Bachelor’s degree and the medical content is the same for all communities, but students learn intrinsic competencies in community-specific settings. During the 3-year Bachelor’s programme, students have to stay in the learning community they chose during the admissions process. We approached the choice for a specific learning community as a proxy for vocational interest.

In the Global Health learning community, the focus lies on major health issues that influence individuals and populations worldwide. The Sustainable Care learning community focuses on continuous, effective, timely, safe and affordable care for the (elderly) patient at home. In this community, the focus is on the sustainability of health care after 2025. The Intramural Care community focuses on more complex health care in which teams of professionals have to deal with (in-house) patients with increasingly complex diagnoses. Themes in this community are patient safety, academic research underlying medical treatment and collaborations between various medical specialists. In the Molecular Medicine community, the focus is on the molecular causes of diseases and methods for diagnosis and therapy. The Global Health and Molecular Medicine communities are taught in English, whereas the Sustainable Care and Intramural Care communities are taught in Dutch.

Every year, there are 410 places available in the Bachelor’s programme in Medicine at University of Groningen medical school. In the admissions process, applicants are ranked for the Bachelor’s degree programme on one ranking list, independent of their learning community choice. Applicants are informed about this. The highest-ranking applicants are offered a place in the programme in the learning community of their choice. The size of the different learning communities therefore differs between years.

In addition to the regular Bachelor’s programme, University of Groningen offers a pre-master programme for graduate-entry students. This one-year programme provides students with a certificate with which they can enrol in the same 3-year Master’s degree programme as students with a Bachelor’s degree in Medicine. Students who are eligible for the graduate-entry programme have acquired a Bachelor’s degree outside, but related to, the field of medicine, such as biomedical sciences or pharmaceutics. Every year, a maximum of 45 students is admitted to the graduate-entry programme. We approached the graduate-entry applicant group as a group with previous academic experience.

Participants

616 applicants participated in the selection process for the Bachelor in Medicine at University of Groningen for the academic year 2015–2016 (mean age 19.0; 68.4% females). All participating applicants were included in the analyses. Applicants had to choose one out of four learning communities: Global Health (n = 126), Sustainable Care (n = 149), Intramural Care (n = 225), or Molecular Medicine (n = 116).

225 applicants applied for the graduate-entry programme for 2015–2016 (mean age 24.1; 64.8% females). For 12 applicants, date of birth was not registered. These students were excluded from the analyses. Therefore, 213 applicants to the graduate-entry programme were included in the analyses.

The privacy policy of the University of Groningen states that student records can be used for research purposes, as long as reports cannot be traced back to individual students (University of Groningen2015). In accordance with this privacy policy, data were derived from the university administration and anonymised for analysis.

SJT measurement

The SJT was a (small) part of both the selection process for the Bachelor’s degree programme and the graduate-entry programme. The selection process consisted of an assessment of cognitive qualities combined with an assessment of ‘non-academic’ skills. Within the ‘non-academic’ assessment, there were different assignments, among which the SJT. The SJT contained six different scenarios, each accompanied by five behavioural responses. The scenarios are displayed in Table 1. All applicants sat the same six-scenario SJT.

Table 1 SJT scenarios
Full size table

All scenarios had the same lay-out (see “Appendix”): for each scenario, applicants had to decide whether each response was professional or unprofessional. They had to rank all behavioural responses on a bar which was divided in two parts: unprofessional and professional. Subsequently, applicants were asked to give arguments for their placement on the bar of the different responses. Applicants could obtain 10 points for the ranking assignment, and 10 points for their argumentation. For the ranking part of each assignment, points were granted according to the following rules:

  • All behavioural responses in the correct half of the bar and in the correct order: 10 points

  • Extremities correct, all responses in the correct half of the bar (but wrong order): 8 points

  • Extremities correct, correct order, (middle) responses in wrong half of the bar: 5 points

  • Extremities wrong, wrong order, responses in correct half of the bar: 3 points

  • All other cases where 5 responses are placed on the bar: 1 point

For the argumentation part of each assignment, 2 points per behavioural response could be obtained. For each response, 2 points were awarded for a valid, complete argument. 1 point was awarded when an argument was valid but incomplete.

The SJT was designed by a committee consisting of physicians, ethicists and psychologists. The scenarios were designed to reflect realistic situations within health care. In the design of the SJT, the committee aimed at formulating a diverse set of scenarios placed in different settings in the medical and academic context. Due to the low number of scenarios and the diversity of the scenario settings, the complete SJT reached only a moderate internal consistency, with a Cronbach’s alpha of 0.43. This was not considered a problem, as the different settings might measure different, but relevant, dimensions of professionalism. The SJT was only a small part of the non-academic assessment, accounting for approximately 25% of the final score in the non-academic assessment, and 12.5% of the final selection score.

All scenarios were discussed in the committee and the order of appropriateness of the behavioural responses was discussed until consensus was reached. Subsequently, the committee defined the answer key. After the selection process, the SJT was rated by graduate degree law students according to this predefined answer key. These students were specifically trained for this assessment.

Analysis

We performed multivariate analysis of covariance (MANCOVA) with scores on the SJT scenarios as dependent variables and the different groups of applicants, gender and age as independent variables. To examine pairwise differences between applicants in the different groups, we performed Bonferroni post hoc multiple comparison tests. All analyses were conducted using IBM SPSS for Windows version 23 (IBM Corp 2015).

Results

Descriptive statistics

Descriptive statistics for percentages of females and mean age per group are displayed in Table 2. Analysis of variance (ANOVA) showed no significant differences in the percentages of females between the five groups. The mean age among applicants for the Global Health community was significantly higher than among applicants for the Sustainable Care and Intramural Care communities. The mean age of graduate-entry applicants was significantly higher than in the other groups.

Table 2 Descriptive statistics for percentage of females and mean age per group
Full size table

Impact of applicant characteristics on SJT performance

Multivariate analysis of covariance (MANCOVA) showed that all independent variables were significantly related to SJT performance (Pillai’s TraceGroup = .47, F24,3280 = 18.34, p < .001; Pillai’s TraceGender = .03, F6,817 = 4.96, p < .001; Pillai’s TraceAge = .02, F6,817 = 3.18, p < .01).

Group means for each SJT scenario, adjusted for gender and age effects, are displayed in Table 3. Bonferroni post hoc multiple comparison tests showed that there were significant differences between applicants for the different learning communities.

Table 3 MANCOVA analyses for SJT performance with Bonferroni post hoc pairwise between-group comparisons
Full size table

Graduate-entry applicants outperformed several other groups on all SJT scenarios (Mean difference (MD) = 1.41–3.34, p < .05). Significant group differences for each scenario are displayed in the most right column of Table 3.

Female applicants outperformed male applicants on the Housekeeping, Overcrowded practical, and Great-aunt scenarios (MD = .42–.94, p < .05)(Table 2).

Age was positively related to performance on the Housekeeping and Great-aunt scenarios (F1, 822 = 5.72 and 6.43, respectively, p < .05) (Table 2).

Discussion

In the current study, we examined whether vocational interests, previous academic experience, gender, and age were related to performance on SJT scenarios set in different contexts. Effect sizes were small to moderate. We found that vocational interest was related to performance on the different SJT scenarios. Previous academic experience was related to higher performance in the SJT, regardless of age. Female applicants outperformed male applicants on three out of six scenarios, whereas age was positively related to performance on two out of six scenarios.

This study provides, to our best knowledge, a first indication that SJT performance is related to the vocational interests of applicants. Upon closer inspection, we noticed that the scenarios for which there was a difference between the groups of students with different vocational interests, were fitting for the different learning community choices. For example, applicants with an interest in the themes of Sustainable Care performed better than the other groups on the scenario that depicted an ethical dilemma regarding an elderly couple and their need for home care. Similarly, applicants with an interest in Molecular Medicine outperformed their peers on the scenario that describes a laboratory setting. For this scenario, the proportion of explained variance by the model was much higher than for the other scenarios, indicating that this scenario might have been particularly attractive to the Molecular Medicine applicants and graduate-entry applicants than to the other applicant groups. It seems that, similar to the positive influence of a good ‘fit’ between personal interests and job characteristics on job performance, a good match between an SJT scenario setting and the vocational interests of applicants seems related to higher SJT performance.

We found a positive relation between previous experience in the academic setting and SJT performance, regardless of SJT setting and age. The higher performance among graduate-entry applicants might be related to the process of professional identity development, in which students increasingly identify themselves with a certain professional group (Crossley and Vivekananda-Schmidt 2009; Gude et al. 2005). It is possible that applicants with prior experience in the academic setting have developed a professional identity suited to the academic culture, whereas the professional identity of applicants without this experience is still more tuned to high school or other professional environments they may have worked in. Another explanation for the higher scores among experienced applicants may be that these applicants are advantaged by their previous education, which, for graduate-entry applicants, is related to the field of medicine. These students may have previous experience in a medical environment that helps them in the SJT.

Female applicants outperformed male applicants on three out of six scenarios, regardless of vocational interests and age. This was in line with previous research showing that in SJTs, females slightly outperform males (Lievens 2013; Whetzel et al. 2008). A possible explanation for the difference between females and males in performance on the difference scenarios could be that females generally score higher on personality factors such as conscientiousness and agreeableness, and some scenarios might relate to these factors more strongly than others (Whetzel et al. 2008). We found age effects for two out of six scenarios, on which older applicants scored better than younger applicants.

The relation between SJT performance on the different scenarios and applicants’ vocational interests was unanticipated. Therefore, we could refer to this effect as adverse impact. It could be deemed a threat to fairness that with our choice for certain settings for the SJT scenarios, we gave applicants who have a personal affinity with these settings an advantage over applicants interested in other health care settings. However, when looking at this effect from a social accountability perspective (Woollard 2006), it could provide medical schools with an opportunity to select applicants with an interest in areas of health care in which there is, or will be, a shortage of medical professionals. For example, in countries where there is a shortage of medical professionals in rural areas, medical schools could implement an SJT in which many scenarios are set in a rural contexts, in order to give an advantage to those applicants who are interested in working in a rural area. Similarly, SJT scenarios could be designed in such a way that applicants with an interest in working with elderly people (a growing demographic group that often needs long-term health care), are more likely to perform well on a SJT scenario setting than applicants with an interest in working with children (a relatively healthy demographic group, which is decreasing in size). However, this study only provides a first indication of the impact of vocational interests on performance in related SJT settings and the effects should be confirmed and analysed in further detail before it can be used for selection decisions and social accountability purposes.

Our study adds to the existing literature by providing a first indication of an association of applicants’ personal preferences and experience with SJT performance in medical school admissions. Earlier research has shown adverse impact of gender and ethnicity, but other influences have not been thoroughly examined. However, the current study has some limitations. First, our analyses were carried out at a single university, where applicants can only apply for one of four learning communities, which focuses on a specific area within the medical field. This is a rather unique situation, and it remains unclear whether this setup draws different applicants than universities where all applicants are admitted into the same programme. Our results may therefore not be generalizable to other contexts. However, this setup did provide us with the possibility to examine the relation between vocational interest and performance on specific SJT scenario settings. Second, our analyses were limited to six SJT scenarios. In this design, differences in SJT scenario settings will have a larger impact on selection outcomes than in designs with more scenarios. When a larger number of scenarios are used, the total SJT can cover more areas of health care and therefore the impact of matches and mismatches between applicant and scenario settings will have a smaller influence.

In conclusion, this study indicates that applicants’ vocational interests, academic experience, gender and age are related to performance on SJT scenarios in different settings. Especially the first effect might be helpful in selecting appropriate candidates for areas of health care in which more professionals are needed.