Introduction

Places in medical school are scarce and medical education and training are expensive for providers and learners. Therefore, medical schools aim to offer the places available only to those applicants with the highest probability of successful medical training and subsequent career. To reach this goal, medical schools have developed several selection procedures, including interviews, admission tests and other measures of personal competencies (Kreiter and Axelson 2013), although the evidence that these procedures indeed do deliver better achieving students (Salvatori 2001; Siu and Reiter 2009), let alone better professional doctors (Papadakis et al. 2005) is limited. Whereas traditionally the focus in selection has been on academic indicators, there is increasingly more attention for non-academic attributes that are considered important for success in clinical practice (Patterson et al. 2016). As with all selection tools, it is critical to explore the reliability and validity of approaches to selecting for non-academic, or personal qualities.

A parameter frequently used in student selection procedures is prior academic attainment, such as the grade point average for the final undergraduate examinations (uGPA). uGPA shows a strong relationship with student achievement in medical school (Siu and Reiter 2009) but explains just 16–25% of the variance in pre-clinical achievement (Salvatori 2001) and less than 10% in clinical achievement (Benbassat and Baumal 2007; Veloski et al. 2000). This decrease in predictive value of prior academic attainment by increasing time from medical school admission has also been shown for schools with undergraduate entry in the Netherlands and Germany (Stegers-Jager et al. 2015b; Trost et al. 1998). Additionally, setting high academic grades as a threshold for entering medicine has an adverse impact for non-traditional applicants including those from minority or lower social-economic backgrounds (Cleland et al. 2012).

It has been shown that certain characteristics, such as ability, motivation, ambition and conscientiousness, have, at the very least, a moderately positive bearing on student performance (Ferguson et al. 2003; Lievens et al. 2002) The Multiple Mini Interview as well as Situational Judgement Tests, more recently developed to embed non-academic skills into the selection procedure, showed favourable results even for clinical performance (Lievens 2013; Pau et al. 2013), although it is not easy to validate the use of such tests for selection purposes due to the absence of control groups (Kulatunga-Moruzi and Norman 2002). Thus, performance in medical school appears to be multifactorial with intellectual ability as well as personality and motivation playing an important role (Collins et al. 1995), resulting in two types of predictors; academic predictors with prior academic attainment as their best representative, and non-academic predictors which remain a less well explored area.

An alternative way of operationalizing non-academic skills is examining applicants’ extracurricular activities during pre-university education (puECAs) (O’Neill et al. 2011; Schripsema et al. 2014; Urlings-Strop et al. 2013). An advantage of this method is that it is not based on a ‘single’ test administration but that it reflects a student’s development over the last couple of years, hence increasing its authenticity. Astin (1999) proposed an involvement theory where involvement was defined as active participation in all kinds of (extra)curricular and social activities. Highly involved students had a lower risk to drop out (Astin 1975). Pike (2000) reported that involvement in a variety of curricular and co-curricular activities was directly related to growth in general abilities. Huang and Chang (2004) found that improvements of academic, communication and interpersonal skills were associated with intra- and extracurricular involvement. Using puECAs for selection to medical school showed encouraging results in preventing dropout (O’Neill et al. 2011; Urlings-Strop et al. 2009), and predicting pre-clinical (Schripsema et al. 2014) and clinical achievement (Urlings-Strop et al. 2013, 2011). This latter finding raised the question why students selected on the basis of their participation in extracurricular activities during pre-university education receive higher grades in clinical training.

In the Netherlands students for medical school are in part admitted by lottery and in part selected by a medical school-specific selection procedure (Ten Cate 2007). This situation presents a unique control group of randomly (lottery-) admitted students compared with those selected by a school-specific procedure which at our medical school combined non-academic (puECAs) and academic (cognitive tests) criteria (Urlings-Strop et al. 2009). We hypothesized that selected students who had completed their pre-university education with the same GPA as their lottery admitted controls, but had shown the ambition and ability to participate in pre-university extracurricular activities, use this same ambition and ability to continue performing extracurricular activities at medical school. The aim of the current study is to examine whether students who were selected on the basis of their puECAs persisted in their ECAs during medical school (msECAs) and whether this persistent participation in msECAs explains their better achievement in the clinical years of medical school. If so, this would further support the choice of using puECAs as a non-academic selection tool in medical school selection procedures.

Methods

Selection procedure

Since 2000, there have been three ways to gain admittance to medical school in the Netherlands: a school-specific selection procedure (S); the national lottery system (L), and direct access for students with a pu-GPA above 8.0 (D). This D-group was excluded from the analyses. The local selection procedure at Erasmus MC Medical School consists of two steps. In the first step, applicants are assessed according to the quality and quantity of extracurricular activities before application in one or more of the following five categories: (1) activities in health care, (2) activities in management and organisation, (3) activities related to the development of a (individual) talent e.g. for music, sport or science; (4) (extracurricular) academic education; and (5) additional subjects during pre-university education. In the second, academic step, applicants take five tests on a medical subject preceded by informative classes. These locally developed tests focus on the subjects logical reasoning, scientific thinking, epidemiology and pathology, anatomy and mathematics. Applicants who are ranked above the mean in the first step of selection are invited to proceed to the second step. In the second step applicants need to pass four of the five tests and to achieve an average score across the five tests of ≥5.5 (on a 10-point scale, 1 = poor, 10 = excellent). When the target number of students to be selected is not met, more students are admitted through the lottery system. A more extensive description of the selection procedure has been provided previously (Urlings-Strop et al. 2009).

Curriculum

The undergraduate medical curriculum at Erasmus MC Medical School has been described previously (Urlings-Strop et al. 2011). The medical curriculum at the time of the study consisted of a 4-year pre-clinical phase followed by a 2-year clinical phase. The clinical phase consisted of a period of 15 weeks of general clinical training and 69 weeks of discipline-specific clerkships comprising 10 different rotations. At the end of each clerkship, the student’s performance was assessed using a combination of patient-related assessment and oral examination and rewarded with a grade between 5 (unsatisfactory) and 10 (excellent).

Variables

Participation in msECAs

A first outcome measure of this study was participation in extracurricular activities during medical school (msECAs). Three types of extracurricular activities were considered: (1) completing a research master program, (2) conducting important administrative or organisational functions at Erasmus MC Medical School and (3) enrolling in an additional degree course at Erasmus University Rotterdam. All students were allowed to participate in one or more of these types of extracurricular activities, although entry criteria applied (e.g. for the research masters, see below) and in some cases places were limited (e.g. for the board positions). These three types of msECAs were chosen as they could reliably be measured and required a substantial time investment of the students.

Research masters

Erasmus MC Medical School offers motivated and talented students a scholarship for one of the four officially accredited Research Master programmes: clinical epidemiology, neuroscience, molecular medicine and clinical research. The requirement for enrolling the programme is obtaining the 60 credits of the first year at the end of that year. The study load of these programmes is 120 credits during two years, running parallel to the regular medical curriculum.

Board positions

Students can apply for a position in the board of the medical student union, membership of the curriculum committee, membership of the faculty council, and membership of the university council. Students fulfil a position in one of these participatory decision-making committees mostly for the duration of one academic year, although in some cases for two or more years. They can fulfil these positions during the second through fourth year of the pre-clinical curriculum.

Additional degree

Some students choose to enrol in an additional full-time degree course at Erasmus University Rotterdam such as law or philosophy, running parallel to the medical curriculum and lasting four years with a study load of 60 credits per year. All students are allowed to apply for an additional course at any time during the pre-clinical curriculum.

Participation in at least one of these three types of msECAs was coded as yes on this dichotomous variable.

Clerkship GPA

A second outcome measure of this study was clerkship GPA. Clerkship GPA was calculated as the mean of the grades obtained on the 10 discipline specific clerkships. Grades were given on a 10-point scale (1 = poor, 10 = excellent) and 5.5 was the cut-off pass/fail mark.

Pre-university GPA

As pre-university GPA (pu-GPA) is known to be associated with performance at medical school, it was included in the analyses as a confounder/covariate.

Pre-university GPA represents a students’ mean grade obtained during the final year of pre-university education. Final grades in the Netherlands are based on school examinations (50%) and the national examination (50%). Within each cohort, pu-GPAs were translated into Z-scores in order to diminish cohort differences.

Participants and procedure

During the four years of the experiment (2001–2004), 389 students were selected (S-group) and 938 students were admitted by lottery (L-group). Of the S-group and L-group, 338 (86.9%) and 755 (80.5%) were eligible to start clerkships respectively (Urlings-Strop et al. 2011). Follow-up for all students was at least 5.5 years allowing sufficient opportunity to take part in any of the msECAs. Over the years 2001 through 2009, we collected information about student participation in one of the three classes of msECAs. Information about participation in a Research Master programme or an additional degree course was derived from the Erasmus University Rotterdam student administration systems. Proof of membership of the students’ union and curriculum committee was obtained through the Erasmus MC annual reports, members of the university council through the Erasmus University Rotterdam annual reports and members of the board of the student fraternity from their yearbooks.

Students’ grades were obtained from the university administration system and delivered anonymously to the investigators. Participants in this study did not suffer any adverse consequences of being a subject in this study. According to Dutch law, this study was exempt from ethical approval requirements.

Statistical analysis

First we assessed associations between admission group and participation in any and in specific types of msECAs using Chi squared tests. To reveal whether the kind of puECA of the S-group was associated with participation in msECAs, we evaluated the participation in msECAs for each puECA-category used in the selection procedure separately.

A p value of <0.05 was considered statistically significant. Effect sizes (ES) were calculated directly from Chi squared tests with ES ≈ 0.10 indicating a small effect, ES ≈ 0.30 a medium effect, and ES ≈ 0.50 a large effect (Hojat and Xu 2004).

Second, we used logistic regression to calculate an odds ratio (OR) for the effect of admission group on participation in msECA, adjusted for pu-GPA. To assess whether pu-GPA had the same association with participation in msECA for selected and lottery admitted students, the interaction term ‘admission group’ x ‘pu-GPA’ was included in the model that also included admission group and pu-GPA as main effects.

Third, to examine the inter-relationships between admission group, participation in msECAs and clerkship grade we used a two-way analysis of covariance (ANCOVA) with admission group and participation in msECAs as independent variables and pu-GPA as covariate. Simple effects analysis was used to study the effect of participation in msECA for each of the two admission groups. Again a p value of <0.05 was considered significant. In addition, effect sizes, partial eta squared (η2), were obtained with values of 0.01, 0.06 and 0.14 indicating small, medium or large effects, respectively (Cohen 1988).

Analyses were performed using IBM SPSS Statistics for Windows Version 21.0.

Results

On the qualifying date (1 January 2015), 1087 (99.6%) of the 1093 eligible students from cohorts 2001–2004 had completed all ten discipline-specific clerkships. Overall, 174 (16.0%) of these students participated in one of the three defined types of msECAs (Table 1). This percentage was almost twice as high for the S-group (23.7%) compared to the L-group (13.0%). S-group students in particular more often completed a Research Master or fulfilled a board position. For students in the S-group participation in msECA was not associated with a particular puECA category (Table 2).

Table 1 Students participation in extracurricular activities (msECAs) per admission group
Full size table
Table 2 Pre-admission ECA (puECA) category and participation in ECAs at medical school (msECA) of the S-group students
Full size table

Both admission group and pu-GPA were significant predictors of participation in msECAs (Table 3). However, we found a statistically significant differential effect of admission group by pu-GPA (Table 3, Fig. 1). The effect of pu-GPA on participation in msECA was prominent for L-group students but absent for S-group students. Specifically, with a pu-GPA below 1.0 (z-score) L-group students were much less likely to participate in msECAs than students in the S-group with a similar pu-GPA. Although the interaction effect suggests that with a high pu-GPA (z >1.0) the L-group students were more likely to participate in msECAs than the S-group students with a similar GPA (Fig. 1), additional analyses revealed that for the relatively small number of students with a high pu-GPA (n = 125; 12.1%) there was no statistically significant difference in participation rate between S-group and L-group students (χ2(1) = 0.123, p = 0.73).

Table 3 Multiple Logistic Regression Model—Predictors of participation in msECAS
Full size table
Fig. 1
figure 1

Interaction effect between pu-GPA and admission type on the probability of participation in msECA. pu pre-university; GPA grade point average; S-group selected students; L-group lottery-admitted students; msECA extracurricular activities performed during medical school

Full size image

The two-way ANCOVA regarding differences in the effect of participation in msECAs on clerkship GPA for students in the S-group and the L-group showed significant main effects of both participation in msECAs (F (1, 1030) = 9.88, p < 0.01, partial η2 = 0.009) and admission group (F(1,1030) = 20.72, p < 0.001, partial η2 = 0.020). msECA participants had higher grades than non-participants and selected students had higher grades than lottery-admitted students. However, there also was a significant interaction effect between participation in msECAs and admission group (F(1,1030) = 8.50, p < 0.01, partial η2 = 0.008), indicating that participation in msECAs has a different association with clerkship GPA for selected than for lottery-admitted students. Specifically, S-group students that participated in msECAs had significantly higher clerkship grades than S-group non-participants, while this difference did not exist among L-group students (see Table 4 and Fig. 2).

Table 4 Clerkship GPA by admission group and participation in msECA
Full size table
Fig. 2
figure 2

Interaction effect between participation in msECA and admission group on clerkship grade. S-group selected students; L-group lottery admitted students; msECA extracurricular activities performed during medical school. * Estimated marginal means of clerkship grade, covariate evaluated at the value pre-university GPA = −0.07 (mean)

Full size image

Discussion

This study indicates that persistent activities of students selected on extracurricular activities explain their better achievement during the clinical years of medical school. Selected students not only participate more often in extracurricular activities during medical school than lottery-admitted students, their participation is also not associated with their pu-GPA, whereas lottery-admitted students tend to only participate if they have a high pu-GPA. Finally, participation in extracurricular activities is associated with higher clerkship grades for selected students but not for lottery-admitted students.

The finding that students selected on extracurricular activities before medical school participated more often in extracurricular activities during medical school than their lottery-admitted comparisons was according to our expectations based on the principle of behaviour consistency: the best predictor of future behaviour is past behaviour in similar situations (Ouellette and Wood 1998; Stegers-Jager et al. 2015b). However, to our knowledge this is the first study to report on the persistence of extracurricular activities after admission despite the fact that several medical school have used extracurricular activities as a selection criterion (O’Neill et al. 2011; Schripsema et al. 2014; Urlings-Strop et al. 2009). Interestingly, participation in extracurricular activities during medical school was not associated with the type of extracurricular activities employed before medical school. Apparently, it is rather the intention or ambition to perform extracurricular activities in general than a specific type of extracurricular experience that influences the decision to participate in extracurricular activities at medical school (Huang and Chang 2004).

One of the most striking outcomes is that for selected students the decision to participate in extracurricular activities during medical school does not depend on their pu-GPA whereas it does for lottery-admitted students, despite the absence of differences in pu-GPA or pre-clinical achievement between the two admission groups (Urlings-Strop et al. 2009, 2011). A possible explanation, as suggested previously by Schripsema et al. (2015), is that the selected students have better time management skills than lottery-admitted students with similar pre-university GPAs. These skills not only enable them to participate in the time-consuming selection procedure during their pre-university examinations, but also to participate in extracurricular activities during medical school. Another possible explanation has to do with students’ self-efficacy, i.e., their belief in their ability to succeed in specific situations. As the most powerful source of self-efficacy beliefs is past performance (Bandura 1997), it can be expected that students who were able to participate in extracurricular activities next to their pre-university education are more confident that they will be able to do so successfully during medical education than students who did not have this positive past experience.

Another striking finding is that participation in extracurricular activities is associated with higher clerkship grades, but only for selected students. Lottery-admitted students do not seem to benefit from their participation in extracurricular activities, at least not with respect to their clerkship grades. A first possible explanation is that the rationale for participation in extracurricular activities is different for selected than for lottery-admitted students. A theory that appears promising in explaining voluntary participation is Higgins' regulatory focus theory (Higgins 1997, 1998). Following this theory it might be that S-group students participate in extracurricular activities because they ‘want to’ (promotion focus), while L-group students participate because they feel they ‘have to’ (prevention focus). This might have been particularly so in the first years of our selection procedure when applicants were less aware of the requirements for admission. As described by Lucieer et al. (2016) it might be that in more recent years—since the requirements for admission have become more transparent—applicants invest time in extracurricular activities just because they want to enter medical school. Possible differences in regulatory focus between lottery-admitted and selected students and their relation with clerkship grades may be an interesting area for further research.

As we have suggested previously, participation in extracurricular activities may favour the development of relevant non-academic qualities and skills that will contribute to better clinical performance (Urlings-Strop et al. 2013). Indeed, another medical school that included puECAs in their procedure concluded that selected students had better skills in terms of collaboration, communication, reflection, ethical decision making and professional behaviour during the first three years of medical school (Schripsema et al. 2014). These competencies in particular were rated important for the clerkship years by clerkship directors (Windish et al. 2004).

The observation that lottery-admitted students do not improve their clinical achievement after participating in msECAs suggests that early or long-term, persistent participation is required to acquire competencies that are multi-usable in other settings (Huang and Chang 2004). Selected students and lottery-admitted students who participate in extracurricular activities during medical school may be different types of students. It might be that selected students always (both before and after admission) look for additional activities, irrespective of their pu-GPA, whereas only the lottery-admitted students with higher pu-GPA—with probably also better pre-clinical achievement (Benbassat and Baumal 2007)—participate in msECAs. Apparently, the personality types represented by the selected students are rated more favourably in the subjective grading in clinical training (Kassebaum and Eaglen 1999). It might be interesting to explore in further studies whether selected students score higher on personality traits such as extraversion and agreeableness that may be beneficial for their future professional practice (Lievens et al. 2002) and as such might also be rated higher by clinical examiners.

The strengths of this study are its large sample size and the long-term follow-up. In addition, the availability of the lottery-admitted students gave us the unique opportunity to compare the participation in msECAs and its relation with clinical achievement for students selected on puECAs with those of randomly admitted students. This enabled us to note the differential effect of pu-GPA on participation in msECA for lottery-admitted and selected students and the differential effect of participation in msECA on clinical achievement.

This study also has some limitations. The number and diversity of msECAs was limited to those that could be reliably measured, i.e. accredited research masters, official board functions and/or an additional degree course and are therefore probably underestimated. Furthermore, time-consuming msECAs were chosen on purpose, since one of the requirements at the selection of medical students was a minimum number of 4 h per week during at least 2 years spent in participation in puECAs before medical school. Only students that passed the first year successfully could apply for a research master, although grades were not a selection criterion for these masters. Additionally, no data was available about participation in puECAs of the lottery-admitted students. However, as we were still able to compare the msECAs for both admission groups, this absence will probably not affect the conclusions of our study. Finally, this study was performed in one medical school. Further replication studies are required to establish whether our results can be generalised to other populations.

The present study has some practical implications for medical school selection procedures. A first implication for medical schools is to include the assessment of puECAs in the selection procedure, since these may predict participation in msECAs in turn leading to better clinical achievement. Apparently using puECAs enables medical schools to attract and select students who are willing and able to continue performing extracurricular activities, and consequently also have a higher chance of better clinical grades. The fact that for selected students participation in msECAs was not related to their pu-GPA suggest that using puECAs in selection enables medical schools to identify those applicants with a lower pu-GPA who have a high chance of good clinical achievement. In other words, the use of puECAs as a selection criterion seems to have additional value to the use of pu-GPA. As an added benefit—contrary to our expectations and those of others—selection on puECAs has recently been shown not to disadvantage non-traditional applicants from minority or lower social-economic backgrounds (Stegers-Jager et al. 2015a), whereas selection on pu-GPA does (Cleland et al. 2012). However, it might still be that self-selection instigated by the use of puECAs as a criterion in the selection procedure is stronger for non-traditional applicants than for traditional applicants at other medical schools. Additionally, as suggested above, there is a risk that when requirements of puECA participation for admission become more transparent, applicants may choose to participate in puECAs because they feel they have to do so to have a chance to enter medical school, and not because they want to [Lucieer et al. 2016 and Higgins’ regulatory focus theory (Higgins 1997, 1998)]. Therefore it would be interesting to search for tools to assess the underlying students traits that lead to persistent msECAs associated with better clinical achievement.

A second practical implication of this study is that medical school should offer sufficient possibilities for extracurricular activities for students, since participation in msECAs may lead to better clinical achievement. Although it might be tempting to strongly stimulate participation of all students, it is—in view of the lack of an effect of msECA participation for lottery-admitted students—questionable whether this will lead to the desired results. As stated above, the effect of msECA participation on clinical achievement may depend on an underlying trait of the participating student, rather than being an effect of the participation itself.

In conclusion, the results of our study suggest that persistent participation in extracurricular activities endorsed better clinical achievement for selected students, supporting the inclusion of ECAs in the selection procedure. More insight in the rationale behind participation in extracurricular activities during medical school may explain differences found between lottery-admitted and selected students.