Advertisement

Tertiary Education and Management

, Volume 25, Issue 4, pp 367–388 | Cite as

How do admission committees select? Do applicants know how they select? Selection criteria and transparency at a Dutch University

  • Anastasia KuryshevaEmail author
  • Harold V. M. van Rijen
  • Gönül Dilaver
Open Access
Original Paper

Abstract

In this study, we investigated the currently applied selective admission criteria and tools of the two-year research master’s programs of both the Graduate Schools of Life Sciences and Natural Sciences of Utrecht University (the Netherlands). In addition, we evaluated their transparency to applicants. Both admissions staff members and applicants participated. To determine admission criteria that are important for admission decisions, we ranked 51 admission criteria and, on their basis, combined into six domains: academic background, grades, cognitive ability, research background, personality and personal competencies, motivation factors. To evaluate transparency, we contrasted the perceptions of applicants with the actual importance of admission criteria, as reported by admission staff members. We found that admissions criteria related to personality and personal competencies are less important in admission decisions than criteria related to grades, academic background and motivation. The applicants find the admissions decisions transparent to a moderate degree. This study also revealed that selectors use criteria and tools both with and without predictive value for later graduate performance. Moreover, some of the currently applied admission instruments might be prone to admission biases. We advocate selectors to use admission criteria and tools that are evidence-based, resistant to admission biases, and transparent to the applicants.

Keywords

Admissions Selection Graduates Applicants Transparency 

Introduction

In the Netherlands, the number of master’s students attending research universities is steadily increasing (Vereniging van Samenwerkende Nederlandse Universiteiten 2018; see Fig. 1). This increase is partly associated with growing numbers of international students, which has almost tripled during the last ten years (Fig. 1). The international attractiveness of Dutch graduate education is not surprising, since all 13 Dutch research universities appear in the top 250 world university rankings (Times Higher Education 2018) and offer numerous English-taught programs, especially at master’s level. Selective admissions are, therefore, becoming both necessary and complex due to the growing numbers of applicants and increased diversification of the application files. Meanwhile, two recent legal changes in the Dutch admission university system - a switch from open admission and lottery systems to selective admissions, and the requirement to fully complete bachelor degree before the start of a master’s program - are both associated with the societal expectations of higher education selective admissions to be fair, transparent and inclusive.
Fig. 1

Numbers of master students at research Dutch universities across the last years. Statistics of the association of research universities in the Netherlands (Vereniging van Samenwerkende Nederlandse Universiteiten, 2018)

Evaluative quality criteria for selection procedures

Patterson and Ferguson (2010) suggested twelve evaluative criteria for selection procedures that should be considered when designing a selection process (Table 1). They draw specific attention to the criteria of validity, reliability, fairness and applicant reactions as those of crucial importance in selection design. Validity refers to how accurate scores on a selection measure predict future outcomes. Reliability refers to consistency of selective admission assessments across different conditions (Cleland et al. 2012). Fairness, in broad terms, refers to absence of biases toward different applicants’ groups, for example, minorities (Tillema et al. 2011). Applicant reactions refer to perceived fairness and clarity of selective admission process. Closely related both to fairness and to applicant reactions is transparency that refers to applicants’ assessment of the selective admission process rather than its content; it indicates the necessity of providing clear and understandable requirements (Tillema et al. 2011). Transparent admissions allow applicants to estimate whether their academic background, prior achievements and non-cognitive attributes are suitable for a program and what they could learn or improve in order to get a chance of admission.
Table 1

Evaluative criteria for selection procedures

Reliability and validity of selection tools

Candidate / applicant reactions

Ease of interpretation

Generality of use

Cost

Practicality / administrative convenience

Legality

Availability of analytical expertise

Fairness

Educational impact

Mechanisms for obtaining feedback

Arrangements for ongoing evaluation and development

Adapted from “Selection for medical education and training” by F. Patterson and E. Ferguson, 2010, Understanding Medical Education: Evidence, Theory and Practice, p. 356. Copyright 2010 by the Association for the Study of Medical Education

Disregarding these fundamental quality criteria in a selective admission process might have a wide range of negative consequences. Namely, using invalid and unreliable selective admission criteria might lead to a higher number of false-positive cases – students, who are admitted to a program, but are unable to complete it (Admissions to Higher Education Steering Group 2004), and a higher number of false-negative cases - rejected applicants, who would have been able to successfully complete the program, but did not get a chance to (Ooijen-van der Linden et al. 2017). Insufficient level of transparency or misleading information about selective admission criteria, documents and processes creates a disadvantage for insufficiently informed applicants such as failing to acquire knowledge and skills that are of importance for a graduate program, failing to realistically assess the likelihood to be accepted into a graduate program (Nauta 2000), and discouraging applicants with low self-confidence from applying (Admissions to Higher Education Steering Group 2004).

A call for more transparency by decentralized selection has been made by policy makers (Adnett et al. 2011; Admissions to Higher Education Steering Group 2004), higher education organizations (e.g. Utrecht University Education Guideline 2017), and by the students themselves (Interstedelijk Studenten Overleg 2014). Following this call, some universities began publishing minimum, median and top study scores (Bagshaw 2016), standardized test scores and background information of their admitted students such as race, ethnicity and geographical origins (La Noue 2003). These actions certainly contribute to articulating clearly to future applicants the kind of students that have a chance to be admitted. However, even if a university implements such policies, what still stays ambiguous to applicants is how the decision-makers actually select applicants who they will then accept (La Noue 2003).

How do admission committees select?

In the context of graduate education, where the admission process is often decentralized, the selective admission decisions are made by the academic staff members who participate in a program’s committee (further referred as admissions committee members). Research across different fields shows that in admissions applications, the admissions committee members are looking for indicators of both academic background and performance on one hand and of personality, motivation and attitudes, on the other.

For academic background and performance, admissions committee members consider important a match or fit between interests and skills of an applicant and what the program offers (Karazsia and McMurtry 2012; Karazsia et al. 2013; Posselt 2016; Walpole et al. 2002); the quality of applicants’ undergraduate experience (Walpole et al. 2002), such as prior institution - where and how an applicant received the undergraduate credits (Kogan et al. 2015), prior degree in a specific field, transcripts, applicant’s work experience, and grade point average (GPA) (MasterMind Europe 2017). In criteria related to personality, motivation and attitudes, admission committee members put attention on applicant’s integrity and honesty (Walpole et al. 2002), communication skills, critical thinking, ethical behavior (Conlon et al. 2012) and personal skills (MasterMind 2017).

The measurement of some of these criteria have been criticized for being prone to biases in selective admissions practice (Kira Talent 2018; Posselt 2016). Namely, when it comes to assessment of personality, motivation, and attitudes, such tools as the interviews, recommendation letters, CVs and motivation letters are often used. Interviews, CVs and motivation letters represent self-reported measures and when applied in a high-stake situation of selective admission, they are susceptible to faking by the applicants (Niessen et al. 2017). When interpreting recommendation letters, a measure which is not self-reported, the admission committee members sometimes indicate a need “to read between the lines” or “to detect an extra glow of enthusiasm about the student” (Walpole et al. 2002 p. 5), which indicates the susceptibility of this measure to biases as well.

The current study

In this study, we are looking into two overarching questions: how do admission committees select? Do applicants know how they select? We first examine the current selective admission criteria applied in practice for research-oriented programs at two large European graduate schools. We discuss whether they satisfy such primary evaluative criteria as evidence on their validity and reliability, and fairness in terms of absence of biases towards certain applicants. We then look at transparency of those criteria towards applicants. The aim of this study is therefore to determine the selective admission criteria that are important for the decision making at research programs, and to investigate whether they are accurately perceived by the applicants.

Context of the study

Utrecht University is a higher education institution which offers research-oriented education in contrast to higher professional education that is offered by universities of applied sciences. A specific focus on research-oriented education means that the main goal of study programs is “independent academic participation or the professional use of academic knowledge” (Nuffic 2015, p. 11). For example, the mission of one of graduate schools participating in this study is “to train future scientists who will be capable of drawing on their acquired knowledge, understanding and skills to conduct top-class research around the world” in academia or industry (Graduate School of Life Sciences 2019). During a two-year master’s programs, a student focuses in a specific field and learns about research methods. A final thesis is an obligatory component of a research-oriented study program.

Two Graduate Schools of Utrecht University were involved in this research: Graduate School of Life Sciences (GSLS) and Graduate School of Natural Sciences (GSNS). The GSLS is organized by three faculties: medicine, veterinary sciences and science (biology, pharmaceutical sciences and chemistry); the GSNS is organized by the faculty of Science (mathematics, physics and information and computing sciences).

Not only has the number students attending Dutch research universities increased (Fig. 1), but also the numbers of applicants. For example, at the GSLS, there were over 2000 applicants in 2018, yet there was only room for 450 students. This brings urgency to evaluate the applied selective admission criteria and methods in terms of their compliance with evaluative quality criteria as described in the literature (Patterson and Ferguson 2010) and requested by the University itself (Utrecht University Education Guideline 2017).

Research questions

The curriculum of the life sciences research master’s programs is internship-focused, while the curriculum of the natural sciences research master’s programs is course-focused. We would expect that the relative importance of admission criteria is different for different types of curricula. Our first research question, therefore, is: What kind of selective admission criteria are reported to be important in decision making at the Life and Natural Sciences graduate schools by staff responsible? Not only the criteria themselves, but also their measurements should meet the evaluation quality criteria. We therefore examine how certain criteria are measured. This brings us to the second research question: What admission tools are most frequently applied? In addition, we aim to empirically analyse transparency of admissions and therefore, our third research question is: Are the applied criteria transparent for the applicants? These questions are tackled in a cross-sectional descriptive study that uses a survey methodology. We also discuss validity, reliability, and fairness of applied criteria based on prior literature in the discussion session.

Method

Procedure and measures

The study was approved by the Dutch Association for Medical Education Ethical Review Board (#969). An invitation to participate in the survey, containing the URL for the online survey, an information letter and a consent form, were sent to the participants by email.

Admission criteria

The 53 possible admission criteria were derived from the graduate schools’ admission web-pages and admission guidelines, scientific literature on criteria that are valued by selection committees worldwide (Conlon et al. 2012; Kogan et al. 2015; MasterMind Europe 2017; Walpole et al. 2002; Weiner 2014), scientific literature on evidence-based factors of student performance (Craig et al. 2004; Hall et al. 2017; Kuncel et al. 2001, 2004; Pacheco et al. 2015; Poropat 2009; Trapmann et al. 2007; von Stumm et al. 2011), and exploratory conversations with the admissions committee members on what they are looking for in admission applications. In the survey, rating of each of the 53 admission criteria was on a 5-point Likert scale (from 1 = not very important to 5 = extremely important) and 0 indicated that the criterion was not used by admissions committee members/was not perceived by applicants as being used. As two of the suggested criteria – “topic of bachelor’s thesis/research project/internship” and “grade for bachelor’s thesis/research project(s)” were often found to be not available at the moment when students apply, we excluded them from the analysis, which left us with a list of 51 admission criteria.

Six coders - researchers in the higher education field and experts in (bio)medical education - were asked to assign 51 criteria into domains. The confirmatory factor analysis with Oblimin rotation delivered a similar but not completely identical solution. We explain this by the small sample size of the program coordinators and relatively small sample size of natural science applicants (for reliable factor analysis, there should be at least four or five times as many observations as there are criteria). Therefore, we report the results based on experts’ categorization. Six domains were established: “academic background”, “grades”, “cognitive ability”, “research background”, “personality and personal competencies” and “motivation factors” (the average percentage of agreement was 66%). Their interrater reliabilities (Cronbach’s alphas) are presented in Table 2. The coders could not assign two criteria, “understanding of social relevance of research” and “qualitative assessments by previous mentors”, and therefore these two criteria were excluded from the univariate analysis on domains.
Table 2

Admission domains: Cronbach’s alphas and selective admission criteria included per domain

Domain

The GSLS Admissions committees’ members

The GSLS Applicants 2017–2018

The GSNS Admissions committees’ members

The GSNS Applicants

Specific criteria included

Academic background (11 criteria)

.51

.66

.70

.72

-Type of prior education institution (a preference for either a Dutch research university/comparable international university, or for university college, or university of applied sciences (HBO)

-Academic writing skills

-Oral and written proficiency in English at an advance level (based in standardized language tests; applicable for international students only)

-Oral and written proficiency in English at an advance level based on additional language assessment (based on interviews/documents)

-Experience in team work

-Correct communication both in writing and orally

-Prior academic background (Biomedical sciences, Biological sciences, etc.)

-Content of previous education matches with Master’s program

-Number of credit points earned for relevant course(s)

-Number of EC (credit points) earned during prior education

-Time to prior degree (i.e. time between starting date and graduation date of a bachelor program)

Grades (4 criteria)

.87

.65

.61

.63

-Undergraduate Grade Point Average (uGPA)

-Increase in grades across the years of prior education

-Cum Laude graduation

-Grades for relevant courses

Cognitive ability (6 criteria)

.89

.77

.87

.79

-General cognitive ability (Intelligence)

-Intellectual curiosity (Intellectual engagement in the pursuit of knowledge)

-Critical thinking, skills, logic, problem-solving skills

-Participation in honors programs

-The ability to understand the scientific method

-Talent

Research background

(5 criteria)

.73

.80

.54

.83

-Previous research experience (amount and quality)

-A clear interest in a multidisciplinary research approach

-Quality of research experience

-The ability to design and carry out research projects

-Mastery of field relevant methods and techniques

Personality and personal competencies

(14 criteria)

.94

.94

.96

.95

-Agreeableness (likability and friendliness)

-Conscientiousness (trustworthiness and will to achieve)

-Emotional stability (emotional adjustment as the opposite to anxiety and neuroticism)

-Extraversion (activity and sociability)

-Openness to experience (imaginativeness, broadmindedness, and artistic sensibility)

-Initiative/willingness to take a challenge

-Interpersonal skills (the ability to communicate or interact well with other people)

-Interests (sports, creative activities etc.)

-Integrity, fairness/honesty

-Maturity/responsibility/work habits

-Leadership qualities

-Independence

-Volunteering experience (amount and quality)

-Time management skills/Timeliness

Motivation factors

(9 criteria)

.89

.82

.87

.88

-Knowledge about program applying for

-Motivation for studying at UU

-Motivation for doing research

-Motivation for doing a Master’s program

-Motivation for this particular program

-The desire to acquire certain skills relevant for this program (e.g. science, laboratory-, business skills)

-Preliminary plans for the Master’s program (e.g. having an idea about the laboratory where a student wants to have an internship)

-Ambitions: what a student wants to achieve with the concrete Master’s program?

-Travelling experience

Cronbach’s alpha was used for reliability estimate of the six domains. Two criteria are not included into these domains due to no agreement between the experts: “Qualitative assessments by previous mentors (recommendation letter writer references/ratings)” and “Understanding of social relevance of research”

Admission documents and tools

The admissions committee members were asked to indicate which documents and methods (transcripts/grade lists, course content, CV, motivation letter, letter(s) of recommendation, interview, English language test, writing sample, personal acquaintance) they use in assessing different groups of admission criteria (such as education background, disciplinary background, cognitive ability, personality, motivation and attitude).

Participants and response

At each Graduate School, admissions committee members and applicants to two-year research master’s programs (2017–2018 and 2018–2019 cohorts) participated. The participants’ characteristics and response rate are presented in the Table 3. Relatively low response rate is associated with rejected applicants not being inclined to respond to the survey from a graduate school that rejected them.
Table 3

Participants’ characteristics and response rate

 

GSLS

GSNS

Admissions committee members

Response rate admission committee members

(N responded admission committee members)

33% (N = 16)

100% (N = 10)

Coverage of programs (N responded Master’s programs)

80% (N = 12)

100% (N = 10)

Applicants

Response rate (N responded applicants)

22% (N = 327)

13% (N = 153)

Coverage of programs

100% (N = 15)

100% (N = 10)

% accepted applicants out of all respondents

89%a

94%

Rejected applicants

11%

6%

Bachelor’s degree at Utrecht University

42% (n = 138)

26% (n = 39)

Bachelor’s degree at another Dutch research university

21% (n = 68)

26% (n = 39)

Bachelor’s degree at a Dutch university college

5% (n = 16)

0

Bachelor’s degree at a Dutch university of applied sciences

4% (n = 12)

9% (n = 14)

Bachelor’s degree at a college or a university outside of the Netherlands

26% (n = 86)

39% (n = 59)

Other type of prior education (e.g. both Bachelor’s and Master’s degree)

2% (n = 7)

0.01% (n = 2)

aNine students responded who were still on the waiting list. However, their data is excluded from the analysis, since they have not undergone the whole selective admission process to that moment

Data analysis

Analysis was conducted in SPSS 25. There were no missing values for the admissions committee members from both graduate schools; 0.2% of values in responses of the GSLS applicants and 0.4% of values in responses of the GSNS applicants were missing completely at random. To handle these missing values, the expectation-maximization (EM) algorithm in SPSS was used. In analysis, first, the descriptive statistics were obtained in order to rank the importance of admission criteria, and the frequencies of admission tools’ usage were obtained. To assess transparency, the importance of admission criteria, as reported by admissions committee members, was contrasted to the importance as perceived by applicants. This was done using Spearman correlation and 2-factorial analysis of variance (ANOVA) (1st factor = admission committees’ members versus applicants; 2nd factor = a domain of admission criteria).

Results

Research question 1: What kind of selective admission criteria are reported to be important in decision-making at the Graduate School of Life Sciences and at the Graduate School of Natural Sciences?

We examined the importance of suggested admission criteria in selective admission decisions on two levels: on the level of specific criteria and on the level of general admission domains. For examination of specific admission criteria’s importance, the means of 51 selective admission criteria were ranked according to their importance in actual admission decisions as reported by admission committee’s members (Table 4).
Table 4

Ranks, Means and Standard Deviations of Selective Admission Criteria: Admissions Committee Members’ Perspectives and Applicants Perceptions at the Graduate Schools of Life Sciences and Natural Sciences

Graduate School of Life Sciences

Graduate School of Natural Sciences

 

Admission committees’ members

Applicants

 

Admission committees’ members

Applicants

Admission criterion

Rank

M (SD)

Rank

M (SD)

Admission criterion

Rank

M (SD)

Rank

M (SD)

Motivation for this particular program

1

4.44 (0.81)

1

4.34 (0.90)

Content of previous education matches with master’s program

1

4.40 (0.70)

3

3.87 (1.10)

Content of previous education matches with master’s program

2

4.31 (0.79)

7

3.73 (0.93)

Prior academic background (Astronomy, Artificial Intelligence, Physics, etc.)

2

4.10 (0.88)

6

3.76 (0.94)

Type of prior educational institution (a preference for either a Dutch research university/comparable international university. or for university college. or university of applied sciences (HBO)

3

4.00 (1.03)

16

3.47 (1.16)

Type of prior educational institution (a preference for either a Dutch research university/comparable international university. or for university college. or university of applied sciences (HBO)

3

3.80 (0.92)

8

3.66 (1.18)

Motivation for doing research

4

3.94 (1.18)

2

4.00 (0.91)

Number of credit points earned for relevant course(s)

4

3.70 (0.67)

16

3.30 (1.32)

Prior academic background (Biomedical sciences. Biological sciences. etc.)

5

3.81 (1.05)

4.5

3.86 (0.83)

Grades for relevant courses

6

3.60 (0.52)

18

3.26 (1.14)

Knowledge about the program applying for

6

3.63 (1.09)

8

3.73 (0.98)

Mastery of field relevant methods and techniques

6

3.60 (1.51)

20

3.16 (1.22)

Preliminary plans for the master's program (e.g. having an idea about the laboratory where the student wants to have an internship)

7

3.56 (1.50)

36.5

2.86 (1.32)

Motivation for this particular program

6

3.60 (1.51)

1

4.20 (0.89)

Undergraduate Grade Point Average (uGPA)

8.5

3.44 (1.26)

24

3.26 (1.04)

Undergraduate Grade Point Average (uGPA)

8

3.50 (1.18)

24

3.03 (1.12)

Ambitions: what a student wants to achieve with the concrete master's program?

8.5

3.44 (1.41)

13

3.54 (1.05)

Oral and written proficiency in English at an advanced level (based on standardized language tests; applicable for international students only)

9

3.40 (1.51)

34

2.73 (1.82)

Motivation for doing a Master's program

10.5

3.38 (1.50)

4.5

3.86 (1.03)

Motivation for doing research

10

3.30 (0.82)

10

3.49 (1.16)

Grades for relevant courses

10.5

3.38 (1.02)

17

3.44 (1.05)

Number of credit points earned during prior education

11.5

3.20 (1.40)

22

3.10 (1.30)

Previous research experience (amount and quality)

12

3.19 (1.52)

20

3.37 (1.26)

Knowledge about the program applying for

11.5

3.20 (0.92)

2

3.90 (0.84)

Oral and written proficiency in English at an advanced level (based on standardized language tests; applicable for international students only)

13.5

3.06 (1.69)

41

2.56 (1.93)

Qualitative assessments by previous mentors (Recommendation letter writer references/ratings)

13

3.00 (1.15)

40

2.42 (1.61)

A clear interest in a multidisciplinary research approach

13.5

3.06 (1.61)

12

3.56 (1.17)

Talent

14.5

2.70 (2.06)

41

2.36 (1.44)

The desire to acquire certain skills relevant for the program (e.g. science-. laboratory- or business- skills)

15.5

3.00 (1.59)

9

3.73 (1.08)

Initiative / willingness to take a challenge

14.5

2.70 (1.49)

9

3.58 (1.26)

Number of credit points earned for relevant course(s)

15.5

3.00 (1.37)

28

3.17 (1.19)

Cum Laude graduation

17

2.60 (2.01)

44

2.15 (1.47)

Intellectual curiosity (Intellectual engagement in the pursuit of knowledge)

17

2.94 (1.95)

3

3.96 (1.08)

Motivation for studying at UU

17

2.60 (1.07)

12

3.44 (1.13)

Correct communication both in writing and orally

19.5

2.81 (1.80)

11

3.63 (1.14)

Time to prior degree (i.e. time between starting date and graduation date of a bachelor program)

17

2.60 (0.84)

39

2.42 (1.22)

Oral and written proficiency in English at an advanced level based on additional language assessment (based on interviews/documents)

19.5

2.81 (1.80)

31

3.09 (1.44)

The desire to acquire certain skills relevant for the program (e.g. science-. laboratory- or business- skills)

21

2.50 (1.51)

14

3.37 (1.33)

Qualitative assessments by previous mentors (Recommendation letter writer references/ratings)

19.5

2.81 (1.38)

36.5

2.86 (1.51)

Motivation for doing a Master's program

21

2.50 (1.65)

4

3.82 (1.02)

Participation in honors programs

19.5

2.81 (1.47)

40

2.58 (1.32)

The ability to understand the scientific method

21

2.50 (2.01)

13

3.39 (1.30)

Motivation for studying at UU

23.5

2.69 (1.58)

14

3.52 (1.13)

Previous research experience (amount and quality)

21

2.50 (1.35)

35

2.69 (1.31)

Mastery of field relevant methods and techniques

23.5

2.69 (1.49)

34

3.01 (1.14)

Increase in grades across the years of prior education

21

2.50 (1.27)

42

2.28 (1.27)

Cum Laude graduation

23.5

2.69 (1.74)

45

2.32 (1.44)

Oral and written proficiency in English at an advanced level based on additional language assessment (based on interviews/documents)

24

2.40 (1.90)

38

2.60 (1.69)

Increase in grades across the years of prior education

23.5

2.69 (1.45)

39

2.60 (1.31)

Ambitions: what a student wants to achieve with the concrete Master's program?

25.5

2.30 (1.16)

11

3.48 (1.18)

Quality of research experience

26

2.63 (1.78)

30

3.12 (1.27)

Academic writings skills

25.5

2.30 (1.95)

26

3.01 (1.34)

Time to prior degree (i.e. time between starting date and graduation date of a bachelor program)

27

2.50 (1.21)

44

2.32 (1.16)

Correct communication both in writing and orally

27

2.20 (1.69)

15

3.33 (1.29)

Initiative / willingness to take a challenge

28.5

2.31 (1.99)

10

3.68 (1.08)

Intellectual curiosity (Intellectual engagement in the pursuit of knowledge)

28

2.10 (2.18)

5

3.79 (1.19)

The ability to understand the scientific method

28.5

2.31 (1.82)

15

3.51 (1.30)

Critical thinking skills, logic, problem-solving skills

29.5

2.00 (2.16)

7

3.67 (1.25)

Conscientiousness (trustworthiness and will to achieve)

30

2.19 (2.04)

18

3.40 (1.28)

The ability to design and carry out research projects

29.5

2.00 (1.41)

28

2.84 (1.49)

Interpersonal skills (the ability to communicate or interact well with other people)

32

2.13 (1.82)

26

3.18 (1.33)

A clear interest in a multidisciplinary research approach

32

1.90 (1.91)

19

3.19 (1.37)

Traveling experience

32

2.13 (1.41)

51

1.70 (1.09)

Participation in honors programs

32

1.90 (1.66)

43

2.20 (1.37)

General cognitive ability (intelligence)

32

2.13 (2.00)

23

3.26 (1.16)

Quality of research experience

32

1.90 (1.73)

37

2.62 (1.46)

Interests (sports. creative activities. etc.)

34

2.06 (1.34)

49

2.18 (1.19)

Independence

34

1.80 (1.69)

30

2.81 (1.51)

Academic writing skills

36

2.00 (1.93)

25

3.24 (1.23)

General cognitive ability (intelligence)

35

1.70 (1.89)

17

3.28 (1.25)

Volunteering experience (amount and quality)

36

2.00 (1.41)

50

2.08 (1.15)

Preliminary plans for the Master's program (e.g. having an idea about the laboratory)

36.5

1.60 (1.35)

29

2.82 (1.41)

Extraversion (activity and sociability)

36

2.00 (1.41)

48

2.23 (1.31)

Maturity/responsibility/work habits

36.5

1.60 (1.71)

21

3.12 (1.50)

Independence

38.5

1.94 (1.84)

26

3.19 (1.48)

Integrity/fairness/honesty

38

1.50 (2.01)

23

3.06 (1.55)

Critical thinking skills. logic. problem-solving skills

38.5

1.94 (2.05)

6

3.80 (1.18)

Conscientiousness (trustworthiness and will to achieve)

39.5

1.40 (1.84)

25

3.02 (1.41)

Understanding of social relevance of research

40.5

1.88 (1.54)

33

3.01 (1.38)

Understanding of social relevance of research

39.5

1.40 (1.43)

33

2.75 (1.47)

Number of EC (credit points) earned during prior education

40.5

1.88 (1.36)

38

2.86 (1.31)

Interpersonal skills (the ability to communicate or interact well with other people)

41.5

1.20 (1.32)

31

2.76 (1.40)

Talent

42.5

1.81 (1.94)

42

2.51 (1.36)

Experience in team work

41.5

1.20 (1.23)

32

2.75 (1.55)

Maturity/responsibility/work habits

42.5

1.81 (1.72)

21

3.34 (1.43)

Volunteering experience (amount and quality)

44

1.10 (0.88)

50

1.56 (1.11)

The ability to design and carry out research projects

44.5

1.75 (1.73)

29

3.17 (1.29)

Openness to experience (imaginativeness, broad-mindedness, and artistic sensibility)

44

1.10 (1.45)

36

2.66 (1.48)

Agreeableness (likability and friendliness)

44.5

1.63 (1.63)

43

2.45 (1.33)

Emotional Stability

44

1.10 (1.60)

44

2.13 (1.50)

Experience in team work

46

1.56 (1.55)

19

3.37 (1.20)

Time management skills / Timeliness

46.5

1.00 (1.33)

27

2.89 (1.61)

Openness to experience (imaginativeness. broad-mindedness. and artistic sensibility)

47

1.44 (1.71)

35

2.98 (1.38)

Leadership qualities

46.5

1.00 (1.70)

48

1.82 (1.33)

Emotional Stability

48

1.38 (1.59)

46

2.28 (1.49)

Travelling experience

49

0.80 (0.63)

51

1.56 (1.14)

Integrity, fairness/honesty

49

1.31 (1.82)

22

3.32 (1.57)

Interests (sports, creative activities, etc.)

49

0.80 (.0.63)

47

1.99 (1.26)

Time management skills / Timeliness

50

1.25 (1.73)

32

3.01 (1.42)

Agreeableness (likability and friendliness)

49

0.80 (1.14)

46

2.03 (1.36)

Leadership qualities

51

1.13 (1.54)

47

2.26 (1.32)

Extraversion (activity and sociability)

51

0.70 (1.16)

49

1.79 (1.28)

On the level of specific criteria

For the GSLS, the most important criteria were related to the domains “motivation factors” (n = 5), “academic background” (n = 3) and “grades” (n = 2). No criteria related specifically to research background, cognitive ability or personality aspects appear amongst the most important. For the GSNS, the highest positions are given to criteria related to “academic background” (n = 5) and “grades” (n = 2). There was also one criterion (“mastery of field relevant methods and techniques”) that is related to research background (#6). In contrast to the GSLS, specific criteria related to motivation also appear in the top-10, but only two of them and not in the highest positions (#6 and #10). In spite of the difference in ranking per criterion, the correlational analysis of the whole list of admission criteria revealed that overall, the importance, which both the GSLS and the GSNS members of admission committees assign to selective admission criteria is similar to a high extent: rPearson (49) = .76, p < .001.

On the level of general admission domains

After examination of specific admission criteria, we turned to a higher order level - exploration of relative importance of general selective admission domains. Their means and standard deviations are presented in Table 5.
Table 5

Means and standard deviations for domains of admission criteria as reported by admission committees’ members and perceived by applicants at the Graduate School of Life Sciences (GSLS) and the Graduate School of Natural Sciences (GSNS)

Domain of admission criteria

GSLS

GSNS

Admissions committees’ members

Applicants

Admissions committees’ members

Applicants

M

SD

M

SD

M

SD

M

SD

Academic background

2.89

0.89

3.21

0.48

3.03

0.97

3.14

0.49

Grades

3.05

0.42

2.90

0.53

3.05

0.58

2.68

0.55

Cognitive ability

2.32

0.46

3.27

0.61

2.15

0.38

3.12

0.68

Research background

2.66

0.56

3.24

0.22

2.38

0.76

2.90

0.26

Personality and personal competencies

1.75

0.39

2.83

0.55

1.27

0.52

2.52

0.62

Motivation factors

3.35

0.68

3.48

0.78

2.49

0.87

3.34

0.77

For the GSLS, a univariate ANOVA revealed that there is a significant large effect of a domain on average importance of criteria, F(5, 43) = 9.02, p < .001, η2 = .51. Hochberg post-hoc examination indicated that the “personality and personal competencies” domain is less important compared to three other domains: “academic background” (p = .001), “grades” (p = .009) and “motivation factors” (p < .001). Also, the domain “motivation factors” is found to be more important than the domain “cognitive ability” (p = .041).

Analysis of the GSNS reports from the admission committee members showed that there is also a significantly large effect of domain on average importance of criteria, F(5, 43) = 8.77, p < .001, η2 = .51. Hochberg post-hoc examination revealed that the “personality and personal competencies” domain is less important compared to three other domains: “academic background” (p < .001), “grades” (p = .001) and “motivation factors” (p = .005).

A two-way factorial ANOVA revealed that there is no significant difference between reports of the GSLS and the GSNS members of admission committees, F(5, 86) = 1.30, p = .271, η2 = .07.

Research question 2: What admission tools are most frequently applied?

At the GSLS, such admission tools as a motivation letter and a CV, an interview, a letter of recommendation and personal acquaintance, if applicable, are all used extensively for assessment of admission criteria. For example, a motivation letter is used for assessment not only of criteria related to motivation itself, but also of criteria related to personality, cognitive ability and previous academic background. Next, there are tools, namely, grade list and course content, that are used in a more focused way. For example, course content is used predominantly for assessment of academic background, but not for assessment of personality and motivation. Finally, a writing sample and an English language test are not frequently used: only a few members of admission committee (25%) indicate that they apply these tools.

At the GSNS, similarly to the GSLS, such admission tools as a motivation letter, a CV and a letter of recommendation are used frequently and for measurement of diverse admission criteria, while a grade list and a course content are also used extensively, but more focused (namely, for measurement of academic background). In contrast to the GSLS, interviews and inferences based on personal acquaintance are not of frequent use at all. Like at the GSLS, at the GSNS, an English language test and a writing sample are not often used for selective admission decisions.

Research question 3: Are the applied criteria transparent for the applicants?

The importance of selective admission criteria at the actual admission decisions and transparency of selective admissions for the applicants were examined on two levels: on the level of specific selective admission criteria and on the level of general admission domains.

For the GSLS, the correlation between the importance of specific admission criteria in decision making as reported by admission committees’ members and the perceived importance of the same criteria by the applicants, is of a moderate-to-large size, rPearson (51) = .53, p < .001. Two-way analysis of variance on admission domains revealed that there is significant interaction effect between the respondent group (admission committee members versus applicants) and the type of domain (academic background, grades, cognitive ability, research background, personality and personal competencies, and motivation factors) on their importance, F(5, 86) = 2.51, p = .036, η2 = .13. As can be seen from Fig. 2, the extent to which the GSLS applicants perceive criteria related to academic background, grades, research background and motivation to be important is in coherence with reports from admission committee members. The GSLS applicants overestimate the importance of criteria related to cognitive ability and personality.
Fig. 2

Mean importance of admission criteria domains in selective admission decisions with 95% confidence intervals, reported by the GSLS admission committees’ members (n = 16) and perceived by the GSLS applicants (n = 327)

For the GSNS, similar results were obtained. Reports of admission committees’ members on importance of specific admission criteria are correlated with applicants’ perceived importance to a large extent, rPearson (51) = .63, p < .001. On the level of admission domains, two-way ANOVA revealed a significant interaction effect: the applicants perceive the importance of admission criteria domains differently from what the admission committees’ members report, F(5, 86) = 2.98, p = .016, η2 = .15. Figure 3 shows that four domains – academic background, grades, research background and motivation - were perceived by the GSNS applicants to have similar importance in admission decisions as was reported by the admission committee members. Like their GSLS peers, the GSNS applicants overestimate the importance of criteria related to cognitive ability and personality.
Fig. 3

Mean importance of admission criteria domains in selective admission decisions with 95% confidence intervals, reported by the GSNS admission committees’ members (n = 10) and perceived by the GSNS applicants (n = 153)

Discussion

In this study, we aimed to answer two central questions: How do admission committees of Life and Natural Sciences research Master’s programs select? Do applicants of two particular Life and Natural Graduate Schools know how the admission committee select? We examined what kind of specific selective admission criteria and admission domains are important in admission decisions at the Life and Natural Sciences Graduate Schools. We found that criteria related to applicants’ academic background, grades, motivation factors, but also cognitive ability and research background, played an important role in admission decisions. We also examined what kind of admissions tools are used the most. We found that an unstructured motivation letter, a CV and a grade list are the most frequently used tools, but other tools are used for measurement of various criteria as well. Finally, we investigated how accurate do applicants, who have just undergone the admission process, perceive the importance of admission criteria in decisions regarding their application. We found that applicants’ perceptions were accurate to a moderate degree. At both graduate schools, criteria related to academic background, research background, grades, cognitive ability and motivation factors were reported by admission committee members to be important, but not the criteria related to personality and personal competencies. Below we discuss the evidence on the criteria and related admission tools per admission domain.

Academic background

Our findings suggest that a match between content of previous education and a master’s program is by far one of the most important criteria in admission decisions (rank #2 at the GSLS and rank #1 at the GSNS). This finding is consistent with other studies (Karazsia and McMurtry 2012; Karazsia et al. 2013; Posselt 2016; Walpole et al. 2002). Also, we find that criteria related to quality of undergraduate experience (a prior degree in a specific field, type of prior educational institution, grades) are reported to be important in admission decisions, which is also in line with existing literature (MasterMind Europe 2017; Kogan et al. 2015). To the best of our knowledge, there are no comprehensive studies on association between previous academic background (such as type of previous institution etc.) and graduate academic achievements. Nevertheless, it seems plausible that prior academic background in a certain discipline (ranked #5 by the GSLS and #2 by the GSNS) provides an advantage for admission in a program within a similar discipline. However, the high rankings of a criterion “Type of prior educational institution” (a preference for either a Dutch research university/comparable international university or for university college or university of applied sciences (HBO) - rank #3 at both graduate schools) raises concerns, since giving a decision power to this criterion might place applicants from certain type of institutions in a dis/advantageous position. In order to account for differences in acquired level of knowledge and skills at different types of higher education institutions, one possible solution could be to provide an opportunity for the applicants to deliver the results of validated graduate standardized tests as evidence of their abilities, necessary for starting education at a graduate level, as there is evidence that standardized tests are valid predictors for graduate student study success (e.g. Kuncel et al. 2001; Oh et al. 2008)

Grades

Both on a level of specific criteria and on a level of more general domains, we find that criteria related to grades play an important role in admission decisions in both graduate schools. For instance, undergraduate GPA and grades for relevant courses are in top-10 at both Life and Natural Sciences Graduate Schools. On the level of a more general domains, grades are shown to be of a moderate importance in admission decisions. Importance of grades in admission decisions at both GSLS and GSNS might be regarded as a good practice, as prior findings indicate that undergraduate GPA, and last-year GPA in particular, are valid predictors of a number of future performance indicators (Kuncel et al. 2001; Zimmermann 2016).

Cognitive ability

At both GSLS and GSNS, criteria related to cognitive ability are reported to be of a slight-to-moderate importance, and are less important than motivation, academic background, research background and grades. Inference about cognitive ability is done based on different admission tools (including motivation and recommendation letter, interviews, personal acquaintance). A variety of tools for inferring information about cognitive ability of an applicant raises concerns, since some of those admission tools firstly, were not developed for cognitive ability measurement, and secondly, have serious issues with reliability and validity indicators (Patterson et al. 2016). The tests that are primarily developed for measurement aptitude and cognitive ability (e.g. Miller Analogies Test, Graduate Record Examinations, Graduate Management Admission Test) are not required at the examined programs.

Research background

Not surprisingly, the current study also finds that criteria related to research experience play a role in admission decisions. There are only two studies in the life sciences field that investigated the predictive validity of amount of previous research experience and their findings are contradictory: while in one study an association between amount of prior research experience and research productivity was found (Weiner 2014), no similar association was found in another study that tried to replicate the findings of the first one (Hall et al. 2017). While it is plausible to assume that previous research experience is associated with academic success at research-oriented programs, at the moment the existing evidence is insufficient for using it as a factor with decisive power. We therefore suggest to exercise caution in its usage in admissions practice.

Motivation factors

We find that motivation factors play an important role in admission decisions, especially at the GSLS and to a lesser extent at the GSNS. Namely, at the GSLS, six out of top 10 criteria were related to motivation, and at the GSNS, two out of top 10. When we look at the motivation factors as a general domain, we see that motivation factors are a bit more than moderately important at the GSLS, and slightly-to-moderate important at the GSNS. Finally, criteria related to motivation factors were reported to be measured predominantly by unstructured motivation letters, CV, interviews and unstructured recommendation letters. While some motivational aspects were indeed shown to have associations with academic performance (e.g. Robbins et al. 2004), operationalization of motivation in admissions raises serious concerns, as three indicated admission tools are self-reported measures.

Researchers clearly call against using self-report measures in student selection (Murphy et al. 2009; Niessen et al. 2017; Patterson et al. 2016; Wouters 2016). It was demonstrated that self-reported measures induce a self-presentation effect, or, in other words, applicants tend to fake their reports in the selection context, which represent a high-stake situation (Niessen et al. 2017). Also, these measures have a small predictive value towards future academic performance and do not provide sufficient incremental validity in addition to prior grades and results of standardized admission tests (Goho and Blackman 2006; Hell et al. 2007; Murphy et al. 2009). As for the measure that is also extensively used for assessment of applicant’s motivation and is not self-reported – a recommendation letter - the scientific evidence exists for its practical value only in its standardized form (Kuncel et al. 2014).

Personality and personal competencies

Low importance of personality factors for admission decisions is not in line with prior indications of the importance of personality and personal competencies in student selection (Conlon et al. 2012; Walpole et al. 2002). At least two possible explanations arise. The first one might be related to the design of our study since we used self-reports of admission committee members and therefore the answers could be socially desirable. Other methods (e.g. observations) could shed light on possible importance of applicant’s personality during the admission process that may have been avoided in a self-report. Another possible explanation may be that for research-oriented master’s programs, the applicant’s personality indeed does not play an important role in decision making regarding their admission. An interview study on selective admissions to undergraduate programs supports this explanation in the context of American and English elite universities. The study found that admission on academic record alone is typical for English universities and is an exception at the American universities, where much more attention is given to personal attributes of applicants (Mountford Zimdars 2016). Also, the survey study of the MasterMind project showed that only one-third of European programs consider personal skills in admission decisions (MasterMind Europe 2017). It would not be surprising if the admission practices in the Netherlands are more in line with other European countries’ practices rather than with American ones, and personality and personal competencies do not play a crucial role in admissions.

A limited usage of personality traits and competencies in admission decisions might be regarded as a good practice. One of the most stable findings is that a personality trait such as conscientiousness is associated with academic performance (Poropat 2009; Schneider and Preckel 2017; Trapmann et al. 2007), but the predictive validity of other personality traits - openness to experience, agreeableness, neuroticism and extraversion - range from negligible to small, with overall conclusion that they do not have a substantial impact on academic performance. One may argue that results of conscientiousness might be used in admissions, but since personality testing is a self-report measure, which is susceptible for faking (Niessen et al. 2017), we would warn against such a practice.

Are the applied admission criteria transparent for applicants?

We explored whether the applicants, who already know their admission decision, accurately perceive the criteria based on which the decision regarding their applications was made. Our results show that although transparency is of a moderate to large extent on the level of specific criteria, on the level of general academic domains it is not the case. Applicants perceive accurately certain domains (academic background, grades, research background and motivation factors), but not others (cognitive ability and personality are overestimated in terms of their importance in admission decisions by applicants of both graduate schools).

These results suggest that some of information that is presented to applicants (e.g. at the admission web-pages) provides a clear description of a criterion (e.g. minimum scores for English test, or a specific list of courses, which an applicant had to follow during previous education), but other information might be unclear (e.g. such criteria as “talent” or “high level of academic and professional ability”). Along with that, it might also be that applicants do not use enough information sources (such as programs’ webinars, Master’s Open Days, communication with a program coordinator via email), in order to get informed about criteria, based on which their application will be assessed. As one of the applicants commented: “I did not find much concrete accessible information on the selection criteria, but I did not look very intensely”.

Strength, limitations, further directions and conclusions

Among the strengths of this study are the availability of data from (a) both admission committee members and applicants and from (b) two graduate schools with different scope and different curricula designs. The information from admission committees’ members allowed us to describe the actual admission practices, and information from the applicants allowed us to assess whether these practices are clear to those for whom they are designed. Next, we used a comprehensive list of specific admission criteria, which allowed us to conduct analysis both on a level of specific criteria and on a level of domains – when several similar criteria were considered together. With this approach, we were able to capture the specific trends in applicants’ estimation of admission criteria importance. The fact that for both graduate schools we found either correct estimation or overestimation in the same admission criteria domains speaks for a possible generalizability of our results to research master’s programs. Furthermore, to the best of our knowledge, this is only the second study in the European context that explored admission practices both from admission committee members’ and applicants’ views (see MasterMind Europe 2017 for the survey across Europe), but the first one that directly contrasted perceived importance against actual practices of selective admissions.

Despite these strengths, the consideration of some limitations should be made when interpreting the findings. Firstly, a self-report by admission committee members leaves a possibility that the reported importance of certain admission criteria does not completely correspond with their real importance in the daily practice of selective admission decision making. Nonetheless, we assume that the staff members’ reports closely resemble their practice. Secondly, in the third part of the study, dedicated to transparency of selective admission criteria, most of the respondents were accepted applicants. Therefore, the results might have been affected by a selection bias: the accepted applicants might be those who had a chance to learn about the “unspoken” admission rules in contrast to students who did not have a chance to learn such implicit information from their environment. Thirdly, the study is conducted within one higher education institution that could narrow down the generalizability. Yet the study involves two large graduate schools, one of which is interfaculty, and multiple master’s programs within each of them, so we expect the results to be relevant for (life and natural) science programs more generally.

In summary, this study contributes to our knowledge on what admission committee members are looking for in future graduate students who apply for a research master’s program. As a practical implication, the study suggests that the quality of admissions should be a subject of evaluation and constant improvement in terms of scientific evidence, fairness and transparency. The results are relevant for applicants, researchers in higher education, in particular with a focus on admissions quality, and for professionals, who are involved in the (daily) practice of selective admissions in higher education institutions.

Notes

Acknowledgments

The authors would like to thank prof. dr. Marijk van der Wende for her supervision of the overarching PhD project focused on graduate selective admissions. The authors also would like to thank dr. ir. J.B.A. (Twan) Maintz for organizing the survey distribution at the Utrecht University Graduate School of Natural Sciences.

Funding information

This study had internal institutional funding.

References

  1. Admissions to Higher Education Steering Group. (2004). Fair admissions to higher education: Recommendations for good practice. London: Schwartz S. https://www.spa.ac.uk/resources/schwartz-report. Accessed 28 March 2019.
  2. Adnett, N., McCaig, C., Slack, K., & Bowers-Brown, T. (2011). Achieving “transparency, consistency and fairness” in English HE admissions: Progress since Schwartz? Higher Education Quarterly, 65, 12–33.  https://doi.org/10.1111/j.1468-2273.2010.00468.x.CrossRefGoogle Scholar
  3. Bagshaw, E. (2016, September 19). Sydney university reveals real ATAR admissions scores. The Sydney Morning Herald. https://www.smh.com.au/education/sydney-university-reveals-atar-admissions-scores-20160919-grjjie.html. Accessed 3 September 2019.
  4. Cleland, J. A., Dowell, J., McLachlan, J., Nicholson, S., and Patterson, F. (2012). Identifying best practice in the selection of medical students: Literature review and interview survey (Research Report). https://www.gmc-uk.org/about/what-we-do-and-why/data-and-research/research-and-insight-archive/identifying-best-practice-in-the-selection-of-medical-students. Accessed 27 March 2019.
  5. Conlon, P., Hecker, K., & Sabatini, S. (2012). What should we be selecting for? A systematic approach for determining which personal characteristics to assess for during admissions. BMC Medical Education, 12, 1–10.  https://doi.org/10.1186/1472-6920-12-105.CrossRefGoogle Scholar
  6. Craig, P. L., Gordon, J. J., Clark, R. M., & Langendyk, V. (2004). Prior academic background and student performance in assessment in a graduate entry programme. Medical Education, 38, 1164–1168.  https://doi.org/10.1111/j.1365-2929.2004.02043.x.CrossRefGoogle Scholar
  7. Goho, J., & Blackman, A. (2006). The effectiveness of academic admission interviews: An exploratory meta-analysis. Medical Teacher, 28, 335–340.  https://doi.org/10.1080/01421590600603418.CrossRefGoogle Scholar
  8. Graduate School of Life Sciences (2019). Mission of the GSLS. Retrieved from https://www.uu.nl/en/education/graduate-school-of-life-sciences/about-us/mission-of-the-gsls. Accessed 22 May 2019.
  9. Hall, J. D., O’Connell, A. B., & Cook, J. G. (2017). Predictors of student productivity in biomedical graduate school applications. PLoS One, 12(1), e0169121.  https://doi.org/10.1371/journal.pone.0169121.CrossRefGoogle Scholar
  10. Hell, B., Trapmann, S., Weigand, S., and Schuler, H. (2007). Die Validität von Auswahlgesprächen im Rahmen der Hochschulzulassung - eine Metaanalyse. [Abstract] Psychologische Rundschau, 58 93.  https://doi.org/10.1026/0033-3042.58.2.93.CrossRefGoogle Scholar
  11. Interstedelijk Studenten Overleg. (2014). Meer transparantie bij decentrale selectie. Het belang en een framework. https://www.iso.nl/website/wp-content/uploads/2014/12/1415-meer-transparantie-bij-decentrale-selectie3.pdf. Accessed 27 March 2019.
  12. Karazsia, B. T., & McMurtry, C. M. (2012). Graduate admissions in pediatric psychology: The importance of undergraduate training. Journal of Pediatric Psychology, 37, 127–135.  https://doi.org/10.1093/jpepsy/jsr067.CrossRefGoogle Scholar
  13. Karazsia, B. T., Stavnezer, A. J., & Reeves, J. W. (2013). Graduate admissions in clinical neuropsychology: The importance of undergraduate training. Archives of Clinical Neuropsychology, 28, 711–720.  https://doi.org/10.1093/arclin/act056.CrossRefGoogle Scholar
  14. Kogan, L. R., Stewart, S. M., Schoenfeld-Tacher, R., and Hellyer, P. W. (2015). Perceptions of veterinary admissions committee members of undergraduate credits earned from community colleges or online compared to traditional 4-year institutions. Open Veterinary Journal, 5, 71–84. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4629565/. Accessed 27 March 2019.
  15. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the graduate record examinations: Implications for graduate student selection and performance. Psychological Bulletin, 127, 162–181.  https://doi.org/10.1037/0033-2909.127.1.162.CrossRefGoogle Scholar
  16. Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2004). Academic performance, career potential, creativity, and job performance: Can one construct predict them all? Journal of Personality and Social Psychology, 86, 148–161.  https://doi.org/10.1037/0022-3514.86.1.148.CrossRefGoogle Scholar
  17. Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A meta-analysis of letters of recommendation in college and graduate admissions: Reasons for hope. International Journal of Selection and Assessment, 22, 101–107.  https://doi.org/10.1111/ijsa.12060.CrossRefGoogle Scholar
  18. la Noue, G. R. (2003). The new imperative for admissions transparency. Academic Questions, 16(3), 23–29.  https://doi.org/10.1007/s12129-003-1076-9.CrossRefGoogle Scholar
  19. MasterMind Europe. (2017). Admissions to English-taught programs (ETPs) at Master’s level in Europe – Procedures, regulations, success rates and challenges for diverse applicants. Amsterdam: ACA, StudyPortals, and Vrije Universiteit Amsterdam.Google Scholar
  20. Mountford Zimdars, A. (2016). Meritocracy and the university: Selective admission in England and the United States. London; New York: Bloomsbury Academic.CrossRefGoogle Scholar
  21. Murphy, S. C., Klieger, D. M., Borneman, M. J., & Kuncel, N. R. (2009). The predictive power of personal statements in academic admissions: A meta-analysis and cautionary tale. [Abstract]. College and University, 84, 83.Google Scholar
  22. Nauta, M. M. (2000). Assessing the accuracy of psychology undergraduates’ perceptions of graduate admission criteria. Teaching of Psychology, 27(4), 277–280.Google Scholar
  23. Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2017). Measuring non-cognitive predictors in high-stakes contexts: The effect of self-presentation on self-report instruments used in admission to higher education. Personality and Individual Differences, 106, 183–189.  https://doi.org/10.1016/j.paid.2016.11.014.CrossRefGoogle Scholar
  24. Nuffic (2015). Education system in the Netherlands. www.nuffic.nl/documents/459/education-system-the-netherlands.pdf. Accessed 22 May 2019.
  25. Oh, I., Schmidt, F. L., & Shaffer, J. A. (2008). Admission test (GMAT) is even more valid than we thought: A new development in Meta- analysis and its implications for the validity of the GMAT. Academy of Management Learning and Education, 7, 563–570.  https://doi.org/10.5465/amle.2008.35882196.CrossRefGoogle Scholar
  26. Pacheco, W. I., Noel, R. J., Porter, J. T., & Appleyard, C. B. (2015). Beyond the GRE: Using a composite score to predict the success of Puerto Rican students in a biomedical PhD program. CBE Life Sciences Education, 14(2), ar13.  https://doi.org/10.1187/cbe.14-11-0216.CrossRefGoogle Scholar
  27. Patterson, F., & Ferguson, E. (2010). Selection for medical education and training. In T. Swanwick (Ed.), Understanding medical Education: Evidence, theory and practice (pp. 352–365).  https://doi.org/10.1002/9781444320282.ch24.CrossRefGoogle Scholar
  28. Patterson, F., Knight, A., Dowell, J., Nicholson, S., Cousans, F., & Cleland, J. (2016). How effective are selection methods in medical education? A systematic review. Medical Education, 50, 36–60.  https://doi.org/10.1111/medu.12817.CrossRefGoogle Scholar
  29. Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, 322–338.  https://doi.org/10.1037/a0014996.CrossRefGoogle Scholar
  30. Posselt, J. R. (2016). Inside graduate admissions: Merit, diversity, and faculty gatekeeping. Cambridge and London: Harvard University Press.CrossRefGoogle Scholar
  31. Robbins, S. B., Lauver, K., Le, H., Davis, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A Meta-analysis. Psychological Bulletin, 130, 261–288.  https://doi.org/10.1037/0033-2909.130.2.261.CrossRefGoogle Scholar
  32. Schneider, M., & Preckel, F. (2017). Variables associated with achievement in higher education: A systematic review of meta-analyses. Psychological Bulletin, 143, 565–600.  https://doi.org/10.1037/bul0000098.CrossRefGoogle Scholar
  33. Kira Talent (2018). Breaking down bias in admissions: The how-to guide to preventing admissions bias at your school. http://start.kiratalent.com/breaking-down-admissions-bias/ Accessed 27 March 2019.
  34. Tillema, H., Leenknecht, M., & Segers, M. (2011). Assessing assessment quality: Criteria for quality assurance in design of (peer) assessment for learning - a review of research studies. Studies in Educational Evaluation, 37, 25–34.  https://doi.org/10.1016/j.stueduc.2011.03.004.CrossRefGoogle Scholar
  35. Times Higher Education (2018, October 1). Best universities in the Netherlands 2019. https://www.timeshighereducation.com/student/best-universities/best-universities-netherlands accessed 27 March 2019.
  36. Trapmann, S., Hell, B., Hirn, J.-O. W., & Schuler, H. (2007). Meta-analysis of the relationship between the big five and academic success at university. Zeitschrift Für Psychologie / Journal of Psychology, 215, 132–151.  https://doi.org/10.1027/0044-3409.215.2.132.CrossRefGoogle Scholar
  37. Utrecht University. Education Guideline (2017). https://www.uu.nl/en/education/education-at-uu/the-educational-model. Accessed 27 March 2019.
  38. van Ooijen-van der Linden, L., van der Smagt, M. J., Woertman, L., & te Pas, S. F. (2017). Signal detection theory as a tool for successful student selection. Assessment and Evaluation in Higher Education, 42, 1194–1207.  https://doi.org/10.1080/02602938.2016.1241860.CrossRefGoogle Scholar
  39. Vereniging van Samenwerkende Nederlandse Universiteiten [The Association of Research Universities in the Netherlands] (2018). [Graph illustrations Aantal ingeschreven studenten and Internationale studenten]. Ingeschreven studenten. https://www.vsnu.nl/f_c_ingeschreven_studenten.html. Accessed 02 January 2019. 
  40. von Stumm, S., Hell, B., & Chamorro-Premuzic, T. (2011). The hungry mind: Intellectual curiosity is the third pillar of academic performance. Perspectives on Psychological Science, 6, 574–588.  https://doi.org/10.1177/1745691611421204.CrossRefGoogle Scholar
  41. Walpole, M., Burton, N. W., Kanyi, K., & Jackenthal, A. (2002). Selecting successful graduate students: in-depth interviews with GRE ® users. (report no. GREB-99-11R, RR-02-08). Princeton: Educational Testing Service.  https://doi.org/10.1002/j.2333-8504.2002.tb01875.x.CrossRefGoogle Scholar
  42. Weiner, O. D. (2014). How should we be selecting our graduate students? Molecular Biology of the Cell, 25, 429–430.  https://doi.org/10.1091/mbc.E13-11-0646.CrossRefGoogle Scholar
  43. Wouters, A. (2016). Effects of medical school selection on the motivation of the student population and applicant pool (Doctoral dissertation). http://dare.ubvu.vu.nl/handle/1871/55083. Accessed 27 March 2019.
  44. Zimmermann, J. (2016). Information Processing for Effective and Stable Admission (doctoral dissertation). https://www.research-collection.ethz.ch/handle/20.500.11850/117220. Accessed 27 March 2019.

Copyright information

© The Author(s) 2019

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Graduate School of Life SciencesUtrecht University and Center of Education and Training, University Medical Center UtrechtUtrechtThe Netherlands

Personalised recommendations