Introduction

In the Netherlands, the number of master’s students attending research universities is steadily increasing (Vereniging van Samenwerkende Nederlandse Universiteiten 2018; see Fig. 1). This increase is partly associated with growing numbers of international students, which has almost tripled during the last ten years (Fig. 1). The international attractiveness of Dutch graduate education is not surprising, since all 13 Dutch research universities appear in the top 250 world university rankings (Times Higher Education 2018) and offer numerous English-taught programs, especially at master’s level. Selective admissions are, therefore, becoming both necessary and complex due to the growing numbers of applicants and increased diversification of the application files. Meanwhile, two recent legal changes in the Dutch admission university system - a switch from open admission and lottery systems to selective admissions, and the requirement to fully complete bachelor degree before the start of a master’s program - are both associated with the societal expectations of higher education selective admissions to be fair, transparent and inclusive.

Fig. 1
figure 1

Numbers of master students at research Dutch universities across the last years. Statistics of the association of research universities in the Netherlands (Vereniging van Samenwerkende Nederlandse Universiteiten, 2018)

Full size image

Evaluative quality criteria for selection procedures

Patterson and Ferguson (2010) suggested twelve evaluative criteria for selection procedures that should be considered when designing a selection process (Table 1). They draw specific attention to the criteria of validity, reliability, fairness and applicant reactions as those of crucial importance in selection design. Validity refers to how accurate scores on a selection measure predict future outcomes. Reliability refers to consistency of selective admission assessments across different conditions (Cleland et al. 2012). Fairness, in broad terms, refers to absence of biases toward different applicants’ groups, for example, minorities (Tillema et al. 2011). Applicant reactions refer to perceived fairness and clarity of selective admission process. Closely related both to fairness and to applicant reactions is transparency that refers to applicants’ assessment of the selective admission process rather than its content; it indicates the necessity of providing clear and understandable requirements (Tillema et al. 2011). Transparent admissions allow applicants to estimate whether their academic background, prior achievements and non-cognitive attributes are suitable for a program and what they could learn or improve in order to get a chance of admission.

Table 1 Evaluative criteria for selection procedures
Full size table

Disregarding these fundamental quality criteria in a selective admission process might have a wide range of negative consequences. Namely, using invalid and unreliable selective admission criteria might lead to a higher number of false-positive cases – students, who are admitted to a program, but are unable to complete it (Admissions to Higher Education Steering Group 2004), and a higher number of false-negative cases - rejected applicants, who would have been able to successfully complete the program, but did not get a chance to (Ooijen-van der Linden et al. 2017). Insufficient level of transparency or misleading information about selective admission criteria, documents and processes creates a disadvantage for insufficiently informed applicants such as failing to acquire knowledge and skills that are of importance for a graduate program, failing to realistically assess the likelihood to be accepted into a graduate program (Nauta 2000), and discouraging applicants with low self-confidence from applying (Admissions to Higher Education Steering Group 2004).

A call for more transparency by decentralized selection has been made by policy makers (Adnett et al. 2011; Admissions to Higher Education Steering Group 2004), higher education organizations (e.g. Utrecht University Education Guideline 2017), and by the students themselves (Interstedelijk Studenten Overleg 2014). Following this call, some universities began publishing minimum, median and top study scores (Bagshaw 2016), standardized test scores and background information of their admitted students such as race, ethnicity and geographical origins (La Noue 2003). These actions certainly contribute to articulating clearly to future applicants the kind of students that have a chance to be admitted. However, even if a university implements such policies, what still stays ambiguous to applicants is how the decision-makers actually select applicants who they will then accept (La Noue 2003).

How do admission committees select?

In the context of graduate education, where the admission process is often decentralized, the selective admission decisions are made by the academic staff members who participate in a program’s committee (further referred as admissions committee members). Research across different fields shows that in admissions applications, the admissions committee members are looking for indicators of both academic background and performance on one hand and of personality, motivation and attitudes, on the other.

For academic background and performance, admissions committee members consider important a match or fit between interests and skills of an applicant and what the program offers (Karazsia and McMurtry 2012; Karazsia et al. 2013; Posselt 2016; Walpole et al. 2002); the quality of applicants’ undergraduate experience (Walpole et al. 2002), such as prior institution - where and how an applicant received the undergraduate credits (Kogan et al. 2015), prior degree in a specific field, transcripts, applicant’s work experience, and grade point average (GPA) (MasterMind Europe 2017). In criteria related to personality, motivation and attitudes, admission committee members put attention on applicant’s integrity and honesty (Walpole et al. 2002), communication skills, critical thinking, ethical behavior (Conlon et al. 2012) and personal skills (MasterMind 2017).

The measurement of some of these criteria have been criticized for being prone to biases in selective admissions practice (Kira Talent 2018; Posselt 2016). Namely, when it comes to assessment of personality, motivation, and attitudes, such tools as the interviews, recommendation letters, CVs and motivation letters are often used. Interviews, CVs and motivation letters represent self-reported measures and when applied in a high-stake situation of selective admission, they are susceptible to faking by the applicants (Niessen et al. 2017). When interpreting recommendation letters, a measure which is not self-reported, the admission committee members sometimes indicate a need “to read between the lines” or “to detect an extra glow of enthusiasm about the student” (Walpole et al. 2002 p. 5), which indicates the susceptibility of this measure to biases as well.

The current study

In this study, we are looking into two overarching questions: how do admission committees select? Do applicants know how they select? We first examine the current selective admission criteria applied in practice for research-oriented programs at two large European graduate schools. We discuss whether they satisfy such primary evaluative criteria as evidence on their validity and reliability, and fairness in terms of absence of biases towards certain applicants. We then look at transparency of those criteria towards applicants. The aim of this study is therefore to determine the selective admission criteria that are important for the decision making at research programs, and to investigate whether they are accurately perceived by the applicants.

Context of the study

Utrecht University is a higher education institution which offers research-oriented education in contrast to higher professional education that is offered by universities of applied sciences. A specific focus on research-oriented education means that the main goal of study programs is “independent academic participation or the professional use of academic knowledge” (Nuffic 2015, p. 11). For example, the mission of one of graduate schools participating in this study is “to train future scientists who will be capable of drawing on their acquired knowledge, understanding and skills to conduct top-class research around the world” in academia or industry (Graduate School of Life Sciences 2019). During a two-year master’s programs, a student focuses in a specific field and learns about research methods. A final thesis is an obligatory component of a research-oriented study program.

Two Graduate Schools of Utrecht University were involved in this research: Graduate School of Life Sciences (GSLS) and Graduate School of Natural Sciences (GSNS). The GSLS is organized by three faculties: medicine, veterinary sciences and science (biology, pharmaceutical sciences and chemistry); the GSNS is organized by the faculty of Science (mathematics, physics and information and computing sciences).

Not only has the number students attending Dutch research universities increased (Fig. 1), but also the numbers of applicants. For example, at the GSLS, there were over 2000 applicants in 2018, yet there was only room for 450 students. This brings urgency to evaluate the applied selective admission criteria and methods in terms of their compliance with evaluative quality criteria as described in the literature (Patterson and Ferguson 2010) and requested by the University itself (Utrecht University Education Guideline 2017).

Research questions

The curriculum of the life sciences research master’s programs is internship-focused, while the curriculum of the natural sciences research master’s programs is course-focused. We would expect that the relative importance of admission criteria is different for different types of curricula. Our first research question, therefore, is: What kind of selective admission criteria are reported to be important in decision making at the Life and Natural Sciences graduate schools by staff responsible? Not only the criteria themselves, but also their measurements should meet the evaluation quality criteria. We therefore examine how certain criteria are measured. This brings us to the second research question: What admission tools are most frequently applied? In addition, we aim to empirically analyse transparency of admissions and therefore, our third research question is: Are the applied criteria transparent for the applicants? These questions are tackled in a cross-sectional descriptive study that uses a survey methodology. We also discuss validity, reliability, and fairness of applied criteria based on prior literature in the discussion session.

Method

Procedure and measures

The study was approved by the Dutch Association for Medical Education Ethical Review Board (#969). An invitation to participate in the survey, containing the URL for the online survey, an information letter and a consent form, were sent to the participants by email.

Admission criteria

The 53 possible admission criteria were derived from the graduate schools’ admission web-pages and admission guidelines, scientific literature on criteria that are valued by selection committees worldwide (Conlon et al. 2012; Kogan et al. 2015; MasterMind Europe 2017; Walpole et al. 2002; Weiner 2014), scientific literature on evidence-based factors of student performance (Craig et al. 2004; Hall et al. 2017; Kuncel et al. 2001, 2004; Pacheco et al. 2015; Poropat 2009; Trapmann et al. 2007; von Stumm et al. 2011), and exploratory conversations with the admissions committee members on what they are looking for in admission applications. In the survey, rating of each of the 53 admission criteria was on a 5-point Likert scale (from 1 = not very important to 5 = extremely important) and 0 indicated that the criterion was not used by admissions committee members/was not perceived by applicants as being used. As two of the suggested criteria – “topic of bachelor’s thesis/research project/internship” and “grade for bachelor’s thesis/research project(s)” were often found to be not available at the moment when students apply, we excluded them from the analysis, which left us with a list of 51 admission criteria.

Six coders - researchers in the higher education field and experts in (bio)medical education - were asked to assign 51 criteria into domains. The confirmatory factor analysis with Oblimin rotation delivered a similar but not completely identical solution. We explain this by the small sample size of the program coordinators and relatively small sample size of natural science applicants (for reliable factor analysis, there should be at least four or five times as many observations as there are criteria). Therefore, we report the results based on experts’ categorization. Six domains were established: “academic background”, “grades”, “cognitive ability”, “research background”, “personality and personal competencies” and “motivation factors” (the average percentage of agreement was 66%). Their interrater reliabilities (Cronbach’s alphas) are presented in Table 2. The coders could not assign two criteria, “understanding of social relevance of research” and “qualitative assessments by previous mentors”, and therefore these two criteria were excluded from the univariate analysis on domains.

Table 2 Admission domains: Cronbach’s alphas and selective admission criteria included per domain
Full size table

Admission documents and tools

The admissions committee members were asked to indicate which documents and methods (transcripts/grade lists, course content, CV, motivation letter, letter(s) of recommendation, interview, English language test, writing sample, personal acquaintance) they use in assessing different groups of admission criteria (such as education background, disciplinary background, cognitive ability, personality, motivation and attitude).

Participants and response

At each Graduate School, admissions committee members and applicants to two-year research master’s programs (2017–2018 and 2018–2019 cohorts) participated. The participants’ characteristics and response rate are presented in the Table 3. Relatively low response rate is associated with rejected applicants not being inclined to respond to the survey from a graduate school that rejected them.

Table 3 Participants’ characteristics and response rate
Full size table

Data analysis

Analysis was conducted in SPSS 25. There were no missing values for the admissions committee members from both graduate schools; 0.2% of values in responses of the GSLS applicants and 0.4% of values in responses of the GSNS applicants were missing completely at random. To handle these missing values, the expectation-maximization (EM) algorithm in SPSS was used. In analysis, first, the descriptive statistics were obtained in order to rank the importance of admission criteria, and the frequencies of admission tools’ usage were obtained. To assess transparency, the importance of admission criteria, as reported by admissions committee members, was contrasted to the importance as perceived by applicants. This was done using Spearman correlation and 2-factorial analysis of variance (ANOVA) (1st factor = admission committees’ members versus applicants; 2nd factor = a domain of admission criteria).

Results

Research question 1: What kind of selective admission criteria are reported to be important in decision-making at the Graduate School of Life Sciences and at the Graduate School of Natural Sciences?

We examined the importance of suggested admission criteria in selective admission decisions on two levels: on the level of specific criteria and on the level of general admission domains. For examination of specific admission criteria’s importance, the means of 51 selective admission criteria were ranked according to their importance in actual admission decisions as reported by admission committee’s members (Table 4).

Table 4 Ranks, Means and Standard Deviations of Selective Admission Criteria: Admissions Committee Members’ Perspectives and Applicants Perceptions at the Graduate Schools of Life Sciences and Natural Sciences
Full size table

On the level of specific criteria

For the GSLS, the most important criteria were related to the domains “motivation factors” (n = 5), “academic background” (n = 3) and “grades” (n = 2). No criteria related specifically to research background, cognitive ability or personality aspects appear amongst the most important. For the GSNS, the highest positions are given to criteria related to “academic background” (n = 5) and “grades” (n = 2). There was also one criterion (“mastery of field relevant methods and techniques”) that is related to research background (#6). In contrast to the GSLS, specific criteria related to motivation also appear in the top-10, but only two of them and not in the highest positions (#6 and #10). In spite of the difference in ranking per criterion, the correlational analysis of the whole list of admission criteria revealed that overall, the importance, which both the GSLS and the GSNS members of admission committees assign to selective admission criteria is similar to a high extent: rPearson (49) = .76, p < .001.

On the level of general admission domains

After examination of specific admission criteria, we turned to a higher order level - exploration of relative importance of general selective admission domains. Their means and standard deviations are presented in Table 5.

Table 5 Means and standard deviations for domains of admission criteria as reported by admission committees’ members and perceived by applicants at the Graduate School of Life Sciences (GSLS) and the Graduate School of Natural Sciences (GSNS)
Full size table

For the GSLS, a univariate ANOVA revealed that there is a significant large effect of a domain on average importance of criteria, F(5, 43) = 9.02, p < .001, η2 = .51. Hochberg post-hoc examination indicated that the “personality and personal competencies” domain is less important compared to three other domains: “academic background” (p = .001), “grades” (p = .009) and “motivation factors” (p < .001). Also, the domain “motivation factors” is found to be more important than the domain “cognitive ability” (p = .041).

Analysis of the GSNS reports from the admission committee members showed that there is also a significantly large effect of domain on average importance of criteria, F(5, 43) = 8.77, p < .001, η2 = .51. Hochberg post-hoc examination revealed that the “personality and personal competencies” domain is less important compared to three other domains: “academic background” (p < .001), “grades” (p = .001) and “motivation factors” (p = .005).

A two-way factorial ANOVA revealed that there is no significant difference between reports of the GSLS and the GSNS members of admission committees, F(5, 86) = 1.30, p = .271, η2 = .07.

Research question 2: What admission tools are most frequently applied?

At the GSLS, such admission tools as a motivation letter and a CV, an interview, a letter of recommendation and personal acquaintance, if applicable, are all used extensively for assessment of admission criteria. For example, a motivation letter is used for assessment not only of criteria related to motivation itself, but also of criteria related to personality, cognitive ability and previous academic background. Next, there are tools, namely, grade list and course content, that are used in a more focused way. For example, course content is used predominantly for assessment of academic background, but not for assessment of personality and motivation. Finally, a writing sample and an English language test are not frequently used: only a few members of admission committee (25%) indicate that they apply these tools.

At the GSNS, similarly to the GSLS, such admission tools as a motivation letter, a CV and a letter of recommendation are used frequently and for measurement of diverse admission criteria, while a grade list and a course content are also used extensively, but more focused (namely, for measurement of academic background). In contrast to the GSLS, interviews and inferences based on personal acquaintance are not of frequent use at all. Like at the GSLS, at the GSNS, an English language test and a writing sample are not often used for selective admission decisions.

Research question 3: Are the applied criteria transparent for the applicants?

The importance of selective admission criteria at the actual admission decisions and transparency of selective admissions for the applicants were examined on two levels: on the level of specific selective admission criteria and on the level of general admission domains.

For the GSLS, the correlation between the importance of specific admission criteria in decision making as reported by admission committees’ members and the perceived importance of the same criteria by the applicants, is of a moderate-to-large size, rPearson (51) = .53, p < .001. Two-way analysis of variance on admission domains revealed that there is significant interaction effect between the respondent group (admission committee members versus applicants) and the type of domain (academic background, grades, cognitive ability, research background, personality and personal competencies, and motivation factors) on their importance, F(5, 86) = 2.51, p = .036, η2 = .13. As can be seen from Fig. 2, the extent to which the GSLS applicants perceive criteria related to academic background, grades, research background and motivation to be important is in coherence with reports from admission committee members. The GSLS applicants overestimate the importance of criteria related to cognitive ability and personality.

Fig. 2
figure 2

Mean importance of admission criteria domains in selective admission decisions with 95% confidence intervals, reported by the GSLS admission committees’ members (n = 16) and perceived by the GSLS applicants (n = 327)

Full size image

For the GSNS, similar results were obtained. Reports of admission committees’ members on importance of specific admission criteria are correlated with applicants’ perceived importance to a large extent, rPearson (51) = .63, p < .001. On the level of admission domains, two-way ANOVA revealed a significant interaction effect: the applicants perceive the importance of admission criteria domains differently from what the admission committees’ members report, F(5, 86) = 2.98, p = .016, η2 = .15. Figure 3 shows that four domains – academic background, grades, research background and motivation - were perceived by the GSNS applicants to have similar importance in admission decisions as was reported by the admission committee members. Like their GSLS peers, the GSNS applicants overestimate the importance of criteria related to cognitive ability and personality.

Fig. 3
figure 3

Mean importance of admission criteria domains in selective admission decisions with 95% confidence intervals, reported by the GSNS admission committees’ members (n = 10) and perceived by the GSNS applicants (n = 153)

Full size image

Discussion

In this study, we aimed to answer two central questions: How do admission committees of Life and Natural Sciences research Master’s programs select? Do applicants of two particular Life and Natural Graduate Schools know how the admission committee select? We examined what kind of specific selective admission criteria and admission domains are important in admission decisions at the Life and Natural Sciences Graduate Schools. We found that criteria related to applicants’ academic background, grades, motivation factors, but also cognitive ability and research background, played an important role in admission decisions. We also examined what kind of admissions tools are used the most. We found that an unstructured motivation letter, a CV and a grade list are the most frequently used tools, but other tools are used for measurement of various criteria as well. Finally, we investigated how accurate do applicants, who have just undergone the admission process, perceive the importance of admission criteria in decisions regarding their application. We found that applicants’ perceptions were accurate to a moderate degree. At both graduate schools, criteria related to academic background, research background, grades, cognitive ability and motivation factors were reported by admission committee members to be important, but not the criteria related to personality and personal competencies. Below we discuss the evidence on the criteria and related admission tools per admission domain.

Academic background

Our findings suggest that a match between content of previous education and a master’s program is by far one of the most important criteria in admission decisions (rank #2 at the GSLS and rank #1 at the GSNS). This finding is consistent with other studies (Karazsia and McMurtry 2012; Karazsia et al. 2013; Posselt 2016; Walpole et al. 2002). Also, we find that criteria related to quality of undergraduate experience (a prior degree in a specific field, type of prior educational institution, grades) are reported to be important in admission decisions, which is also in line with existing literature (MasterMind Europe 2017; Kogan et al. 2015). To the best of our knowledge, there are no comprehensive studies on association between previous academic background (such as type of previous institution etc.) and graduate academic achievements. Nevertheless, it seems plausible that prior academic background in a certain discipline (ranked #5 by the GSLS and #2 by the GSNS) provides an advantage for admission in a program within a similar discipline. However, the high rankings of a criterion “Type of prior educational institution” (a preference for either a Dutch research university/comparable international university or for university college or university of applied sciences (HBO) - rank #3 at both graduate schools) raises concerns, since giving a decision power to this criterion might place applicants from certain type of institutions in a dis/advantageous position. In order to account for differences in acquired level of knowledge and skills at different types of higher education institutions, one possible solution could be to provide an opportunity for the applicants to deliver the results of validated graduate standardized tests as evidence of their abilities, necessary for starting education at a graduate level, as there is evidence that standardized tests are valid predictors for graduate student study success (e.g. Kuncel et al. 2001; Oh et al. 2008)

Grades

Both on a level of specific criteria and on a level of more general domains, we find that criteria related to grades play an important role in admission decisions in both graduate schools. For instance, undergraduate GPA and grades for relevant courses are in top-10 at both Life and Natural Sciences Graduate Schools. On the level of a more general domains, grades are shown to be of a moderate importance in admission decisions. Importance of grades in admission decisions at both GSLS and GSNS might be regarded as a good practice, as prior findings indicate that undergraduate GPA, and last-year GPA in particular, are valid predictors of a number of future performance indicators (Kuncel et al. 2001; Zimmermann 2016).

Cognitive ability

At both GSLS and GSNS, criteria related to cognitive ability are reported to be of a slight-to-moderate importance, and are less important than motivation, academic background, research background and grades. Inference about cognitive ability is done based on different admission tools (including motivation and recommendation letter, interviews, personal acquaintance). A variety of tools for inferring information about cognitive ability of an applicant raises concerns, since some of those admission tools firstly, were not developed for cognitive ability measurement, and secondly, have serious issues with reliability and validity indicators (Patterson et al. 2016). The tests that are primarily developed for measurement aptitude and cognitive ability (e.g. Miller Analogies Test, Graduate Record Examinations, Graduate Management Admission Test) are not required at the examined programs.

Research background

Not surprisingly, the current study also finds that criteria related to research experience play a role in admission decisions. There are only two studies in the life sciences field that investigated the predictive validity of amount of previous research experience and their findings are contradictory: while in one study an association between amount of prior research experience and research productivity was found (Weiner 2014), no similar association was found in another study that tried to replicate the findings of the first one (Hall et al. 2017). While it is plausible to assume that previous research experience is associated with academic success at research-oriented programs, at the moment the existing evidence is insufficient for using it as a factor with decisive power. We therefore suggest to exercise caution in its usage in admissions practice.

Motivation factors

We find that motivation factors play an important role in admission decisions, especially at the GSLS and to a lesser extent at the GSNS. Namely, at the GSLS, six out of top 10 criteria were related to motivation, and at the GSNS, two out of top 10. When we look at the motivation factors as a general domain, we see that motivation factors are a bit more than moderately important at the GSLS, and slightly-to-moderate important at the GSNS. Finally, criteria related to motivation factors were reported to be measured predominantly by unstructured motivation letters, CV, interviews and unstructured recommendation letters. While some motivational aspects were indeed shown to have associations with academic performance (e.g. Robbins et al. 2004), operationalization of motivation in admissions raises serious concerns, as three indicated admission tools are self-reported measures.

Researchers clearly call against using self-report measures in student selection (Murphy et al. 2009; Niessen et al. 2017; Patterson et al. 2016; Wouters 2016). It was demonstrated that self-reported measures induce a self-presentation effect, or, in other words, applicants tend to fake their reports in the selection context, which represent a high-stake situation (Niessen et al. 2017). Also, these measures have a small predictive value towards future academic performance and do not provide sufficient incremental validity in addition to prior grades and results of standardized admission tests (Goho and Blackman 2006; Hell et al. 2007; Murphy et al. 2009). As for the measure that is also extensively used for assessment of applicant’s motivation and is not self-reported – a recommendation letter - the scientific evidence exists for its practical value only in its standardized form (Kuncel et al. 2014).

Personality and personal competencies

Low importance of personality factors for admission decisions is not in line with prior indications of the importance of personality and personal competencies in student selection (Conlon et al. 2012; Walpole et al. 2002). At least two possible explanations arise. The first one might be related to the design of our study since we used self-reports of admission committee members and therefore the answers could be socially desirable. Other methods (e.g. observations) could shed light on possible importance of applicant’s personality during the admission process that may have been avoided in a self-report. Another possible explanation may be that for research-oriented master’s programs, the applicant’s personality indeed does not play an important role in decision making regarding their admission. An interview study on selective admissions to undergraduate programs supports this explanation in the context of American and English elite universities. The study found that admission on academic record alone is typical for English universities and is an exception at the American universities, where much more attention is given to personal attributes of applicants (Mountford Zimdars 2016). Also, the survey study of the MasterMind project showed that only one-third of European programs consider personal skills in admission decisions (MasterMind Europe 2017). It would not be surprising if the admission practices in the Netherlands are more in line with other European countries’ practices rather than with American ones, and personality and personal competencies do not play a crucial role in admissions.

A limited usage of personality traits and competencies in admission decisions might be regarded as a good practice. One of the most stable findings is that a personality trait such as conscientiousness is associated with academic performance (Poropat 2009; Schneider and Preckel 2017; Trapmann et al. 2007), but the predictive validity of other personality traits - openness to experience, agreeableness, neuroticism and extraversion - range from negligible to small, with overall conclusion that they do not have a substantial impact on academic performance. One may argue that results of conscientiousness might be used in admissions, but since personality testing is a self-report measure, which is susceptible for faking (Niessen et al. 2017), we would warn against such a practice.

Are the applied admission criteria transparent for applicants?

We explored whether the applicants, who already know their admission decision, accurately perceive the criteria based on which the decision regarding their applications was made. Our results show that although transparency is of a moderate to large extent on the level of specific criteria, on the level of general academic domains it is not the case. Applicants perceive accurately certain domains (academic background, grades, research background and motivation factors), but not others (cognitive ability and personality are overestimated in terms of their importance in admission decisions by applicants of both graduate schools).

These results suggest that some of information that is presented to applicants (e.g. at the admission web-pages) provides a clear description of a criterion (e.g. minimum scores for English test, or a specific list of courses, which an applicant had to follow during previous education), but other information might be unclear (e.g. such criteria as “talent” or “high level of academic and professional ability”). Along with that, it might also be that applicants do not use enough information sources (such as programs’ webinars, Master’s Open Days, communication with a program coordinator via email), in order to get informed about criteria, based on which their application will be assessed. As one of the applicants commented: “I did not find much concrete accessible information on the selection criteria, but I did not look very intensely”.

Strength, limitations, further directions and conclusions

Among the strengths of this study are the availability of data from (a) both admission committee members and applicants and from (b) two graduate schools with different scope and different curricula designs. The information from admission committees’ members allowed us to describe the actual admission practices, and information from the applicants allowed us to assess whether these practices are clear to those for whom they are designed. Next, we used a comprehensive list of specific admission criteria, which allowed us to conduct analysis both on a level of specific criteria and on a level of domains – when several similar criteria were considered together. With this approach, we were able to capture the specific trends in applicants’ estimation of admission criteria importance. The fact that for both graduate schools we found either correct estimation or overestimation in the same admission criteria domains speaks for a possible generalizability of our results to research master’s programs. Furthermore, to the best of our knowledge, this is only the second study in the European context that explored admission practices both from admission committee members’ and applicants’ views (see MasterMind Europe 2017 for the survey across Europe), but the first one that directly contrasted perceived importance against actual practices of selective admissions.

Despite these strengths, the consideration of some limitations should be made when interpreting the findings. Firstly, a self-report by admission committee members leaves a possibility that the reported importance of certain admission criteria does not completely correspond with their real importance in the daily practice of selective admission decision making. Nonetheless, we assume that the staff members’ reports closely resemble their practice. Secondly, in the third part of the study, dedicated to transparency of selective admission criteria, most of the respondents were accepted applicants. Therefore, the results might have been affected by a selection bias: the accepted applicants might be those who had a chance to learn about the “unspoken” admission rules in contrast to students who did not have a chance to learn such implicit information from their environment. Thirdly, the study is conducted within one higher education institution that could narrow down the generalizability. Yet the study involves two large graduate schools, one of which is interfaculty, and multiple master’s programs within each of them, so we expect the results to be relevant for (life and natural) science programs more generally.

In summary, this study contributes to our knowledge on what admission committee members are looking for in future graduate students who apply for a research master’s program. As a practical implication, the study suggests that the quality of admissions should be a subject of evaluation and constant improvement in terms of scientific evidence, fairness and transparency. The results are relevant for applicants, researchers in higher education, in particular with a focus on admissions quality, and for professionals, who are involved in the (daily) practice of selective admissions in higher education institutions.