Struggling with strugglers: using data from selection tools for early identification of medical students at risk of failure
- 185 Downloads
Struggling medical students is an under-researched in medical education. It is known, however, that early identification is important for effective remediation. The aim of the study was to determine the predictive effect of medical school admission tools regarding whether a student will struggle academically.
Data comprise 700 students from the University of New South Wales undergraduate medical program. The main outcome of interest was whether these students struggled during this 6-year program; they were classified to be struggling they failed any end-of-phase examination but still graduated from the program. Discriminate Function Analysis (DFA) assessed whether their pre-admission academic achievement, Undergraduate Medicine Admission Test (UMAT) and interview scores had predictive effect regarding likelihood to struggle.
A lower pre-admission academic achievement in the form of Australian Tertiary Admission Rank (ATAR) or Grade Point Average (GPA) were found to be the best positive predictors of whether a student was likely to struggle. Lower UMAT and poorer interview scores were found to have a comparatively much smaller predictive effect.
Although medical admission tests are widely used, medical school rarely use these data for educational purposes. The results of this study suggest admission test data can predict who among the admitted students is likely to struggle in the program. Educationally, this information is invaluable. These results indicate that pre-admission academic achievement can be used to predict which students are likely to struggle in an Australian undergraduate medicine program. Further research into predicting other types of struggling students as well as remediation methods are necessary.
KeywordsMedical education Undergraduate Australia Prediction Medical student
UNSW: University of New South Wales
AustralianTertiary Admission Rank
Grade Point Average
The Undergraduate Medicine and Health Sciences Admission Test
Research into medical student selection often focuses on prediction models for identifying applicants who would excel as students and less so on those who are likely to fail . However, there is a group of students known as ‘strugglers’ who fall into neither of these categories. These struggling students are the ones who encounter academic and/or personal difficulties, which causes them to struggle throughout their study in a medical program . Whilst this is a broad categorisation, often the literature defines such groups differently to one another and uses varying names for the students, such as ‘problem learners’, ‘at-risk students’, ‘disruptive students’, and ‘marginal students’ [2, 3]. The prevalence of strugglers is rarely reported but is believed to lie in the range of 10–15% among undergraduate medical programs [2, 3, 4], though rates as high as 36.6–47.3% of students have been reported to have had impeded progress through their medical course . This number could potentially be explained by the usage of different definitions or the fact that struggling students are often unidentified, or poorly reported [2, 6]. These students often report lower motivation, increased frustration and boredom, and lower confidence in their ability to learn [3, 7]. Furthermore, sstruggling has been shown to have a strong association with withdrawal from the course and has also been shown to have significant negative effect upon both the students themselves as well as the universities they attend [1, 3].
Currently, the main challenge posed by these struggling students is the need for effective tools and methods to accurately identify them, so that remediation plans can be put in place [7, 8, 9, 10]. These students are not necessarily incapable of completing their course, and in fact, it has been suggested that up to 95% of the struggling medical students do indeed have the essential attributes to successfully complete a medical program  . G Paul, G Hinman, S Dottl and J Passon  survey, found that students enter medical school with the same level of coping and learning skills, which suggests that struggling students are those who experience more trouble managing their time and the course material. Identifying characteristics specific to students within this group and incorporating them into the medical school selection process could be beneficial as this would allow the implementation of early targeted intervention. Considering the scarcity of knowledge on this topic, research into this area is greatly needed.
The ideal scenario would be to predict those students who may have future difficulties by using data made available through the medical selection process. Previous research suggests that there are a few factors that may be useful for this purpose: lower grades in biology and late offers to the medical program; male gender [12, 13]; lower high school grades [4, 14]; and an inverse relationship between Medical College Admission Test (MCAT) scores and academic difficulty . Some studies reported weak associations between demographic characteristics (e.g. gender, ethnicity etc.) and struggling, but no evidence was found to suggest that factors other than personal attitudes and aptitudes, or previous educational experiences, have any significant impact on the performance of medical students .
The group of strugglers is not negligible and too little is known about how best to identify them early in the program. It is therefore suggested that research should be undertaken which aims to: (1) establish a valid set of criteria for identifying struggling students; (2) identify factors that best predict struggling performance; and (3) identify methods for effective intervention. The current study focuses on points (1, 2), whereas point (3) should be addressed in future research.
List of variables
Mean score across all three UMAT sections
ATAR (or equivalent)
Measure of prior academic achievement, either from secondary school (ATAR) or from attendance at a previous university program (GPA)
Total score of their UNSW admission interview
Whether the student had failed in any of the major course assessment (described below) whilst still completing the UNSW medicine program.
The present study examines four consecutive cohorts (commencing years 2006–9) of students that have been through and graduated from the UNSW undergraduate medical program in recent years (admissions and assessment outcomes). We established a data set that combined student’s admission scores and their examination results by cross-referencing their identification numbers. Each student was categorised as struggling or not struggling; struggling students were defined as any student who failed any end-of-phase examination but still graduated from the program. Only the results of the end-of-phase examinations were used as other examinations throughout the program are mostly modular which means that students might undertake them at different times in their course. In comparison, all the students must undertake the end-of-phase examinations at the same point in the medicine program which makes these examinations a more fair and direct comparison between the students. Additionally, students can often proceed with the course and resit a failed course examination at a later point whereas they must successfully pass the end-of-phase exams to proceed. All students that failed during the medical program or dropped out prior to completion were excluded from the analysis. Additionally, no students applying through international pathways were included in the analysis as the admission tools used for those students are different in most cases.
There were 700 students in our sample of four cohorts that had both admission scores and examination results. The total study population was 700 UNSW undergraduate medical program students of which 55.3% were female and 44.7% were male. The 700 were comprised of 77.4% local and 22.6% rural students.
We performed a Discriminate function analysis (DFA) to measure how well the current admission tools employed by UNSW were in categorising and differentiating the struggling students from the others. DFA is a statistical procedure that identifies a mathematical function that maximises the accuracy of determining which group a person would belong to, based on the chosen variables . In this study, DFA was used to determine the optimal separation of the students into two groups: a struggling and a not struggling group based on their admission scores. Any student that had to repeat at least one of their end-of-phase assessments or failed an assessment but completed the course was classified as struggling. All other students who graduated from the medicine program without having failed or repeated an end-of-phase assessment were classified as not struggling. The struggling status of the students was compared against their admission scores of ATAR, UMAT and interview scores using the DFA by IBM SPSS v22.0 . The ATAR stands for ‘Australian Tertiary Admission Rank’ which is a number between 0.00 and 99.95 that indicates a student’s position relative to all the students in their age group in NSW in Australia . Applicants/students coming from other states in Australia may have different secondary school results which are then transformed to an ATAR equivalent. UMAT stands for The Undergraduate Medicine and Health Sciences Admission, which is an aptitudes test measuring three domains: Logical Reasoning and Problem Solving; Understanding People; and Non-verbal Reasoning .
The DFA generates a function that best discriminates between this subgroup of students and the other students.
The research was conducted under the ethics approval granted by the Human Research Ethics Committee ofthe University of New South Wales (reference, HC15421, HREAPG: Health, Medical, Community and Social).
The ethics approval does not require consent from participants (either for using the data for research or for publication), thus such consent was not obtained. This study used administrative data held UNSW Medicine.
DFA summary table
Predicted Group Membership
Efficacy of selection tools in predicting students likely to struggle
Canonical Discriminate Function Coefficients
ATAR (or Equivalent)
Functions at Group Centroids
The mathematical function produced by the DFA is summarised by the Canonical Discriminate Function Coefficients and the Group centroids (Table 3). Using the corresponding coefficients for each admission factor the optimal cut-score can be produced. Since struggling is a negative value whereas not-struggling is a positive value, any student that falls above the cut-score determined by the DFA would be most likely to be not struggling compared to any student that is below the cut-score. The standardised values allow a comparison of the predictive effect of each admission factor and this shows that ATAR (or equivalent) has the greatest predictive effect of struggling students out of the three admission tools that were analysed.
This study focused on an overlooked, yet important topic in medical student admissions, while determining whether or not selection tools (ATAR, UMAT, and interview scores) had any utility in identifying which students would most likely struggle during an Australian undergraduate medical program. We found that all three admission scores had a slight association, but that the ATAR had the strongest inverse association out of the three admission tools in differentiating struggling students from the other students.
Comparison to other studies
This study is both novel in the target population we examined as well as in the techniques utilized. Most studies in this area primarily focus on the use of admission tools in identifying students that are most likely to perform well at university, or in identifying those that are likely to fail [21, 22]. This study differs by focusing upon the lesser-researched population of ‘struggling’ students [21, 22]. For the purposes of this study, the focus was on students who struggle academically; defined as any student that successfully graduated from the program but had encountered issues with the course (e.g. failing one or more exams) (see: Method section). The second novel aspect of this project was in the use of DFA for exploring the effectiveness of current medical school selection tools .
Other current research regarding admission tools and medical school performance are generally focused on students that excel and students that fail as opposed to looking at the struggling students in between. The consensus of these studies as well as the few that have looked at struggling students is that pre-admission ATAR/GPA is the most effective tool in predicting student performance in the medical program [5, 24, 25, 26, 27]. This is often assumed to be the case as ATAR/GPA may be a reflection of either an initial shortcoming in biological concepts and processes, or a shortcoming in their academic ability such as their study skills .
Future policy and research
Our findings raise two main points with respect to the use of these admission tools in admitting students. Firstly, our findings agree with the consensus that previous academic achievement (comprised of either ATAR or GPA) seems to be the most effective admission tool in helping to determine which students should be admitted, but this does question the use of the interview and UMAT with equal weighting to a student’s ATAR or GPA [5, 29]. This is because the other two admission tools have a substantially lower predictive effect than that of the ATAR, though perhaps those two tools are useful in identifying other characteristics that the university finds desirable in its students .
The other issue that our findings raise is that the ability of these admission tools to classify these students is neither ineffective nor infallible, but somewhere in between. One question we must consider is whether we can truly say that we can linearly rank/model these students, with the students that do best on the upper end of the scale and the students that fail on the lower end of the scale. Whilst the consensus is that previous academic achievement in the form of ATAR/GPA is the most consistent predictor of medical program success , the results of this study suggest that it is not perfect. In that regard, there is the possibility for improvement in terms of admission tools which may allow better ability to identify students that will struggle as well as in creating a more diverse and desirable range of students.
In terms of how these findings could be applied to the medical education and medical programs at various universities, the most straightforward and potentially beneficial way would be to assist struggling students early to prevent future struggling throughout their time in the medical program . Currently, universities can only provide support once a student has already been shown to be struggling (e.g. failing an exam) but based on our results universities could offer targeted support to the specific students who are likely to struggle. This is in comparison to the current situation where, if universities wanted to offer early support, it would realistically have to be broad support to the entire student population. By using our findings in this way universities could assist students prior to them struggling and prevent these students from suffering the negative consequences of struggling.
Even though our results show that ATAR/GPA can be used reasonably reliably to identify struggling students, more research needs to be done intotheeffective long-term remediation to assist these students. Additionally, research delving in to the benefits of the use of the UMAT and interviews as admission tools may prove useful as they may in fact be more predictive of other desirable qualities and thus may lead to greater diversity in the medical student and doctor population .
The limitations of our study were primarily based on the data we had access to. We were only able to investigate struggling students that had failed in one of the end of phase assessments (as mentioned previously) based on the assessment results we had access to. Because a struggling student was defined as one who failed or had to repeat any end-of-phase assessment but still completed the medicine program, other types of struggling students may have been missed in the scope of this study. Other studies have identified different types of struggling students such as those that may have to postpone or failed assessment because of illness  . Additionally, since this study was focused upon the three main admission tools of UMAT, ATAR/GPA and interview scores, many international students who were admitted without the use of all three of these tools were unable to be included within our study. Being unable to assess all possible different categories of struggling students was unavoidable for our project as we were using previously acquired data. For further research, more detailed data and analysis regarding different types of struggling students may provide greater insight in to the predictability of the current admission tools overall.
The results of this study show that ATAR/GPA is effective in predicting which students are likely to be struggling within the UNSW medical program. Results of this study could be used by other medical schools to assist in early identification of students that are likely to struggle. This would in turn allow the faculty to monitor, track, support, and remediate the students during their time in the medical program. Doing so early is likely to result in better outcomes for these students as well as better prepared and well-equipped physicians entering the medical workforce.
The authors express their gratitude to the UMAT consortium for their support of this research.
BS has initiated the study, designed the methodology, supervised the conduct of the study, interpret the results and contributed to the writing. JL undertook the data management and analysis contributed to the interpretation of the results and to the writing of the manuscript. RT contributed to the interpretation of the results and to the writing of the manuscript. All authors read and approved the final manuscript.
This study was funded by a grant awarded from the UMAT Consortium. The UMAT consortium had no role inthe design of the study, data collection, data analysis and interpretation, orin writing the manuscript.
Ethics approval and consent to participate
The research was conducted under the ethics approval granted by the Human Research Ethics Committee ofthe University of New South Wales (reference, HC15421, HREAPG: Health, Medical, Community and Social).
Consent for publication
The ethics approval does not require consent from participants (either for using the data for research or for publication), thus such consent was not obtained. This study used administrative data held by UNSW Medicine.
The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article. Author Boaz Shulruf is a member of the editorial board for BMC Medical Education. He had no editorial role in the processing or peer review of this manuscript and was blinded to this process.
- 5.Sladek R, Bond M, Frost L, Prior K. Predicting success in medical school: A longitudinal study of common Australian student selection tools. BMC Medical Education. 2016:16(1).Google Scholar
- 14.Shulruf B, Shaw J. How the admission criteria to a competitive-entry undergraduate programme could be improved. High Educ Res Dev. 2014:1–14.Google Scholar
- 17.Garson D. Discriminant function analysis. Asheboro: Statistical Associates Publishing; 2012.Google Scholar
- 18.IBM.: IBM SPSS Statistics 22 Command Syntax Reference: IBM Corp.; 2013.Google Scholar
- 19.What is the ATAR? [https://www.uac.edu.au/future-applicants/atar].
- 20.Australian Council for Educational Research: UMAT 2018. Melbourne, Australia: Australian Council for Educational Research; 2018.Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.