Using the personal background preparation survey to identify health science professions students at risk for adverse academic events
- First Online:
- Cite this article as:
- Johnson, C.W., Johnson, R., McKee, J.C. et al. Adv in Health Sci Educ (2009) 14: 739. doi:10.1007/s10459-009-9156-4
- 221 Views
In the first predictive validity study of a diagnostic and prescriptive instrument for averting adverse academic status events (AASE) among multiple populations of diverse health science professions students, entering matriculates’ personal background and preparation survey (PBPS) scores consistently significantly predicted 1st- or 2nd-year AASE. During 1st-year orientations, 441 entering matriculates in four southwestern schools from dental, medical, and nursing disciplines completed the 2004 PBPS. The following year during 1st-year orientations, 526 entering matriculates in five schools from dental, medical, nursing, and biomedical science disciplines completed the 2005 PBPS. The PBPS identifies and quantifies a student’s noncognitive and cognitive academic performance risks. One standard deviation increments in PBPS risks consistently multiplied 1st- or 2nd-year AASE odds by approximately 140% (p < .05), controlling for underrepresented minority student (URMS) status and school affiliation. Odds of 2nd-year AASE for URMS one standard deviation above the 2004 PBPS mean reached 494% of odds for nonURMS at the mean. PBPS total risks, school affiliation, and URMS status together provided 70–76% correct predictions of 1st- or 2nd-year AASE. PBPS predictive validity did not differ significantly among dental, medical, nursing, or biomedical science schools, or URMS/nonURMS. PBPS sensitivity and specificity approached those for FDA-approved screening mammograms for breast cancer and PSA tests for prostate cancer. PBPS positive predictive values of 42–60% exceeded those for both. The diagnostic and prescriptive PBPS can facilitate proactive targeting of corrective interventions aimed at reducing AASE and attrition among health science education students at risk for academic difficulties.
KeywordsHealth sciences professions studentsUnderrepresented minority studentsStudent retentionStudent attritionHealth sciences graduate educationInstrumentationMeasurementStudent advisingStudent persistence
Over the last half century efforts to understand and reduce student attrition, loss of enrolled students prior to program completion, in higher education have spawned a growing body of research, as reviewed by Johnson et al. (2009a), with mounting emphasis on populations of health science professions students. As part of these efforts, retention, the consistent progression of students in an educational program, has received increasing attention, especially for underrepresented minority students (URMS).
Data from the annual ACT Institutional Data Questionnaire have consistently revealed substantial student attrition losses from MA/professional programs. From 2001 through 2008, annual 1st- to 2nd-year student retention rates varied within a narrow range from 69.2 to 71.6% for public MA/professional programs and from 72.3 to 75.8% for private ones (ACT 2001–2008).
Somewhat lower student attrition rates characterized 26 allied health occupations during the 1989–1990 academic year, ranging from 7.1% for academic health centers and medical schools to 24.3% for vocational and technical schools, according to Committee on Allied Health Education and Accreditation data (cited in Gupta 1991). However, black students not of Hispanic origin experienced significantly higher attrition (25.4%) than other categories of race.
More recent data confirmed similar attrition rates during the 2001 through 2003 academic years. American Medical Association survey responses of 4,711 program directors from 5,701 accredited or approved health professions’ education programs documented annual program attrition rates up to 33.7%, with an overall rate of 13% (AMA 2004). Student attrition losses of this magnitude exact substantial immediate costs to affected institutions (Habley 2004). Student attrition in health science higher education exacts special costs in both shortages and diversity of trained health workforce professionals (Childs et al. 2004; Wells 2003).
Fitzpatrick and Wright (1995) stated that student retention was a critical problem in medical education. Their data showed medical school attrition rates had steadily increased across the country from 1973 through 1992. Tekian (1998) presented evidence that URMS at the University of Illinois at Chicago College of Medicine from 1993 to 1997 experienced disproportionately high attrition because of academic difficulties and that URMS could benefit from special academic attention. Association of American Medical Colleges data (AAMC 2007) reported stable M.D. degree completion rates for medical students ranging from 80.6 to 82.2% for three matriculating classes (1987, 1992, 1995) over 4 years, but disparate degree completion rates for different racial or ethnic groups most apparent for Black/African-American students at years four and five.
Richardson (2000) indicated that interest in the achievement of URMS has particularly been lacking at research universities, where unique opportunities and resources for research and collaborations among faculty and students exist. Anecdotal observations and reports suggest the existence of perceptions among some faculty and administrators that once students have risen to a health science professional or graduate level, student retention programs and resources can receive reduced priority, especially where recruitment and admissions efforts have selected superior students. Recent research by Johnson et al. (2009a) has documented that in some health science professional or graduate schools where students had low levels of risk predictive of adverse academic status events (AASE): (1) URMS were risk disadvantaged. (2) URMS constituted especially appropriate candidates for student retention interventions. (3) Retention programs and resources for URMS were especially needed.
Abdur-Rahman and Gaines (1999) asserted, “The high attrition rate of minority nursing students, on a local and national level, has not been effectively addressed” (p. 33). More recently, Wells (2003) noted a paucity of recent research on nursing student attrition and even less on contributors to high nursing student attrition rates among nonwhites.
Costs of attrition
The Institute for Higher Education Policy (1998) has identified many benefits of higher levels of education. Public economic benefits include increased workforce flexibility and greater productivity. Individual economic benefits include higher salaries, employment, and higher savings. Public social benefits include charitable giving and appreciation of diversity. Individual social benefits include health, life expectancy, and quality of life. Beyond loss of these benefits, attrition among health science professions students has substantial additional costs. Among these are shortages and diversity of trained health workforce professionals (Childs et al. 2004; Wells 2003) and more immediate substantial costs to the institutions: lost tuition, fees, faculty lines, and recruitment costs (Habley 2004). The health science schools surveyed during the present investigation reported institutional financial losses ranging from $10,000 to $50,000 per lost student-year, depending upon school and discipline.
Noncognitive or nonacademic factors important for student retention
Numerous research studies document the important role played by noncognitive or nonacademic factors in college undergraduate achievement, attrition or retention (Habley and McClanahan 2004; Johnson et al. 2009a). Undergraduates’ noncognitive background characteristics, attitudes, motivation, social and environmental factors, as well as cognitive indicators, need to be addressed in attempting to identify or ameliorate potential attrition risks (Clewell et al. 2005; Tracey and Sedlacek 1984). Minority students at majority undergraduate institutions may also have greater retention risks than their majority counterparts (Allen 1988; Clewell et al. 2005; Loo and Rolison 1986; Pounds 1987; Smedley et al. 1993).
Nontraditional student retention
Braxton et al. (1988) identified only “subsequent institutional commitments” as significantly related to attrition decisions among 104 freshmen at a midwestern urban commuter university. Metzner and Bean (1987) identified psychological variables as the strongest predictors of attrition for 624 part-time freshmen commuter students. Metzner (1989) concluded that high-quality freshman advising improved attrition by affecting satisfaction in the role of a student, the value of a college education for future employment, GPA, and intent to leave the university, for 1,033 freshmen at a midwestern commuter university. Jeffreys (1998), citing these three research studies, stated that noncognitive variables had been found to influence nontraditional students’ academic achievement and retention more than academic variables.
Nursing or medical student attrition
A qualitative study by Wells (2006–2007), p. 439) of 11 nursing students who left undergraduate baccalaureate nursing programs revealed they left as a result of multiple academic, social, or external stressors. As with undergraduates, medical and nursing student background characteristics, attitudes, motivations, and cognitive indicators (e.g., GPA, entrance exam scores, etc.), have been found to be predictive of academic difficulty and attrition (Abdur-Rahman and Gaines 1999; Abernethy 1999; Cariaga-Lo et al. 1997; Giordani et al. 2001; Huff and Fang 1999; Jeffreys 1998; Price and Balogh 2001).
Appropriate early intervention facilitates health science URMS and nonURMS retention
Although many medical schools have offered enrichment programs for disadvantaged students (Wilson and Murphy 1999), comparatively little recent research has addressed effects of such programs on attrition (Kornitzer et al. 2005; Tekian and Hruska 2004). However, a number of those programs that have had success with URMS and nonURMS in medicine (Abernethy 1999; Giordani et al. 2001; Kornitzer et al. 2005; Tekian and Hruska 2004) or nursing (Abdur-Rahman and Gaines 1999; Hesser et al. 1996; Jeffreys 2001; Price and Balogh 2001) instituted early interventions addressing noncognitive factors like personal growth, study skills, coping techniques, networking, mentoring, guidance, and social support. Giordani et al. (2001) hypothesized that such factors may be critical for consideration in other enrichment programs.
Early diagnosis and prescription for health science URMS and nonURMS retention
Consistent with the key importance of early identification (EID) recognized by Wells, Seidman and others, and based on ACT’s, Spring 2004, WhatWorks in Student Retention national survey returns from 1,061 undergraduate institutions, Habley and McClanahan (2004, pp. 7, 22) listed second among eight recommendations for improving student retention, “Conduct a systematic analysis of the characteristics of your students.” The present study investigates the predictive validity of an instrument that operationalizes Seidman’s EID and implements the recommendations of Well’s, and of Habley and McClanahan: a postmatriculation diagnostic and prescriptive instrument for averting AASE. This instrument is designed to enable diverse populations of health science professions students, their advisors, mentors, committee chairs, or counselors to early detect, proactively pinpoint, and empirically target retention risks for corrective interventions.
Developed by a coauthor (R. Johnson) of the present investigation, the personal background and preparation survey (PBPS) is the first diagnostic and prescriptive instrument for averting AASE among health science professions students. The PBPS was designed to facilitate early identification of who is at risk, the degree of risk, what the risks are, and what to do about them. It identifies students’ noncognitive and cognitive risks, challenges, or concerns that can compete with or inhibit effective academic performance, and that can lead to nonadvancement or attrition; it provides interpretations or recommendations regarding targeting corrective interventions. With a student’s authorization, individualized PBPS Student Reports may be forwarded to student-designated advisor(s) to assist their efforts to timely target risks for amelioration and to avert AASE.
PBPS total risks, controlling for school affiliation and URMS status, have consistently (p < .05) predicted 1st- and 2nd-year AASE that could lead to nonadvancement and attrition among two populations of nursing students. Johnson et al. (2009b) concluded, “The PBPS consistently and significantly facilitated early identification of AASE-risk nursing students, enabling proactive targeting of interventions for risk amelioration and AASE or attrition prevention.”
The present study pursues whether the domain of PBPS predictive validity extends to more broad and diverse populations of health science professions students in a variety of disciplines. It assesses PBPS predictive validity and reliability for URMS and nonURMS among students from multiple health science professions over a period of 2 years.
Selected demographic characteristics of 2004 and 2005 PBPS respondents
First of family in college
At least one child
English primary language
Age older than 30
The research participants for the logistic regression analyses of the present investigation were those newly matriculated students having complete data (no missing data) on three predictors (total risks, school affiliation, and URMS status) from the PBPS and the relevant criterion variable(s) (1st- and/or 2nd-year AASE) from end-of academic year student records. The 2004 research participants (N = 433) included 57 from dentistry, 195 from medicine, 70 students from nursing A, and 111 from nursing B. All had complete data on all variables, except nursing A lacked one participant’s 2nd-year AASE (from June 2006). The 2005 research participants (N = 516) were those having complete data including their 1st-year AASE (from June 2006): 68 from dentistry, 196 from medicine, 63 participants from nursing A, 123 from nursing B, and 66 from biomedical sciences. The institutional review board approved conduct of this research.
The PBPS instrument
The PBPS operationalizes EID in Seidman’s (1996, 2005) retention formula and facilitates implementation of Wells (2003) epidemiological framework for primary (early identifying vulnerable students), and secondary (early identifying students entering the attrition process) diagnostic and prescriptive strategies for averting AASE. Individualized confidential PBPS Reports present each student’s overall and categorized risk totals, identify each student’s risks, and describe interventions or resources for amelioration of identified risks.
Example PBPS risk item stems
Enter the number of languages other than English spoken in your home
The youngest age child under your supervision
Some students in graduate or professional degree programs leave school before receiving a degree. If this happens to you, what would be the most likely cause(s)?
I expect to be able to maintain a “B” (3.0) average or above
My family has always wanted me to go to professional or graduate school
I will require significant financial aid during the completion of my education program
Did you have the opportunity to lead or guide others in accomplishing activities and goals?
Did you experience some form of discrimination during your pre-professional or undergraduate education?
List three offices held or groups you had membership in college or in your community (organizations, memberships, class, local, state or national offices, etc.)
Long range goals
Once I start something, I am determined to finish it
PBPS total risks have previously demonstrated consistent, statistically significant, and substantial predictive validity for the occurrence of 1st- and 2nd-year AASE among populations of nursing students in two diverse nursing schools (Johnson et al. 2009b). Reliability of the 47 dichotomously scored 2004 PBPS noncognitive risk items was previously assessed using the SPSS 13 RELIABILITY procedure, yielding Cronbach α = .77 (Johnson et al. 2009b). The SPSS 15 RELIABILITY procedure yielded Cronbach α = .75 for the 526 respondents to the 64 risk items of the 2005 PBPS and Cronbach α = .80 for an additional 589 new 2006 matriculates enrolled from the five schools and four disciplines of the present study. Students completed the PBPS in approximately 30–40 min.
For the binary AASE criterion variables, AASE occurrence was defined, consistent with Johnson et al. (2009b), as the presence of any of the following adverse academic status event codes in a student’s record during an academic year: (DN) dismissal due to academic record, (FW) fall semester warning, (IG) insufficient grade(s), (LOA) leave of absence, (PT/AP) switched to part time/alternate pathway, (RA) remedial assistance, (RC) repeat of course(s), (RY) repeat of year, (SW) spring semester warning, (SP) suspended from program, (WC) withdrew from course, or (WD) withdrew from program. These 12 academic status codes operationalized precursors or types of student attrition. AASE nonoccurrence was defined as the absence of any of the above AASE occurrence codes during an academic year. AASE nonoccurrence was comprised of two additional academic status codes, (GS) good standing and (SG) student graduated, operationalizing categories of student retention.
Standardized PBPS scores
To relate AASE occurrence to standardized risk scores, 2004 and 2005 PBPS total risks were converted to z-scores. Standardized PBPS total risks enabled more meaningful comparisons for the 2004 PBPS and 2005 PBPS and have a mean of zero, resulting in more meaningful interpretations especially in cases with significant interactions (Jaccard 2001).
Logistic regression analyses
All analyses were conducted using the SPSS 15 LOGISTIC REGRESSION procedure. Significance tests confirmed no violation of logit linearity assumptions. Binary logistic regression assessed the predictive validity of PBPS total risks against the criterion of occurrence of students’ 1st- and/or 2nd-year AASE while assessing and statistically controlling for effects associated with school affiliation (dentistry, medicine, nursing A, nursing B, etc.) and URMS status (URMS, nonURMS). These two factors were statistically controlled because both were previously found to be significant predictors of PBPS total risks (Johnson et al. 2009a). Four hierarchical, two-block, binary logistic regression analyses conducted significance tests following the methodology described by Johnson et al. (2009b). Each 1st-block contained the main-effects predictors: standardized PBPS total risks, URMS status, and school affiliation. Each 2nd-block of predictors contained all two-way and three-way interactions. For the first three analyses, standardized 2004 PBPS total risks served as one of the three main-effects predictors, with 2004 matriculates’ 1st-year AASE, 2nd-year AASE, and 1st- or 2nd-year AASE, successively, as the criterion variable. The fourth analysis employed standardized 2005 PBPS total risks as one of the three predictors, with 2005 matriculates’ 1st-year AASE as the criterion variable. Analyses employed Fisher’s “protected-t” procedure as recommended in the regression context by Cohen and Cohen (1983, pp. 172–176) for alpha level inflation control.
Standardized PBPS total risks consistently predicted adverse academic status events
Predictive validity for AASE
Controlling for both school affiliation and URMS status, standardized PBPS total risks consistently and significantly predicted 2004 matriculates’: (1) 1st-year AASE occurrence (see Table 3), OR = 1.35, p = .02, 95% CI = (1.04, 1.74), (2) 2nd-year AASE occurrence, OR = 1.42, p = .01, 95% CI = (1.08, 1.86), and (3) AASE occurrence during either 1st or 2nd year, OR = 1.36, p = .02, 95% CI = (1.05, 1.75); as well as 2005 matriculates’ 1st-year AASE occurrence, OR = 1.43, p = .002, 95% CI = (1.14, 1.78). Absence of significant interactions indicated magnitudes of the odds ratios above did not vary significantly with either school affiliation or URMS status.
Observed AASE and PBPS risks
First-year AASE occurred for 24% (odds = .32) of the 2004 research participants and 26% (odds = .36) of the 2005 participants, while 2nd-year AASE occurred for 24% (odds = .32) of the 2004 participants. The mean PBPS total risks and standard deviation (M = 9.30, SD = 4.30, N = 441) of the 47 risk items for the 2004 PBPS were less than the mean and standard deviation (M = 14.40, SD = 5.28, N = 526) of the 64 risk items for the 2005 PBPS.
Sensitivity, specificity, positive and negative predictive values
Standardized PBPS total risks, school affiliation, and URMS status together consistently predicted AASE status at the end of the 1st or 2nd year for approximately 70–76% of students (see Table 3). With decision thresholds for predicting AASE for the 2004 and 2005 matriculates set at their 1st-year observed prevalence levels of .24 and .26, respectively, sensitivities (Se) ranged from approximately 54 to 79%. Specificities (Sp) ranged from approximately 73 to 80%. Positive predictive values (PPV) ranged from approximately 42 to 60%. Negative predictive values (NPV) ranged from approximately 82 to 92%. Nagelkerke R2 for the three predictors ranged from .17 to .32.
Controlling for effects associated with school affiliation and URMS status, predicted AASE odds for students one standard deviation above the PBPS mean ranged from 135 to 143% (1.35–1.43 times) those of students at the mean for their 1st or 2nd year (see Table 3). Because odds ratios constitute a multiplicative factor that works like compound interest, these results indicate predicted AASE odds for students one standard deviation above the mean ranged from 182 (1.352 = 1.82 times) to 205% (1.432 = 2.05 times) those of students one standard deviation below the mean for their 1st or 2nd year.
URMS’ predicted AASE
Because odds ratios for the three predictors were statistically independent, predicted odds of 2nd-year AASE for URMS one standard deviation above the mean 2004 PBPS total risks reached 494% (3.48 × 1.42 = 4.94 times) those of nonURMS at the mean, controlling for school affiliation. For the 2005 PBPS, predicted odds of 1st-year AASE for URMS did not differ significantly from those of nonURMS, controlling for both PBPS risk and school affiliation.
Consistent significant (p < .05) predictive validity of the PBPS for AASE extended beyond the populations of nursing students surveyed by Johnson et al. (2009b) to broader populations of diverse health science professions schools in multiple disciplines. PBPS predictive validity did not vary significantly across dental, medical, nursing, or biomedical science schools, or with URMS status. The PBPS consistently and significantly facilitated early identification of AASE-risk dental, medical, nursing, and biomedical sciences URMS and nonURMS and their degrees of risk, enabling proactive targeting of interventions for risk amelioration and AASE or attrition prevention. The PBPS consistently demonstrated high reliability, and significant (p < .05) and substantial predictive validity for 1st- or 2nd-year AASE with 433 new matriculates from four health science professional schools in three disciplines and, the following year, with 516 new matriculates from five health science schools in four disciplines. Each standard deviation increment in PBPS total risks consistently multiplied the odds of 1st- or 2nd-year AASE from 135 to 143% (p < .05), controlling for URMS status and school affiliation. This means AASE odds one standard deviation above PBPS mean ranged from 182 to 205% of those one standard deviation below mean.
For those responding to the 2004 PBPS, URMS status independently more than tripled the odds of 2nd-year AASE. Consequently, 2nd-year AASE odds for URMS one standard deviation above the 2004 mean were 494% those of nonURMS at the mean, controlling for school affiliation. For those responding to the 2005 PBPS, URMS status did not significantly predict 1st-year AASE, controlling for PBPS risk and school affiliation.
Mammography, PSA, and PBPS sensitivity (Se), specificity (Sp), and positive predictive values (PPV)
Instrumentation (binary criterion)
Mammography (breast cancer occurrence)
Humphrey et al. (2002)
National Cancer Institute (2008)
PSA (prostate cancer occurrence)
National Cancer Institute (2008)
US Preventive Services Task Force (2008)
PBPS (AASE occurrence)
The scope of PBPS use has recently been expanded to a number of additional health science professional schools and disciplines, including health information sciences, and additional nursing schools. The PBPS is being used in these schools to help advisors, mentors, or counselors diagnose student risks and prescribe interventions. Faculty, Deans, administrators, and counselors from other health science professional and graduate schools have inquired about using the PBPS. Implementation of the PBPS in similar schools and disciplines has been straightforward. Further research questions and issues concerning generalizability to a broader range of institutions are being investigated with these schools. A plan to expand the scope of use to additional institutions includes development of cost analysis, institutional negotiation, and licensure agreement forms.
Issues or constraints associated with use in other schools and disciplines range from who, when, and how the PBPS is administered, to who will receive the confidential individualized reports subsequent to student authorization. Online administration has been considered, but issues regarding online response rates must be addressed before the 99% + response rates currently achieved offline can be approached. An alternate version of the instrument with some adaptations has been implemented in a preentry program for students needing assistance prior to matriculation into medical school. Occasionally, some institutions have expressed reluctance and skepticism concerning the need for and ability to address noncognitive risks.
The PBPS is the first diagnostic and prescriptive instrument developed specifically to facilitate prevention of AASE, including nonadvancement and attrition, among health science professions students. The present research demonstrates that the PBPS can facilitate early identification of health science professions URMS and nonURMS having AASE-risk disadvantages. Individualized PBPS Reports not only quantify PBPS risks, but identify specific risks, and corrective interventions for prevention of AASE. Further research is needed to demonstrate whether interventions introduced using the PBPS can reduce AASE or attrition for URMS and nonURMS, reduce costs to health science higher education institutions, or positively impact shortages and diversity of health science professionals.
This project was supported by a grant from the Texas Higher Education Coordinating Board Minority Health Research and Education Grant Program.