The role of letters of recommendation in perpetuating or challenging the social stratification of American secondary schools: a quantitative analysis of admission officer assessments in highly selective college admission

Abstract

Admission to highly selective institutions offers a pathway to upward social mobility, particularly for low-resource students. However, entrenched wealth disparities in the United States present serious challenges for admission officers at highly selective institutions. Utilizing individualized holistic review (IHR), including letters of recommendation (LOR), and contextual consideration, highly selective institutions aim to account for wealth inequities. This quantitative analysis, conducted at a highly selective research university, examined 95,326 applicants across 4 years to explore if admission officers’ internal assessments of LORs relate to contextual independent variables beyond students’ control. Statistical methods, including ANOVA, Two-Way ANOVA, Chi-Square, multiple linear regression, and logistic regression, were employed, controlling for standard measures of academic readiness. Findings revealed higher LOR assessments for students facing greater contextual challenge levels, underscoring the ability of the admission process to identify high achieving students from high challenge contexts. Paradoxically, elevated LOR ratings were also observed for students from private and smaller schools, indicating a nuanced interplay of factors in the admission process. Importantly, LOR assessments significantly correlated with admission outcomes, emphasizing the pivotal role of LORs in highly selective college admissions. This study reveals the nuanced role of LORs in admissions, exposing their potential to tackle inequities. It calls for improved IHR training and heightened consideration of contextual variables, offering a crucial contribution to advancing equity in highly selective institutions, fostering more informed admission practices.

1 Introduction

Highly selective private universities in the United States are grappling with escalating challenges, as soaring application volumes collide with limited spaces, prompting increased scrutiny of their admission practices [15, 44, 88]. Within this narrow, but popular landscape, concerns regarding equity in admission processes have garnered significant attention, encompassing dimensions such as athletics, legacy, gender, race, and standardized testing [6, 21, 52, 88, 92]. This discourse not only challenges the multiple institutional missions but also reflects broader societal conversations on access to higher education and social mobility [43, 53].

Against a backdrop of socioeconomic stratification, disparities in income distribution have presented clear challenges to higher education institutions [86]. Such inequalities are further compounded within the realm of public school education, where income-driven segregation persists, impeding equitable access to essential academic resources [69, 76, 77, 87]. This dramatic exacerbation of socioeconomic divisions directly impacts the composition of student bodies at America’s most highly selective colleges, with students from affluent backgrounds disproportionately represented [4, 22]. In response to these equity challenges, institutions have increasingly relied on mechanisms such as Individualized Holistic Review (IHR). IHR is a methodology which evaluates an applicant’s qualifications in context by considering academic achievements, extracurricular activities, and personal qualities, aiming to assess the whole person rather than just quantitative metrics, and incorporating qualitative data like letters of recommendation, personal statements, and interviews to promote a more inclusive and comprehensive assessment [18, 24, 54, 79]. IHR, while comprehensive and context-inclusive, introduces the potential for bias through the subjective interpretation of narrative information. Serious concerns persist regarding inherent biases within the IHR process, particularly in the subjective assessment of LOR, which may inadvertently favor candidates from privileged backgrounds [39, 81].

This investigation contributes to the evolving scholarly discourse by examining the intricate interplay between admission officers’ evaluations of LOR and contextual factors such as school size, type, and measures of inequality. Utilizing a quantitative research approach and drawing data from a highly selective research university, the study seeks to elucidate any discernible impact of contextualized recommendation assessments on admission outcomes. By empirically investigating this relationship, this research aims to offer valuable insights into the complexities of admission processes within the landscape of selective higher education, shedding light on potential avenues for enhancing fairness and equity in admissions practices.

2 Literature review

The increasing selectivity of the most recognized institutions in the United States presents a dual challenge to societal equity, exacerbating and facilitating social mobility simultaneously [80, 87]. Renowned for academic rigor and association with affluent demographics, these institutions are less representative of lower socioeconomic status (SES) students, who often undermatch to lower-tier institutions despite academic potential [8, 16]. Persistent access barriers, exacerbated by information gaps, contribute to underrepresentation of low-SES students in private colleges [30]. And, despite attempts at interventions, policy adjustments have shown limited increases in low-SES enrollment [7, 22]. Addressing wealth stratification in secondary schools poses challenges for selective colleges aiming to promote social mobility, as Pell-eligible enrollment remains stable despite fluctuations in financial aid policies [40]. Bastedo and Flaster’s research challenges problematic assumptions in undermatching research such as researchers' ability to pinpoint impactful colleges, predict admission outcomes, and use SAT scores to reduce postsecondary inequality. [8]. Providing contextual information enhances the likelihood of recommending low-SES students, while the impact of maximizing course rigor on admission outcomes varies [6]. Furthermore, Bastedo et al. [5] identify three distinct interpretations of holistic review among admission officers, highlighting the complexities in implementing Individualized Holistic Review (IHR) across institutions.

2.1 Variables and individualized holistic review

IHR, assessing both academic and non-academic factors, prioritizes elements like high school GPA, course rigor, and standardized test scores as they are crucial predictors of academic success, particularly in selective institutions [7, 81, 82, 86]. Non-academic factors which admission officers consider include student engagement, essays, writing samples, interviews, talents, and recommendations [24, 79]. Despite subjectivity, these qualitative elements offer an “air of objectivity” through uniform ratings by admission officers, tailored to institutional missions [24, 47]. While personal statements are common, their predictive validity beyond grades and test scores is limited [62]. Non-academic personal qualities considered may encompass personality, competencies, performance, attitudes, and learning skills [41, 51] though subjective assessments may introduce biases, favoring student-athletes and exhibiting modest gender bias and moderate interrater reliability in personal statement assessments [2, 45, 68]. Regardless of what factors are considered, underpinning IHR is the consideration of student context [24].

2.2 Letters of recommendation

The inclusion of LOR as part of the American college admission process dates back over 100 years to at least 1913 [9, 10]. That said, LOR have been in existence for over 2000 years and are documented first as a part of Cicero’s writing in 46 BC, the term litterae commendaticiae [26]. Today in the selective college admission process, their role is understood as consistent over time, of moderate importance, and of greater importance at highly selective institutions [20, 79, 81]. Public narrative on LOR is present in the media, op-eds, books, guides, and annual reports, yet peer-reviewed research on LOR is limited though gradually emerging. Given their sensitive and highly confidential nature, access to information is likely a key challenge with research on LOR.

2.3 Unveiling the duality: criticism and value of LOR

From the extant literature, three central criticisms of LOR are their open, unstandardized nature [29, 42, 50], the variation in the context of the writer [19, 55], and bias from the writer, the reader, or both [1, 29, 34, 74, 83].

The absence of standardized LOR raises concerns, with studies indicating that standardized structures can reduce content disparities [31, 75]. However, non-standard LORs may perpetuate inequality [29], as their content varies widely due to the open nature of platforms like the Common Application [38]. The lack of enforced standardization is a persistent challenge in the decentralized college admission system [42]. The significant variation in the experiences and resources of letter writers, particularly in the context of private and public schools, presents inherent challenges [19, 55]. Inherent bias in LORs, influenced by limited knowledge of students, contributes to stereotypes and exhibits gender and race-based differences in assessments [1, 34]. Mitigating bias is addressed in recent admission models like committee-based evaluation (CBE) involving multiple application readers [78, 80]. Despite these challenges, the debate over the value of LORs continues among senior admission leaders, with countervailing arguments emerging to the upside as well.

The literature notes three central values of LORs, in that they provide narratives beyond measures of academic readiness such as GPA and standardized test scores [70, 74, 91], aid with the verification of information by cross-referencing and validating application information from multiple sources [79], and they allow admission officers to better situate the student in their respective context [23, 59].

LOR extend beyond academic readiness, providing valuable insights into qualitative and non-cognitive aspects of students, assessing qualities such as intellectual engagement, creativity, and potential for growth [50, 70, 79, 91]. Admissions professionals value LORs for understanding these dimensions, especially in cases where applicants share similar academic credentials, allowing committees to focus on personal statements, recommendations, and interviews [79]. LORs also play a role in verifying information in applications, with greater reliability when sourced from in-network institutions or writers, ensuring confidence in the admission process [79, 83]. Additionally, LOR are paradoxical, since they are to be situated in context but also share information about student context. LOR help admission readers in understanding special circumstances and providing a complete sketch of the student [58, 59].

2.4 Large-scale empirical studies

Large-scale research on LOR yields varied findings. They show weak but positive correlations with academic readiness indicators like GPA and test scores [50]. Race and gender differences in LORs exist in undergraduate admission, with small but significant variations in language and length [1]. Gender bias appears in LORs for male-dominated faculty positions [57]. Text analyses reveal that certainty of productivity correlates with favorable selection outcomes, while positive emotion with words of discrepancy and tentative statements results in fewer research internship offers [42]. A recent analysis of 31,000 +LORs found content differences by gender, race, and gender-race intersections, but content beyond GPA/test scores couldn’t predict admission outcomes [29]. Kim et al. [48] used advanced natural language processing to analyze how counselor recommendation letters from over 600,000 college applications reflect demographic differences, revealing significant disparities in length and content related to race, socioeconomic status, and school type, and underscores the need for context-sensitive evaluations in admissions processes.

These findings underscore the intricate forces at play within the realm of selective admissions processes, prompting an examination through the lens of theoretical frameworks. In particular, the application of Social Reproduction Theory offers insight into how institutional practices perpetuate socioeconomic inequalities, thereby framing the subsequent analysis within a broader structural context.

3 Sociological and theoretical framework

French sociologist and public intellectual Pierre Bourdieu developed the theory of social reproduction (SRT) within sociology. Specifically, within sociology, SRT explores access to different forms of social capital. The access and role of social capital or an educational currency are relevant to this literature review. Social capital is “made up of social obligations (‘connections’), which is convertible, in certain conditions, into economic capital and may be institutionalized in the form of a title of nobility” [12] (p. 243). Further, social capital is “the aggregate of the actual or potential resources which are linked to possession of a durable network of more or less institutionalized relationships of mutual acquaintance or recognition” [12] (p. 248). Through this lens, higher education is a currency type, and higher education institutions are the locus of access to such currency. Bourdieu defined social reproduction as a theoretical framework to analyze the role of educational institutions and other social sites in perpetuating dominant cultures [33].

SRT identifies the act of schooling as a “primary means” for perpetuating dominant class ideologies and values [89] (p. 1). Additionally, SRT seeks to bring to the forefront educational processes which tend “to ensure the transmission of cultural capital across generations and to stamp pre-existing differences in inherited cultural capital with a meritocratic seal of academic consecration by virtue of the special symbolic potency of the title (credential)” [14] (p. ix). Therefore, access to higher education as a schooling stage may bestow and withhold social capital from individuals within a society. The key questions are (a) how do members of a complex society access social capital, and (b) do existing educational processes account for or promote the inheritance of the social stratification endemic within the schooling system in the United States? LOR have been shown to lack structure and engage ambiguous criteria [29, 81]. Bourdieu theorized that ambiguous evaluation criteria benefitted the privileged classes:

It is quite clear that the more vaguely what they ask for is defined, whether it be a question of knowledge or of presentation, and the less specific the criteria adopted by the examiners, the more they favour the privileged [emphasis added].” [13] (p. 40).

Through this lens, LOR are a mechanism by which institutions, and their members, bestow and transfer social capital to other institutions. It is a way to move from one in-group to another in-group with sanctioning by respected members. In applying SRT to this investigation, LOR can be understood as a form of social capital which can be transferred between parties. Students seeking admission to the most exclusive institutions nationwide seek social and educational capital from those in positions of authority—the teachers and school counselors. In a way, LOR can be understood as a currency bestowed upon certain students at a range of values. Some students will receive high value and praise, while others might receive low value and criticism. Next, consider if high-resource communities possess a specific social capital that exchanges readily with the most selective institutions. Then, consider if the lowest resource communities possess an equally valued social capital. Or, might these be different currencies entirely? Ideally, IHR should act as a counterweight to the theory of social reproduction. Finally, SRT highlights how social structures and institutions can reproduce social inequality and calls for critically examining these structures to create a more equitable society. This research is precisely that—a critical examination.

4 Purpose statement

The purpose of this research is to quantitatively investigate the correlation between admission officer evaluations of student letters of recommendation and various contextual factors, including school size, school type, and indicators of disadvantage as provided by the College Board Landscape^™ Indices [25]. Utilizing data from a single, highly selective research university, the study aims to determine whether a relationship exists between the ratings given to letters of recommendation by admission officers and the contextual variables of the students’ schools. Additionally, the research explores the extent to which these ratings influence the final admission decisions.

5 Research questions

• 1. Is there a relationship between school type and the LOR Rating assigned by admission officers?

• 2. Is there a relationship between school size, Landscape^™ Indices, and the LOR Rating assigned by admission officers?

• 3. Is there a relationship between the LOR Rating and final admission outcome, holding constant for measures of academic readiness?

6 Methodology

This research study leveraged a quantitative method. A quantitative methodology is used to understand the relationship between a specific dependent and a series of independent variables in a data set. A quantitative study’s results use inferential statistics, extensive descriptives, statistical significance testing, confidence intervals, and effect sizes [17, 27, 28, 35].

During the admission process, the institution under investigation assigns multiple ratings on a one to five scale, where five is the strongest rating, and one is the weakest. The ratings are Likert scales. The investigation centered on the internal rating assigned by admission officers to the Letters of Recommendation (LOR Rating) as the dependent variable (DV). It is common for admission offices to assign specific ratings for subjective assessments based on uniform criteria [47, 78, 79, 81]. Admission officers interpret qualitative elements and translate these into a unique sequence of ratings that the admission committee members understand across the entire applicant pool. Assigning individual ratings to particular elements of an application helps admission officers at highly selective institutions distinguish students beyond quantitative academic metrics such as SAT/ACT scores and GPAs [78]. The DV is one such variable which is a quantitative assignment based on a qualitative rubric.

6.1 Sample

With approval from the institution, de-identified data from a single highly selective research university was used for the sample. The sample size was 95,326 valid cases. Valid cases were students attending high schools in the United States and who are U.S. Citizens or U.S. Permanent Residents. The institution under investigation only offers fall enrollment and only first-year applicants were under investigation. Transfer students were excluded.

The institution is located in an urban area with highly reputable undergraduate programs coupled with graduate programs in the professions, including business, medicine and law. The institution has an undergraduate admission rate of approximately 15%. Across all colleges enrolling first-year students, the enrollment is approximately 1900 students. All applicants sampled are applicants to a single college within the university: the College of Arts & Sciences.

6.2 Participant characteristics

In the data set, 60,287 applicants are female, and 35,039 are male. Over time, the gender distribution in the four applicant pools from 2019 to 2022 was similar. The pool consisted of students who were White (33.7%), Asian (32.7%), Black (17.4%), Hispanic (10.5%), Native (0.3%) and 5.3% were unknown. Applicants came from a wide variety of geographic areas across the United States. Regions were broadly defined into five geographic areas: Northeast, Mid Atlantic, Southeast, Central States/Midwest, and Mountain/West. In the data set, the greatest number of applicants were from the Southeast (23%), nearly matched by the Northeast (22.9%). Central/Midwest states (19.3%) were the next most represented, followed by the Mountain/West (17.1%). The Mid Atlantic was the least represented (16.8%) in the applicant pool while 957 (1%) applicants had an unknown region.

School type was the categorical, independent variable under investigation. It is the dependent variable for the first research question. Within the data set, public high schools comprised 65.5% of the valid cases, private high schools (14.1%), parochial schools (8.8%), and home school (0.3%) students, all together, made up a majority of the applicant pool. For 10,812 applicants, the high school type was not disclosed, which was 11.3% of the applicants in the data set. Table 1 reveals the mean SAT score of applicants is 1425, and the mean GPA of applicants is a 3.71.

Table 1 Descriptive Statistics: Mean, Standard Deviation, Skewness, and Kurtosis

6.3 Sampling procedures

The data set combined four applicant data sets from independent admission cycles (Fall 2019, Fall 2020, Fall 2021, and Fall 2022) to produce a robust sample size. Using 4 years of data accounted for year-over-year abnormalities. Additionally, as covered later, using 4 years of data provided 2 years of pre-pandemic data and 2 years during the COVID-19 pandemic.

6.3.1 Instrumentation

In this research study, three distinct application types were used: the Common Application, the Questbridge Application, and the Coalition Application. Each of these application platforms was considered acceptable for the purposes of the investigation, providing a comprehensive means of collecting data from participants. Each application was used to gather detailed information on the applicants’ academic records, personal backgrounds, extracurricular activities, LOR, and personal statements.

6.4 Measures

The dataset for this study includes a variety of variables, each serving specific roles in the analysis. The Start Term variable is categorical and nominal, indicating the term of study start (F2019, F2020, F2021, F2022). Race Recoded is also categorical, classifying participants into six categories: Asian, Black, Hispanic, Native, Unknown, and White. The Sex variable is categorical and dichotomous, denoted as M (male) or F (female). The decision outcome variable, which serves as one of the dependent variables, is categorical and dichotomous, indicating whether a participant was admitted or not admitted. The Curriculum Rating variable, a scalar measure ranging from 0 to 5, assesses academic readiness. Another independent variable, GPA is a continuous variable ranging from 0.00 to 4.00, serving as a measure of academic readiness. Similarly, Max Testing (SAT) is a continuous variable, with scores ranging from 400 to 1600, used to measure academic readiness.

Academic Rating is a variable with categories AAA, AA, A, B, C, and Other, providing another measure of academic readiness. The Academic Rating is a comprehensive categorical variable that combines three key measures of academic readiness: Curriculum Rigor, GPA, and Standardized Testing. AAA are the most academically ready, while C are the least academically ready. The most frequently occurring Academic rating is the ‘A’ rating (23.6%), then the ‘AA’ rating (18.2%), the ‘C’ rating (12.8%), the ‘B’ rating (10.8%), and finally the ‘AAA’ rating (8.5%). Just over 26% (26.1%) of applicants received the ‘Other’ rating as they were missing one of the measures of academic readiness. In many cases, this was likely due to a missing standardized test score due to the COVID-19 test-optional policy.

HS Region is categorical, indicating the high school region (Northeast, Southeast, Mid-Atlanta, Midwest, West). HS Type is categorical, identifying the type of high school attended (Parochial, Private, Public, Homeschool, None Listed), and serves as an independent variable. School Class Size is a numeric continuous variable, ranging from 1 to 1800, and is also an independent variable. The Landscape^™ HS Index and Landscape^™ NH Index are numeric continuous variables, each ranging from 1 to 100, used as independent variables. These indices, namely Landscape^™ High School and Landscape^™ Neighborhood, were developed by the College Board to synthesize contextual variables for admission officers, where a 100 index shows high contextual challenge, and 1 represents low contextual challenge. The indices incorporate publicly available data, such as high school locale, percent of students on free and reduced lunch (FRPL), college attendance, household structure, median family income, housing stability, education level, and crime [26].

The dependent variable in this study was the LOR Rating. The rating is on a one through five Likert scale, where five is the strongest rating. This variable is scalar, and the distance between each rating is understood to be uniform. A rating distribution is encouraged to the admission officers; however, the interpretation of the scheme remains essential. Admission officers read applications geographically, assigning this specific rating to each student in a designated territory. Once the admission officers assign this rating, it does not change for the application's life cycle. The same admission officer assigns the LOR Rating for all students within a designated geographic region. The institution appoints the same admission officer to all applicants within the same high school. Each application includes three LOR: one from a guidance counselor and two from academic teachers. The rating synthesizes the three letters and summarizes school support for the candidate.

The mean LOR Rating was 2.71, the range is one to five, with a standard deviation of 0.638. The LOR Rating frequency showed a central tendency concerning skewness (0.28), n = 76,361. The variance was 0.407. The kurtosis statistic was 0.00 with a standard error of 0.018. Three was the most frequently occurring LOR Rating (54.3%), followed by two (36.5%), then four (8.1%), then one (0.7%), and finally five (0.4%). With respect to other variables used, descriptives are shown in Table 1.

6.5 Procedures

The data format exported into Excel. The data analysis used SPSS v.28. Despite year over year data sets, applicant data sets have identical data dictionaries and variables and can combine with integrity. All data were de-identified. The institution stores admission data in a secure (dual-authenticated login required) database that can regularly produce reports. The data used were ‘end of cycle’ data. These data were pulled from the end of the admission cycle after admission decisions are released, decisions have been finalized, and the data were now static for all applicants. In this data set, applicant data was retrieved at the end of the cycle. Data sources were raw, and data management was necessary to ensure the integrity of the data. Missing data is one of the most pervasive challenges in data analysis [93]. In this analysis, cases with missing data were either excluded listwise (removal of the entire case) or pairwise. Pairwise deletion included the observations with missing values into the statistical analysis; however, when the analysis involved pairs of values, the pairs with missing values were removed. This prevented a loss of usable data.

6.6 Research design and analyses

The first research question investigates whether there is a relationship between the type of high school and the letter of recommendation (LOR) rating assigned by admission officers. In this analysis, the dependent variable (DV) is the LOR rating. The independent variables (IVs) are the type of high school, a categorical variable, and the academic rating to control for measures of academic readiness. To address this research question, descriptive statistics will be used to summarize the data, followed by inferential statistical tests including ANOVA and Two-Way ANOVA to explore interactions between the high school type and academic rating, and Chi-Square tests to examine categorical relationships.

The second research question examines whether there is a relationship between school size, various Landscape^™ indices, and the letter of recommendation (LOR) rating assigned by admission officers. In this analysis, the dependent variable (DV) is the LOR rating. The independent variables (IVs) include school class size, the Landscape^™ high school (HS) index, the Landscape^™ neighborhood (NH) index, GPA, and maximum SAT scores, all of which are continuous variables. Additionally, the curriculum rating is included as a scalar variable, and the high school region, race (recoded), and sex are categorical variables. To address this research question, descriptive statistics will be utilized to summarize the data, followed by multiple linear regression analysis to explore the relationships between the dependent variable and the set of independent variables.

The third research question investigates whether there is a relationship between the letter of recommendation (LOR) rating and the final admission outcome, while controlling for measures of academic readiness. In this analysis, the dependent variable (DV) is the final admission outcome, categorized as either “Non-Admit” or “Admit,” making it a dichotomous variable. The independent variables (IVs) include the academic rating and the LOR rating. Additional independent variables include high school region, race (recoded), and sex, all of which are categorical variables. To address this research question, descriptive statistics will be used to summarize the data. Correlation analysis will be conducted to examine the relationships between variables, and logistic regression will be employed to determine the predictive power of the LOR rating and other factors on the admission outcome, while controlling for academic readiness.

6.7 Ethics statement and data availability statement

The data utilized in this research is not publicly available due to privacy and confidentiality considerations. Specifics requests for data access can be addressed to the author of the investigation. Data has been released with the assurance of institutional anonymity and individual anonymity. All necessary steps have been taken to deidentify the data, ensuring the protection of individuals’ privacy and adherence to ethical standards. The study did not require IRB review because it met requisite standards, is not research with “human subjects”, nor is it a “clinical investigation” as defined in the federal regulations. Finally, informed consent does not apply to this investigation as it solely utilizes anonymized data at both the institutional and individual levels, ensuring confidentiality and privacy.

7 Findings

The study assumes a positive and statistically significant relationship between LOR Rating and academic readiness measures, including course rigor, GPA, and standardized testing. Analysis, employing Pearson Correlation coefficients, reveals a statistically significant but low positive correlation [r(76,358) = 0.233, p < 0.01] with Curriculum Rating, a moderate positive correlation [r(76,356) = 0.301, p < 0.01] with GPA, and a low but statistically significant positive correlation [r(62,954) = 0.270, p < 0.01] with standardized testing. These findings suggest the need for acknowledging and, where possible, controlling for academic readiness measures in addressing research questions.

7.1 Research question 1: is there a relationship between school type and the LOR rating assigned by admission officers?

This question investigated the role of school type and its relationship with the LOR rating. This section shares descriptive statistics before conducting a preliminary ANOVA, followed by a two-way ANOVA to control for measures of academic readiness. The independent variable was the LOR Rating assigned by admission officers, and mean LOR Ratings by school type were calculated. Table 2 displays the mean LOR Rating by the six Academic Ratings, while Table 3 displays the mean LOR Rating by high school type.

Table 2 Descriptive Statistics for Academic Rating and LOR Rating

Table 3 Descriptive Statistics for School Type and LOR Rating

Table 4 captures the distribution of the LOR Rating across the varying High School Types in the data set.

Table 4 Descriptive Statistics: Count of LOR Rating by High School Type

An initial ANOVA was computed before accounting for the Academic Rating of each applicant. For this analysis, using an ANOVA was appropriate as there were five categories of high school type (including no school type listed). The results of the ANOVA are displayed in Table 5.

Table 5 ANOVA for LOR rating

An ANOVA compared mean LOR Ratings among different high school types, revealing a significant difference [F(4, 76,356) = 266.038, p < 0.001]. R² indicates that 1.4% of the variance in LOR Rating is attributed to School Type. Levene’s Test for Homogeneity of Variances (p < 0.001) will be discussed in limitations. Tukey’s HSD post-hoc test identified differences; public high school students (M = 2.67, sd = 0.635) received lower ratings than private (M = 2.87, sd = 0.630), parochial (M = 2.70, sd = 0.631), or unspecified (M = 2.78, sd = 0.634) school types. Public school students had higher ratings than homeschool students (M = 2.52, sd = 0.545). Rankings from highest to lowest LOR Ratings were private (M = 2.87, sd = 0.630), unspecified (M = 2.78, sd = 0.634), parochial (M = 2.70, sd = 0.631), public (M = 2.67, sd = 0.635), and homeschool (M = 2.52, sd = 0.545). Confidence intervals at 0.001 were statistically significant for all mean differences.

7.1.1 Two-way ANOVA

A Two-Way ANOVA is used when the research question investigates two or more independent variables on one dependent variable. In this research question, the investigation explores the relationship, and potential interaction, of applicant high school type and Academic Rating on the LOR Rating. The previous ANOVA revealed a statistically significant relationship between the High School Type and LOR Rating. The two-way ANOVA controlled for the Academic Rating of the students. For this analysis, a Two-Way ANOVA was conducted as there were five categories of high school type (including no school type listed) and six Academic Ratings. The results are shown in Table 6.

Table 6 Two-Way ANOVA Results: Tests of Between-Subjects Effects

A two-way (5-School Type X 6-Academic Rating) ANOVA was calculated to compare the mean LOR Ratings for applicants who attended one of five high school types and had one of six academic ratings. The two-way ANOVA revealed that there was a statistically significant interaction between the effects of high school type and the Academic Rating [F(20,76,331) = 6.176, p < 0.001]. The effect size (η²) was 0.002. There was a weak, positive, statistically significant interaction between the effects of high school type and Academic Rating on the LOR Rating. The adjusted R-Squared is 0.129, or 12.9% of the variation in the LOR Rating is explained by its relationship with High School Type and the Academic Rating. Because there was an interaction effect between the independent variables (school type and academic rating), separate univariate tests must be undertaken to correctly understand the effect sizes of the high school type and the academic rating. The mean square error (0.355) was held constant from the Two-Way ANOVA for the univariate analyses. Tables 7 and 8 show the univariate analyses.

Table 7 Univariate Analysis: Effect of School Type on LOR Rating (DV)^a,b

Table 8 Univariate analysis: effect of academic rating on LOR rating (DV)^a,b

Concerning the applicant high school type, public [F(5,76,331) = 1436.42, p < 0.001], private [F(5,76,331) = 310.95, p < 0.001], parochial [F(5,76,331) = 188.33, p < 0.001] and no school type listed [F(5,76,331) = 82.55, p < 0.001] all showed a statistically significant effect on the mean LOR Rating (p < 0.001). Home school was not statistically significant [F(5,76,331) = 1.65, p > 0.01], but the four other high types were statistically significant (p < 0.001). The effect sizes varied depending on the applicant’s high school type. Public school applicants had the largest effect size (M = 2.67, sd = 0.635, η² = 0.086), then private schools (M = 2.87, sd = 0.630, η² = 0.020), then parochial (M = 2.70, sd = 0.631, η² = 0.012), and then no school listed (M = 2.78, sd = 0.634, η² = 0.005).

Concerning the Academic Rating, all six levels of the academic rating had a statistically significant effect on the mean LOR Rating (p < 0.001). The largest effect was for students with Academic Rating ‘A’ (M = 2.68, sd = 0.592, η² = 0.007). Students with Academic Rating ‘AA’ (M = 2.88, sd = 0.604, η² = 0.004) and ‘Other’(M = 2.60, sd = 0.624, η² = 0.004) had similar effect sizes, while students rated ‘B’ had the next smallest effect size (M = 2.55, sd = 0.579, η² = 0.003). The smallest effect size was on students who were either ‘AAA’ (M = 3.15, sd = 0.642, η² = 0.002) or ‘C’, (M = 2.35, sd = 0.570, η² = 0.002). All effect sizes were considered small, as they were less than 0.01. A post-hoc test (Tukey’s HSD) for pairwise comparisons was calculated to investigate the mean LOR Ratings between high school types by Academic Rating. This analysis revealed 33 of 60 possible interactions had a positive, statistically significant (p < 0.01) association between school mean LOR Ratings.

7.1.2 Results summary for research question 1

The preliminary ANOVA and Two-Way ANOVA analyses yielded significant insights into the data set. First, the interaction between Academic Rating and High School Type had a weak yet positive effect on applicant LOR Rating that was statistically significant [F(20,76,331) = 6.176, p < 0.001]. The effect size (η²) was found to be 0.002, and the adjusted R-squared value was 0.129, indicating that 12.9% of the variation in LOR Rating could be attributed to the relationship between High School Type and Academic Rating. Secondly, a significant relationship was observed between LOR Rating and the applicant’s High School Type (Private, Public, Parochial, No School Type Listed), with p < 0.001. When controlling for academic readiness, 33 of 60 possible interactions showed a positive, statistically significant (p < 0.01) association between school mean LOR Ratings. Out of these, 17 favored private school applicants, 11 favored those with no school type listed, and five favored parochial school applicants. However, none of these interactions were in favor of public high school applicants or those who were homeschooled. Finally, a small yet statistically significant (p < 0.001) relationship was found between all levels of the Academic Rating and LOR Rating.

Using both the preliminary ANOVA (LOR Rating by School Type), and the Two-Way ANOVA (LOR Rating by School Type by Academic Rating), the following table, Table 9, summarizes the total number of statistically significant differences in mean LOR Ratings.

Table 9 Count of statistically significant differences in mean LOR ratings by high school type

To further test if private schools and high LOR Ratings were independent of each other, a Chi-Square (X ²) was performed. Here, two dichotomous variables were transformed. For school type, the categories were private school and non-private school. For LOR Ratings, the categories were high LOR Ratings (4 or 5) and non-high LOR Ratings (1, 2, or 3). A significant X ² test result indicated that the two variables were not independent. A significant interaction was found [X ² (1) = 331.83, p < 0.001], and therefore the independence of high LOR Ratings and school type is rejected.

The mean LOR Rating of applicants from private schools (M = 2.87, sd = 630) was the highest across all school types. Results from the ANOVA indicated that when compared to the mean LOR Ratings of applicants from the four other school types, the mean LOR Rating of applicants from private schools was positive and statistically significant (p < 0.05). Furthermore, when controlling for academic readiness (Academic Rating) in the Two-Way ANOVA, applicants from private schools showed the highest number of positive and statistically significant (p < 0.01) mean LOR Rating differences between school types (17). Only one interaction showed the mean LOR Rating of applicants from private schools to be negative and statistically significant (p < 0.01). In this case, the mean LOR Rating of students with no school type listed was higher than that of applicants from private schools at the lowest Academic Rating (a C). Descriptive statistics revealed that applicants from private schools also received the highest percentage of LOR Ratings of either 4 or 5 (12.5%) and a significant interaction was found [X ² (1) = 331.83, p < 0.001] between school type and LOR Ratings in Chi-Square test of independence. Based on these findings, LOR Ratings and School Type have a significant relationship even when controlling for measures of academic readiness of applicants.

7.2 Research question 2: is there a relationship between school size, landscape^™ indices, and the LOR rating assigned by admission officers?

To comprehensively answer this question, multiple linear regression (MLR) was used as the statistical test to determine if a relationship existed. The results are shown in Tables 10 and 11 respectively.

Table 10 ANOVA from Multiple Linear Regression ^a,b,c

Table 11 Coefficients in multiple linear regression^a,b

A review of both Table 11 and Fig. 1 shows the mean LOR Rating decreased as the high school size increased, regardless of the Academic Rating earned by the student. That said, a more complex relationship appears with respect to the Landscape Indices, shown in Fig. 2.

Fig. 1

Mean LOR ratings by high school size decile and academic rating

Fig. 2

Mean LOR rating by composite landscape indices and academic rating

A multiple linear regression was calculated to predict the LOR Rating based on three independent variables under investigation: High School Class Size, Landscape High School Index, and Landscape Neighborhood Index. Six other independent variables were included: GPA, Max Test Score (SAT), curriculum rating, region, race, and sex. A significant regression equation was found F(9, 58,418) = 1051.02, p < 0.001), with an R² of 0.139. The adjusted R² was equivalent.

High School Size, Landscape High School Index, Landscape Neighborhood Index, Curriculum, GPA and SAT were significant predictors (p < 0.001) of the LOR Rating. First, there existed a significant relationship (p < 0.001) between High School Size (M = 361.61, sd = 256.23) and LOR Rating (M = 2.71, sd = 0.638). There existed a significant relationship (p < 0.001) between Landscape HS Index (M = 27.24, sd = 27.05) and LOR Rating (M = 2.71, sd = 0.638). There existed a significant relationship (p < 0.001) between Landscape NH Index (M = 27.41, sd = 27.67) and LOR Rating (M = 2.71, sd = 0.638).

An examination of the multiple linear regression results shows the Pearson correlation coefficients (zero-order) for each independent variable and the dependent variable, the LOR Rating. This Pearson correlation coefficient, however, did not control for the measures of academic readiness or variables of race, sex, and region. To understand the unique contribution of the independent variable under investigation, the semi-partial correlation coefficient was consulted.

It was found that High School Class Size significantly predicted the LOR Rating (β = 0.000, p < 0.001). When controlling for academic readiness and demographics measures, a significant relationship existed between High School Size (M = 361.61, sd = 256.23) and the highest LOR Ratings (M = 2.71, sd = 0.638). As class size increased, the LOR Rating decreased. The standardized coefficient was − 0.057.

It was found that Landscape High School Index significantly predicted the LOR Rating (β = 0.001, p < 0.001). Furthermore, when controlling for academic readiness and demographics measures, a significant relationship existed between Landscape High School Index (M = 27.24, sd = 27.05) and the highest LOR Ratings (M = 2.71, sd = 0.638). As the Landscape HS Index increased, the LOR Rating also increased. The standardized coefficient was 0.037. For every one standard deviation increase in Landscape Index HS (sd = 27 units), there was an expected increase of 0.037 standard deviations of the Landscape HS in LOR Rating (0.0236) while holding all other independent variables constant.

The Landscape Neighborhood Index significantly predicted the LOR Rating (β = 0.002, p < 0.001). Thus, when controlling for measures of academic readiness and demographics, a significant relationship existed between Landscape NH (M = 27.41, sd = 27.67) and the highest LOR Ratings (M = 2.71, sd = 0.638). As the Landscape NH Index increased, the LOR Rating also increased. The standardized coefficient was 0.067. For every one standard deviation increase in Landscape Index HS (sd = 27 units), there was an expected increase of 0.067 standard deviations of the Landscape HS in LOR Rating (0.0456) while holding all other independent variables constant.

7.2.1 Results summary for research question 2

The investigation used multiple linear regression analysis to investigate the relationship between the LOR Rating and three continuous variables—High School Class Size, Landscape High School Index, and Landscape Neighborhood Index. The results showed that High School Size, Landscape High School Index, Landscape Neighborhood Index, Curriculum, GPA, and SAT were significant predictors of the LOR Rating (p < 0.001). In total, the variables in the regression model explained 13.9% (R²) of the proportion of the variance for a LOR Rating.

In terms of high school class size, the data showed a weak, negative correlation, indicating a significant negative linear relationship between the two variables. Specifically, as class size increased, the LOR Rating decreased. Regarding both Landscape indices and the LOR Rating, the data showed a weak, positive correlation, indicating a significant positive linear relationship between the two variables. Even when controlling for academic readiness and demographic measures, the data showed a significant relationship between Landscape Indices and the LOR Rating. Specifically, as the Landscape Indices increased, the LOR Rating also increased.

7.3 Research question 3: is there a relationship between the LOR rating and admission outcome, holding constant for measures of academic readiness?

Descriptive statistics are shared to answer this question and as the outcome variable was dichotomous, admit or non-admit, a logistic regression was used. Two logistic regression analyses are conducted and compared. This first logistic regression model excludes the LOR Rating while the second model includes it.

7.3.1 Descriptive statistics

The admission status was the DV; therefore, these descriptive statistics examined the measures of academic readiness by admission status, admitted and not admitted. These descriptive statistics, however, are subdivided by admission status in the following section. In the data set, 15,204 students were offered admission, while 80,122 students were not, which is a 15.9% acceptance rate across all four applicant terms. The range of admission rates varied from 20.8% for Fall 2020 term to 12.5% for Fall 2022. With respect to the students who were not offered admission, the mean curriculum rating was 3.97 (maximum 5), the variance was 1.153 and a standard deviation of 1.074. The mean GPA was 3.73, the variance was 0.073, and the standard deviation was 0.271. The mean SAT score was 1434, the variance was 12,919.735, and the standard deviation was 113.665. The LOR Rating has a mean of 2.60, a variance of 0.344, and a standard deviation of 0.586.

Concerning the students offered admission, the mean curriculum rating was 4.37 (maximum 5), the variance was 0.603, and a standard deviation of 0.777. The mean GPA was 3.88, the variance was 0.021, and the standard deviation was 0.144. The mean SAT score was 1485, the variance was 6714.886, and the standard deviation was 81.944. Finally, the LOR Rating had a mean of 3.24, a variance of 0.342, and a standard deviation of 0.585.

Table 12 shows the dichotomous crosstabulation between the admission status and the Academic Rating and illustrates that students with more competitive academic profiles were admitted at higher rates.

Table 12 Admission status by academic rating

7.3.2 Correlation between LOR Rating and admission outcome

To explore the relationship between the LOR Rating and the admission outcome, a Pearson correlation coefficient was calculated to be 0.399 (p < 0.001, n = 76,361). A positive, moderate, statistically significant relationship existed between the LOR Rating and the admission outcome. This correlation did not include the measures of academic readiness but illustrates a preliminary understanding of the relationship.

7.3.3 Logistic regression

This study utilized a sequential logistic regression analysis to investigate the entry order of variables into the model, assessing their significance in predicting outcomes [93]. The analysis focused on dichotomous outcomes, specifically admission (1) or non-admission (0). Logistic regression, a non-linear and more complex alternative to multiple linear regression, employs maximum likelihood estimation to identify the optimal linear combination of predictors for outcome prediction [93]. Assumption testing ensured the appropriateness of logistic regression. Criteria included a binary dependent variable, a robust sample size, independent observations, and consideration of outliers. In this study, 158 outliers with a probability of less than 0.001 were included in the model, as they had minimal impact on the overall cases [93].

7.3.4 Comparing two logistic regression models

The first logistic regression included the DV (admit, non-admit), the measures of academic readiness, and the control variables (race, sex, region). This was considered the baseline, known as Model A. The second logistic regression included the same DV (admit, non-admit), identical academic readiness measures, control variables, but adds the LOR Rating. This was Model B. The models were compared.

7.3.5 Model a: results without LOR rating

Model A is considered the baseline model. Here, a logistic regression was performed to ascertain the effects of measures of academic readiness, geographic location, race, and sex, on the likelihood of gaining admission. The logistic regression model was statistically significant, X² [16, n = 76,361) = 10,445.781, (p < 0.001)]. The model explained 20.3% (Nagelkerke R²) of the variance in gaining admission and correctly classified 80.9% of cases. All categories of academic readiness, race, and sex were statistically significant on the likelihood of gaining admission (p < 0.05). Four of the six geographic regions were statistically significant on the likelihood of gaining admission (p < 0.05).

Logistic regression next introduces the exponentiated coefficient, or exp(B). It is referred to as the odds ratio. The odds ratio is a measure of the effect of an independent variable on the odds of the dependent variable [93]. The odds ratio can be used to compare the effects of different independent variables in the model and to identify which variables have a significant impact on the dependent variable [94]. Table 13 shows each variable, statistical significance, and odds ratio (Exp(B)) in the logistic regression equation for Model A.

Table 13 Model A: variables in the logistic regression equation

With an understanding of Model A, Model B introduces the LOR Rating to the logistic regression model.

7.3.6 Model B: results with LOR rating

Here, a logistic regression was performed to ascertain the effects of measures of academic readiness, geographic location, race, and sex, and the LOR Rating on the likelihood of gaining admission. The logistic regression model was statistically significant, X² [18, n = 76,361) = 19,236.562, (p < 0.001)]. The model explained 35.3% (Nagelkerke R²) of the variance in gaining admission and correctly classified 83.6% of cases. All categories of academic readiness, race, and sex were statistically significant on the likelihood of gaining admission (p < 0.05). Two of the six geographic regions were statistically significant on the likelihood of gaining admission (p < 0.05). All three levels of the LOR Rating were statistically significant (p < 0.001). Table 14 shows each variable, statistical significance, and odds ratio in the logistic regression equation for Model B.

Table 14 Model B: variables in the equation

Finally, logistic regression results produced mean predicted probabilities of admission by Academic Rating and LOR Rating. In this logistic regression analysis, mean predicted probabilities referred to the average probability of admission as predicted by the model for the given the set of independent variables included. These mean predicted probabilities are displayed in Fig. 3 for Model B.

Fig. 3

Mean predicted probability of admission by LOR rating and academic rating

7.3.7 Results summary for research question 3

To examine the effects of measures of academic readiness, geographic location, race, sex, and LOR Rating on the likelihood of admission, a sequential logistic regression was conducted. All categories of academic readiness, race, sex, and LOR Rating were statistically significant on the likelihood of gaining admission (p < 0.05). Two of the six geographic regions were statistically significant on the likelihood of gaining admission (p < 0.05).

Overall, when comparing the two logistic regression models, the model inclusive of the LOR Rating, Model B, explained a greater amount of the variance in the admission outcome and correctly classified 2.7% more cases. The logistic regression Model B was statistically significant [Χ ² (18, N = 76,361) = 19,236.562, p < 0.001] and accounted for 35.3% (Nagelkerke R²) of the variance in admission, accurately classifying 83.6% of cases. Accordingly, there was a significant relationship between the LOR Rating and the admission outcome when controlling for academic readiness and demographics measures.

Students with LOR Ratings of four or five (high support) had nearly five times higher odds of gaining admission than students with LOR Ratings of three (medium support) [OR = 4.740, 95% CI (4.446, 5.030)]. Students with LOR Ratings of three (medium support) had seven times higher odds of gaining admission than students with LOR Ratings of one or two (low support) [OR = 7.021, 95% CI (6.555, 7.521)]. Finally, students with LOR Ratings of four or five (high support) had 33 times higher odds of gaining admission than students with LOR Ratings of one or two (low support) [OR = 33.278, 95% CI (30.566, 36.231)].

7.4 Overview of findings

This section presented the study results for the three research questions. Question one found that private school applicants had the highest mean LOR Rating (M = 2.87, sd = 630) and a positive and statistically significant mean LOR Rating compared to applicants from other school types (p < 0.05) after controlling for academic readiness. Specifically, out of 60 interactions, 33 were positive and statistically significant. Applicants from private schools received 17 positive and statistically significant mean LOR Rating differences between school types, the highest of any school type (p < 0.01, Two-Way ANOVA).

Question two uncovered that multiple linear regression analyses revealed High School Size, Landscape High School Index, Landscape Neighborhood Index, Curriculum, GPA, and SAT as significant predictors (p < 0.001) of the LOR Rating. The regression model explained 13.9% (R²) of the variance in the LOR Rating. High school class size showed a weak, negative linear relationship with the LOR Rating, while both Landscape Indices displayed a weak, positive linear relationship. Moreover, Landscape Indices maintained a significant relationship with the LOR Rating after controlling for academic readiness and demographics.

Question three examined the role of the LOR Rating, along with measures of academic readiness, geographic location, race, and sex, on admission likelihood using logistic regression. The model was significant (p < 0.001) and explained 35.3% of the variance, with 83.6% correct classifications. LOR Rating significantly affected admission likelihood, with students rated three (medium support) being seven times more likely to gain admission than those rated one or two (low support), and students rated four or five (high support) being nearly five times more likely than those rated three (medium support). The logistic regression model, inclusive of the LOR Rating, explained more variance (2.7%) and had a higher correct classification for over 76,000 cases. Thus, the LOR Rating significantly correlated with admission likelihood when controlling for academic readiness.

8 Discussion

The transition from high school to higher education involves a multifaceted journey shaped by various factors, such as economic, cultural, and social resources [95]. For students aspiring to enter the most selective U.S. institutions, academic excellence and robust support from the school community are paramount. In the application phase, high-achieving students submit a range of documents, including academic records, essays, extracurricular summaries, standardized test results, and LOR [64]. This research aims to contribute to the literature by examining the assessment of LOR in the highly selective college admission process.

8.1 High school type discussion (RQ1)

The first research question examined the relationship between admission officers’ LOR Ratings and students’ school types, controlling for academic readiness. Results showed that students from private schools received higher recommendation ratings than those from public and parochial schools, when academic achievement was held constant. Subsequently, the study delves into the literature and investigates potential explanations for this observed pattern.

8.1.1 Knowledge of context through school profiles

Admission officers’ awareness of school context is crucial for effective application assessment [24]. Bastedo and Flaster [8] demonstrated a 13% to 14% increase in the likelihood of admission recommendation when contextual information was available. The type of school a student attends, a key element of student context, significantly influences admission officers’ evaluations [66]. School profiles, curated by guidance counselors, serve as the primary source of contextual information for admission officers [65]. However, research by Nicola [66] reveals variations in profile quality across school types, favoring private and elite institutions. This discrepancy may contribute to higher LOR Ratings for private school students, as admission officers likely possess more comprehensive context information.

Empirical studies by Bastedo and Bowman [8] and Bastedo [5, 6] emphasize the need for consistent, high-quality information on school context. The absence of such information can be detrimental, as observed in the sample of over 95,000 applicants, where 9% lacked school type information, hindering admission officers’ immediate understanding of contextual factors [66]. Ensuring uniform access to robust school context information is imperative for admission officers to accurately position LOR within the broader context of student opportunities and achievements.

8.1.2 Private high schools

Private high schools, the younger siblings of private colleges, influence college destinations, particularly for students from low-resource schools [49]. Initial analysis of Landscape Indices by High School Type suggests that private high schools, on average, pose lower contextual challenges compared to other school types [49]. The financial landscape of private high schools, characterized by tuition fees, introduces exclusivity, hindering access for students with modest resources [37]. Tuition disparities, coupled with varying scholarship and financial aid provisions, contribute to the distinctive context of private schools, where annual tuition ranges from $15,645 at typical private high schools to $67,270 at private boarding schools [37].

Examining the role of high schools in facilitating class advantages in college enrollment, the literature underscores the impact of social and economic class, prompting the need to explore how high school types contribute to these advantages [3, 49]. Despite controlling for academic achievements in this study, which ensured an equitable comparison of students across school types, private school students consistently received higher LOR Ratings than their counterparts from parochial and public schools. Academic preparation, being constant, suggests that other factors inherent to private schools contribute to elevated LOR Ratings, a phenomenon explored in the subsequent section.

8.1.3 “Feeder” private schools and the psychology of familiarity

One consideration as to why private school students receive LOR Ratings higher than any other type of school is that their high school may be most familiar to admission officers. As previously mentioned, school context does matter, and it is a reasonable belief that long-standing relationships between institutions provide a greater understanding of context by admission officers. According to Karabel [46], private feeder high schools—typically elite, private preparatory schools that send a disproportionate number of their students to highly selective colleges—have long influenced the college admissions landscape. While it can be argued that a far more equitable admission landscape exists today than in previous generations, the historical relationships between the elite, and private preparatory schools have not entirely disappeared.

Private feeder high schools can be located in close proximity to the institution they feed, and they may have long-standing established relationships with the admissions staff. A symbiotic relationship exists “between the feeder schools and the elite colleges” since the long history of producing extremely high quality students at elite, private preparatory schools is notable [46]. These institutions exist in what Posselt [74] identified as trust networks where members know each other. Further, the reputation of a highly selective college is likely strong in the locality so it would not be uncommon for local students to aspire to enroll near where they live. However, it may depend on student demographics and background and other factors [61, 73].

For selective college admission practices, feeder high schools can significantly impact the admissions process [95]. Wolniak and Engberg [95] noted that students who attend feeder high schools are more likely to enroll in selective colleges than students from non-feeder high schools, even after controlling for a range of demographic and academic factors and suggested that the presence of feeder high schools creates a social network that encourages college-going behavior.

Two social psychology phenomena may exist around private feeder schools which may directly relate to admission officer assessments of LOR. The first is Polyani’s [72] theory of tacit knowledge, and the second is Zajonc’s [96] mere exposure phenomenon.

Michael Polyani was a philosopher most recognized for his work The Tacit Dimension, first published in 1966 and republished in 2009. In this work, Polyani described how humans acquire knowledge through explicit and tacit mechanisms [72]. While explicit knowledge is transmitted through written or symbolic forms, tacit knowledge is a type of non-conscious learning acquired through lived experience, observation over long periods of time, and immersion [72]. An admission officer reading many applications from specific private feeder schools may develop tacit knowledge and familiarity.

Myers and Twenge [63] define familiarity as the subjective recognition of encountering a stimulus previously. Regarding letters of recommendation and private feeder schools, two types of familiarity are relevant. Firstly, a high school sending numerous applications to a specific college establishes relationship familiarity, fostering credibility and trust between admission staff, college counselors, and teachers [63]. This relationship facilitates information exchange, enhancing college counselors’ understanding of what admission officers seek in LORs. Secondly, admission officers reading a higher volume of LORs from private feeder schools develop associative familiarity, creating a standardized cohort that aids understanding LORs in a specific school context [63]. While these familiarities may not independently lead to higher LOR ratings, they contribute to a deeper understanding of the applicants' context, potentially influencing ratings.

Specific to familiarity, the mere exposure phenomenon was first described by the social psychologist Robert Zajonc in the 1960s. The mere exposure phenomenon is a psychological concept that refers to the tendency for people to, over time, develop a preference or liking for stimuli that are more familiar to them, even if they are not consciously aware of this exposure [96]. In other words, and specific to this research, simply being exposed to a high volume of LOR, or other engagements such as programming, from one school repeatedly can increase and admission officers’ positive feelings towards that stimulus, regardless of whether they consciously remember encountering it before [96]. This phenomenon may be occurring with admission officers who read high volume/private feeder school groups and many recommendations from a particular school.

This section highlights the human psychology of the individualized holistic review process. Admission officers are humans participating in a deeply human process. But perhaps, there is a broader systemic, social, and cultural element that possibly explains why private school students receive higher LOR Ratings from admission officers.

8.1.4 Cultural alignment private high schools and private colleges

Distinctive cultures exist within public and private schools [36], rooted in the sociological framework of Symbolic Interactionism [60]. Private high schools function as complex social entities where students construct meaning around shared values through community interactions. Given the similarities between private high schools and colleges, admission officers may recognize and value these shared values, potentially leading to higher ratings [60]. However, excessive prioritization of students fitting predetermined values can exacerbate socioeconomic disparities perpetuated by limited access to private education [36]. In the evaluation process, admission officers must uphold the principle of Individualized Holistic Review (IHR) to ensure impartial assessment within the context of each student’s high school [36]. While cultural and value alignment may explain higher LOR ratings for private school students, biases from admission officers and letter writers must be addressed to ensure fairness in the college admissions process.

8.1.5 Bias By readers, writers, and the rating

Bias in admission officers reviewing applications from private schools, whether conscious or unconscious, is well-documented [1, 50, 80]. Committee-Based Evaluation (CBE) processes, while aiming to control bias in the final stage, may not address biases in rating assignment [80]. Kuncel et al. [50] noted biases favoring candidates from high-status institutions in LOR, suggesting a potential similar bias for private high schools. Higher LOR Ratings for private school students may stem from conscious or unconscious bias among admission officers. Additionally, inherent biases of letter writers, especially in private schools, may contribute to elevated LOR Ratings [70]. Despite challenges in investigation, this hypothesis warrants serious consideration. The design of rating schemes introduces another layer of bias, with subjective judgments possible in the Likert scale-based system evaluating personal qualities and language use. Finally, the resource-rich and well-informed apparatus around the college admission process in private schools may also explain why private school students receive higher LOR Ratings compared to other school types.

8.1.6 Guidance counselors vs. the apparatus of private school college counseling

Research has shown that college counseling can significantly impact the trajectory of students [44, 56, 71]. However, the role and responsibilities of school counselors can vary deeply across schools, affecting the quality of college counseling services [11, 85, 94]. Moreover, there are notable differences between school counselor roles at private schools and other school types, which can further impact the quality and availability of college counseling services [18]. In fact, a substantial proportion of private schools have at least one dedicated counselor whose sole responsibility is to provide college counseling to students, with non-parochial private schools reporting the highest percentage of dedicated counselors at 75%, followed by parochial schools at 58%, and public schools reporting the lowest percentage at only 33% [18]. Relative to other types of secondary schools, private secondary schools have lower counselor-to-student ratios and their college counselors spend more time specific to university admission advising [20, 71]. Given these differences across school types, it is reasonable to believe private school counselors dedicate more time towards the crafting of letters of recommendation on behalf of their students. They also may have more knowledge of highly selective admission practices. And in the end, the apparatus in private schools translates into higher LOR Ratings for private school students.

8.2 High school size and landscape index discussion (RQ2)

This research question aimed to understand if, and to what extent, high school size had a relationship with the LOR Rating a student received when controlling for measures of academic readiness. The results revealed a negative, statistically significant relationship between high school size and LOR Rating. As the school size increased, the LOR Rating decreased. The most fundamental explanation for this result is that high school size is a proxy for student-to-counselor and student-to-teacher ratios. The larger the school, the larger the caseload of school counselors. Further, it may also be a proxy for the class sizes of teachers. High school size is suggested to impact college admission outcomes due to student-to-counselor ratios [32]. Further, the NACAC State of College Admission Report [18] reveals a similar finding in their data, which is shown below in Table 15:

Table 15 Students per college counselor by school enrollment size

The 2019 NACAC State of College Admission Report [18] shows that larger school size correlates with reduced time for college admission counseling due to high student-to-counselor ratios [18]. Consequently, under-funded public high schools face challenges in providing adequate counseling resources. Recent studies [11, 18] link larger school size to lower LOR Ratings in selective college admissions, even after adjusting for student academic measures. The COVID-19 pandemic exacerbated challenges for guidance counselors, who had to swiftly transition to remote counseling services [84, 85] while dealing with increased student interest in highly selective schools [39]. Despite these pressures, counselors were still expected to write recommendation letters, likely impacting the quality of LOR and contributing to lower LOR Ratings for students in larger schools [85, 90].

In highly selective college admissions, a notable revelation challenges prior assumptions regarding school type and size, emphasizing admission officers’ meticulous consideration of context in the individualized holistic review. The Landscape Composite Index, reflecting challenge levels, exhibited a positive correlation with Letters of Recommendation Ratings, even after adjusting for academic readiness measures. This contradicts the notion that higher LOR Ratings are exclusive to private or smaller schools, as students facing greater systemic challenges earned higher LOR Ratings. This aligns with Individualized Holistic Review (IHR) principles [24], illustrating successful contextual evaluation irrespective of academic achievement. As challenge levels increased, students surpassing their context earned higher LOR Ratings, indicating effective information integration in the IHR process. Notably, at the highest Landscape Indices decile (69.5 or greater), a substantial increase in mean LOR Ratings was observed across Academic Ratings, emphasizing admission officers’ commitment to social justice by contextualizing experiences of the most disadvantaged students [47]. This discovery introduces complexity to the understanding of LOR and IHR, underscoring the significance of LOR in highly selective admissions and warranting further research for comprehensive comprehension.

8.3 Admission outcome discussion (RQ3)

The final research question explored the role of the LOR Rating, along with measures of academic readiness, geographic location, race, and sex, on admission likelihood using logistic regression. The results illustrated the LOR Rating had a significant relationship with admission likelihood when controlling for academic readiness. Applicants receiving high LOR Ratings (4, 5) were 4.7 times more likely to be offered admission than students who received moderate ratings (3) and 33 times more likely to be admitted than students who received low ratings (1, 2). Students receiving moder ratings (3) were seven times more likely to be admitted than students who received low ratings (1, 2).

8.3.1 Role of LOR in logistic regression models

The results showed that the LOR Rating mattered to admission officers and broader admission committees in decision making. One of the insights gained through stepwise logistic regression was that the study could observe changes in the statistical model as independent variables were added to the logistic regression.

Most of the independent variables showed a slight change in odds ratio (Exp(B)) when the LOR Rating variable was added to the logistic regression. However, given a review of the two models, two key observations merit mentioning. The first is that with the LOR Rating added into the equation, there was a reduction in the number of geographical locations that were statistically significant. The difference changes the geographic regions with statistical significance from four to two. The West and Unknown remained statistically significant, while the Mid-Atlantic and Midwest lost statistical significance.

The second observation relates to the influence of the LOR Rating on the Academic Rating when it was added into the equation. Table 16 summarizes the change in odds ratios between the two logistic regression models.

Table 16 Change in Odds Ratio between Logistic Regression Models

The change in odds ratios reveals that the LOR Rating plays a pivotal role in mitigating the impact of the Academic Rating on admission outcomes, supporting the holistic nature of the process beyond academic metrics. This indicates that admission officers consider valuable insights from LORs, extending beyond measures of academic readiness. Notably, the change in odds ratios intensifies with increased academic readiness, particularly at the highest academic spectrum. This suggests that, in competitive academic profiles, LORs become crucial in distinguishing between similar students, aligning with the highly selective nature of the institution under study.

The study draws two conclusions regarding the relationship between LOR Ratings and admission outcomes. Firstly, higher LOR Ratings for private and smaller school students correlate with favorable admission outcomes, emphasizing the LOR Rating’s role in incorporating contextual factors. Secondly, students from higher Landscape Indices also benefit from higher LOR Ratings, recognizing the exceptional support they receive in challenging circumstances. Overall, the findings underscore the importance of the LOR Rating in the admission process, serving as a proxy for contextual factors. Admission officers acknowledge the value of teacher and counselor recommendations, and higher LOR Ratings increase the likelihood of favorable admission outcomes, particularly for students from private schools, smaller schools, and higher Landscape Indices. This study emphasizes the complexity and significance of the LOR Rating in the broader admission landscape.

8.4 Social reproduction theory and the LOR rating

Social Reproduction Theory (SRT), rooted in Sociology, scrutinizes the access and benefits individuals and groups gain from various forms of social capital. This framework, as elucidated by Giroux [33] and Bourdieu [12], delves into how educational institutions perpetuate dominant cultures, reinforcing social hierarchies. Ambiguous evaluation criteria, according to Bourdieu, favor privileged classes, and high-resource communities possess social capital easily exchangeable with selective institutions. Conversely, lower-resource communities may possess valuable social capital, unrecognized by elite institutions. SRT urges a critical examination of structures to foster equity. This research aligns with SRT, investigating how social capital operates in diverse contexts, contributing to the generational reproduction of social inequality.

The study delves into whether Letters of Recommendation function as a form of capital exchanged between members of distinct educational communities. It also explores how contextual variables beyond students’ control (school type, size, Landscape Index) influence the LOR Rating. Findings from RQ1 and RQ2 indicate a significant effect of school size and type on the LOR Rating, while RQ3 establishes the significance of the LOR Rating in admission outcomes. Figure 4 visually demonstrates how LOR integrates with Social Reproduction Theory concerning access to selective colleges and universities.

Fig. 4

Role of LOR rating in Social Reproduction Theory

The above figure illustrates that any relationship between school type, school size, and the LOR Rating is problematic at any rate. The question is not if this is occurring but rather to what extent. Any relationship can be understood as contributing to social reproduction, which is problematic. Nevertheless, the LOR Rating and the consideration of student context can disrupt social reproduction and ensure that students at any type of school or size can be considered equitably for selective university admission. The results from RQ2 suggest that disruption of social reproduction occurs and that admission officers consider context as they assign the LOR Rating. Recall the statistically significant relationship between higher levels of challenge and higher LOR Ratings assigned by admission officers.

8.5 Implications for policy and practice

The findings of this study have implications for both policy and practice. With respect to policy implications, there are two central recommendations: greater standardization of LOR and a greater number of school counselors in public schools.

8.5.1 In policy

Existing research strongly advocates for two key policy recommendations. Firstly, there is a pressing need for greater standardization of Letters of Recommendation in college admissions, as highlighted by Dalal et al. [29], Kuncel [50], and Houser and Lemmons [42]. Current variations in accepting LORs without standardized evaluation forms contribute to inconsistencies. Institutions should reconsider their policies on this matter, engaging in intentional discourse to promote greater equity in the application process. Although standardized evaluation forms may add complexity, careful consideration is essential. Additionally, platforms like the Common Application should educate teachers and counselors on the importance of standardization and equity in the LOR process. Secondly, addressing the shortage of comprehensive school counselors in public high schools is crucial. The research underscores that larger schools with higher student-to-counselor ratios are associated with lower LOR Ratings, emphasizing systemic inequities. Given the evolving roles of school counselors beyond administrative tasks, system-level changes are imperative to provide adequate support, professional development, and resources for their effective contributions to student success [85].

8.5.2 In practice

This study provides actionable recommendations for enhancing the effectiveness of the highly selective undergraduate admissions process. First, it is imperative to clarify the role of Letters of Recommendation internally and externally, revisiting rating schemes and evaluation criteria to align with institutional mission and goals. Clear guidance ensures that LOR offer valuable insights into applicants’ abilities and potential.

Secondly, enrollment leaders should prioritize and enhance the training of admission officers to better evaluate LOR and understand student context. Ongoing professional development can improve their skills in evaluating applications and provide contextual information about applicants’ backgrounds. Retaining admissions officers in specific territories or creating a repository of information can ensure the effective use of contextual knowledge. Third, enrollment leaders should advocate for increased investment in public school counselors to support the integrity of Individualized Holistic Review (IHR). Recognizing the unique challenges faced by public school students and counselors, advocating for additional resources levels the playing field, ensuring equitable access to higher education resources [85].

Next, conducting internal research on admission processes is crucial for highly selective institutions to assess the functioning of LOR. Thorough investigations and analyses can identify unintended biases or disparities in LOR evaluation. Quantitative analysis, akin to this study, along with qualitative methods like interviews or focus groups, can reveal insights for corrective measures. Lastly, enrollment leaders must ensure appropriate staffing levels, especially at highly selective colleges where admission officers handle over 1000 applications each [18]. Revisiting and optimizing admission operations, considering the impact of COVID-19, is essential to maintain the integrity of the admissions process [67, 80].

8.6 Recommendations for future research

Based on the findings of this study, several areas of inquiry merit additional exploration, all contributing to knowledge capital around LOR and their role in the highly selective admission process. This study suggests several avenues for further investigation into the role of letters of recommendation in highly selective college admissions. Research should delve into potential biases in LOR, whether originating from the writer, reader, or the rating scheme, exploring influences such as gender, race, or socioeconomic status [1]. Additional understanding is needed regarding the predictive validity of LOR in assessing academic success, encompassing GPA, academic persistence, and student satisfaction, and examining correlations with LOR content [50].

Qualitative methods should be employed to explore the perspectives of both applicants and recommenders on the LOR process, considering motivations, experiences, and perceptions and accounting for variations across demographic or identity groups [1]. Research should extend beyond U.S. applicants to explore the role and interpretation of LOR in diverse cultural settings, both within the U.S. and internationally, examining cultural norms, expectations, and the influence of cultural factors on LOR content and impact.

Building on prior research, further investigation is warranted into the use of technology, such as natural language processing and machine learning, for analyzing LOR content, developing automated tools to identify biases, and assessing the effectiveness of LOR in predicting success [2]. An unexplored area is the career trajectory of admission officers transitioning to private high school advising offices, with potential research exploring how their expertise influences the LOR writing process in private schools and the resulting advantages for students. Finally, further study is needed on the training and professional development needs of public school counselors and recommenders in writing effective LOR, exploring resources and support systems institutions can provide to ensure meaningful and reliable LOR in the admission process.

8.7 Limitations of study

This study has several fundamental limitations which merit additional discussion. These limitations are related to assumptions within statistical tests, the Academic Rating and SAT/ACT-optional policies, and generalizability.

In the exploration of RQ1, the Two-Way ANOVA revealed a noteworthy statistical concern—Levene’s test for equality of variances yielded a significance level (p < 0.05). This outcome is attributed to the minimal representation of Home School students, comprising a mere 0.03% of cases. That said, likely, this limitation does not critically detract from the findings of RQ1 but somewhat limits the findings specific to students schooled at home.

The LOR Rating, the central dependent variables of the study, presents challenges due to its pronounced central tendency. This situation introduces complexities in statistical testing, where the reliance on central tendency measures (mean, median, mode) can be influenced by outliers. Moreover, the sparing use of very high and very low ratings raises modest concerns about the assumed normal distribution. These aspects collectively shape limitations in the study’s ability to draw robust conclusions from the LOR Rating data.

The multiple linear regression analysis was used in RQ2 of this research study. In this analysis, covariate independent variables were included in the regression model alongside the predictor variables of interest to account for their potential influence on the dependent variable, the LOR Rating. In the specific context of this study, the two measures of academic readiness (GPA and Standardized testing) displayed some multicollinearity in a review of collinearity diagnostics in RQ2. Despite this, the investigation chose to include both measures in the analysis, as they are related but distinct measures of academic readiness. The absence of one of these critical measures was believed to pose a greater limitation to the study, especially given the impact of COVID-19. However, it should be noted that multicollinearity between these two measures of academic readiness poses a limitation, which should be considered while interpreting the study’s findings. Similarly, both Landscape Indices were used as independent variables in RQ2. These two independent variables also displayed multicollinearity. They are related as they examine the challenge levels of both neighborhoods and high schools. Again, these are related and display multicollinearity, but the study elected to leave both independent variables in the study to understand their unique contributions to the outcome.

A substantial shift in Academic Ratings due to the institution’s response to the COVID-19 pandemic deserves attention. The move to SAT/ACT-optional policies led to a marked increase in the “Other” category, shifting from 6 to 47% between 2020 and 2021. While this adjustment aimed at dataset balance, the consequential impact on the Academic Rating, and its representation of student achievement, stands as a recognized limitation stemming from external circumstances. The impact of COVID-19 on the Academic Rating should be noted as a limitation of this study while also recognizing over 70,000 cases had Academic Ratings.

A critical consideration lies in the limited generalizability of the study's findings. The reliance on internal data from a singular institution implies a confined applicability. The unique institutional policies, practices, and contextual factors shaping the data may not align with broader educational landscapes. Consequently, caution is warranted in extending the study’s outcomes beyond its specific institutional boundaries.

9 Conclusion

The landscape of United States higher education is intricately shaped by economic and social factors, leading to a socially stratified environment that permeates the education system. These disparities significantly impact the admission process at highly selective schools, prompting recent public discourse on how LOR may either perpetuate or address the economic challenges faced by applicants. The subjective nature of these letters and the diverse contexts in which they are obtained introduce disparities in the assessment process, potentially affecting admission outcomes for students from varying socioeconomic backgrounds. This nuanced perspective highlights the need for further exploration of how LOR are evaluated in the admission process of highly selective institutions, with a focus on fostering equity.

This quantitative research delves into the relationship between admission officer assessments of LOR and external contextual variables beyond student control, including school type, high school size, and the College Board Landscape Indices. While students from private high schools and smaller schools tend to receive higher mean recommendation ratings, a positive association emerges between mean LOR ratings and higher levels of contextual challenge. This study provides critical evidence on how admission officers can successfully account for student context in the review of letters of recommendation at highly selective institutions. These findings underscore the intricate and contradictory nature of LOR in the highly selective admission process, emphasizing the significance of these contextual variables and the deeply important work of admission officers.

These insights emphasize the ongoing necessity for admission officer training on individualized holistic review (IHR) at highly selective institutions, stressing the critical role of deeply considering student context in the admission process. By accounting for the challenges and opportunities that students from low-resource backgrounds encounter in their educational environments, admission officers can foster more equitable and effective assessment practices. This approach enhances the identification and support of applicants from diverse socioeconomic backgrounds, ultimately promoting upward income mobility for high-achieving students in both low-income and high-challenge contexts. LOR simultaneously challenge and perpetuate the social stratification of American secondary schools. The findings contribute to the intricate but essential conversation about enhancing admission processes at highly selective institutions, despite escalating wealth inequality across the nation. Addressing the subjective nature of LOR and recognizing the conflicting influence of specific contextual variables enables admission officers to strive for a more equitable process. Further research and ongoing efforts towards equitable admission practices are crucial to ensuring that access to highly selective higher education institutions is based on individual potential, excellence, and achievements rather than unknowingly influenced by socioeconomic status.