Preference estimation in centralized college admissions from reported lists

Abstract

This paper proposes a practical and data-driven preference estimation method from reported lists in a deferred acceptance mechanism when there are incentives to report these lists strategically. Data on centralized college admissions from Turkey show many pieces of evidence that students construct their lists strategically according to their admission chances and previous years’ admission outcomes. We develop a preference estimation method to evaluate reported lists within the set of colleges that are considered accessible to each student. This method allows us to create personal choice sets and to estimate student preferences by making valid utility comparisons that are supported by data and theory. We show the robustness of our estimation method compared to the existing estimation methods. A counterfactual admission analysis based on our preference estimates suggests that students from low-SES households are better off under a student sorting rule only based on high school GPAs.

Appendix

1.1 Survey

The survey study that is used in the analysis is conducted by Alkan et al. (2008). In this study, the sample of college applicants are selected from the senior high school students who took the college entrance examination in 2005. For this sample, high schools are selected randomly based on their locations and types. The survey study is given to all senior high school students in the selected high schools. The random selection of high schools takes the population of students in Turkey and performance of the high school types into account. To achieve a representative sample of the population, 33 cities are selected from 12 different geographical regions of Turkey and 448 high schools are selected based on the student populations in the selected cities.

Figure 7 presents the frequencies of the selected number of high schools by city on the map of Turkey. It is observed that students from Istanbul and Ankara are more represented in the sample. Participation rate of high schools is considerably high. Only 2 high schools rejected to participate in this study. Students within the selected high schools were interviewed within the school time in classrooms and absent students were interviewed additionally. In total, 13727 students participated in the survey study, and 898 participants were excluded from the analysis because the information of these students could not pass the internal validation analysis (Figs. 8 and 9).

1.2 Technical details on assumptions in Section 3.3

We restrict our attention to previous year’s admission outcomes as they are the most important information for generating a distribution of cutoffs and make the following assumption to incorporate this information into the subjective admission beliefs of students.

Assumption 2

The mean and support for subjective admission beliefs are as follows:

• 1. $\int {\varOmega }_{ij}({\tilde{p}}_{i}, \sigma ; T) dF(T | t^{-1}_{j^{\prime }}, \xi _i ) \le \int {\varOmega }_{ij^{\prime }} ({\tilde{p}}_{i}, \sigma ^{\prime }; T) dF(T | t^{-1}_j, \xi _i )$ if $t^{-1}_j > t^{-1}_{j^{\prime }}$ for all $i \in N$.

• 2. the support of the distribution of cutoff scores follows the monotonicity w.r.t. previous year’s cutoffs, i.e., ${\overline{t}}_{j}(t_j^{-1})> {\overline{t}}_{j^{\prime }}(t_{j^{\prime }}^{-1}) \; and \; {\underline{t}}_{j}(t_j^{-1}) > {\underline{t}}_{j^{\prime }}(t_{j^{\prime }}^{-1})$ if $t_{j}^{-1} > t_{j^{\prime }}^{-1}$, where ${\overline{t}}_{j}$ is the upper boundary of the cutoff domain for college j and ${\underline{t}}_{j}$ is the lower boundary of the cutoff domain for college j.

where $t_j^{-1}$ denotes the cutoff score of college j in the previous year. Assumption 2.1 says that student i’s admission belief for a college j is lower or equal to admission belief for college $j^{\prime }$ if college j has a higher cutoff score than college $j^{\prime }$ in the previous admission period. Assumption 2.2. restricts the support of subjective admission beliefs. According to Assumption 2.2., subjective admission beliefs satisfy a monotonicity condition to previous year’s cutoff scores such that support of admission beliefs does not switch their orders between adjacent years. These two conditions follow from the fact that cutoff scores across years are highly correlated in many centralized admission systems. It is realistic to expect the same cutoff score orders of colleges across years and the students to generate their beliefs accordingly.

In the centralized admission mechanism, we also observe that students generally submit an incomplete ROL. We make Assumptions 3 and 4 to eliminate peculiarities as a result of incomplete ROLs in the data.

Assumption 3

For all $i\in N$, $0< \kappa < (u_{ij} Pr(t_j \le {\tilde{p}}_i | T^{-1}, \xi _i)$ if $Pr(t_j \le {\tilde{p}}_i | T^{-1}, \xi _i)> 0$.

Assumption 3 says that application cost is not negligible. Students do not add any colleges to their list if they have zero admission beliefs for these colleges. This is a legitimate assumption because even most of the low-scoring students did not submit a complete ROL in 2005.

Assumption 4

$F(T | T^{-1}, \xi _i)$ has continuous and strictly positive density $f(T| T^{-1}, \xi _i)$;

• 1. the support of distribution of cutoff scores is student-specific, i.e., $[{\underline{T}}_i, {\overline{T}}_i]$,

• 2. the boundaries for cutoff support of each college are different, e.g., ${\overline{t}}_{ij} \ne {\overline{t}}_{ij^{\prime }}, {\underline{t}}_{ij} \ne {\underline{t}}_{ij^{\prime }}$ $\forall j, j^{\prime } \in C, \; j\ne j^{\prime }$.

Assumption 4 imposes regularity conditions for the support of the distribution of cutoff scores. According to these conditions, each student has a personal distribution of cutoff score domain that makes her admission beliefs different from other students. A student’s admission beliefs are zero for the colleges whose cutoff scores do not belong to the student-specific cutoff score support. It also assumes that the distributions of cutoff scores do not overlap perfectly. This is a realistic assumption under the strict priority orders because there are no ties among cutoff scores under no ties in priorities.

1.3 Proof of Lemma 2

We want to show that for a given student i with score $p_i$,

$$\begin{aligned} E_i [{\varOmega }_{ij}(p_i, \sigma ) | t_j^{-1} ] = 0 \; \text {if} \; E_i [{\varOmega }_{ik}(p_i, \sigma ) | t_k^{-1} ] = 0\; \text {and} \; t_j^{-1}> t_k^{-1} \forall j\in C \end{aligned}$$

From Assumption 2, we know that subjective admission probabilities decrease with the previous year’s cutoff scores. Therefore, if $E_i [{\varOmega }_{ik}(p_i, \sigma ) | t_k^{-1} ] = 0$ and $t_j^{-1}> t_k^{-1}$ we know that $\int {\varOmega }_{ij}({\tilde{p}}_{i}, \sigma ; T) dF(T (t_j^{-1}, \xi _i) ) \le \int {\varOmega }_{ik} ({\tilde{p}}_{i}, \sigma ; T) dF(T (t_{k}^{-1}, \xi _i)$. Moreover, $E_i [{\varOmega }_{ij}(p_i, \sigma ) | t_j^{-1} ] \ge 0$ because it is probability measure and cannot be less than 0. Then,

$$\begin{aligned}&E_i [{\varOmega }_{ik}(p_i, \sigma ) | t_k^{-1} ] \ge E_i [{\varOmega }_{ij}(p_i, \sigma ) | t_j^{-1} ] \ge 0 \\&E_i [{\varOmega }_{ij}(p_i, \sigma ) | t_j^{-1} ]= 0 \end{aligned}$$

$\square$

1.4 Proof of Theorem 1

According to Theorem 1, $L_i^{*}$ is the ranking of the most preferred colleges within student i’s consideration set. Then, we need to show that

$$\begin{aligned} u_i Pr(L^*_i) -\kappa |L_i^*| \ge u_i Pr(L_i) -\kappa |L_i| \;\; \forall L_i \in {\mathcal {R}}_i \end{aligned}$$

where ${\mathcal {R}}$ is all possible rank order lists in the consideration set of student i and $Pr(L_i)$ indicates the probability of assignments of colleges in the list $L_i$. According to the condition in Theorem 1, the number of reported colleges is strictly fewer than the limit number of colleges in the list, i.e., $|L| < K$. There are two different ways of deviating from listing most preferred colleges in the $L_i$ within student i’s consideration set:

• 1. Adding a college $j\not \in L_i^*$ into list $L_i^*$

• 2. Changing the order of colleges in list $L_i^*$

To show that $L_i$ consists of the most preferred colleges within the consideration set of student i, we need to show that all these deviations have to be worse off for student i.

Case 1: We know that from the condition in Theorem 1 students that have only $|L_i| < K$ number of colleges from which they can attain positive utility in their consideration sets. There are two cases in which student i can add a college j in her lists: College j should be either one from which she derives non-positive utility because she has already added all colleges she prefers to go or it that does not belong to her consideration set. Therefore,

$$\begin{aligned} V_{ij}= Pr(c_j \in L_i) u_{ij} \ge 0 \end{aligned}$$

where $u_{ij} \le 0$ if she does not prefer college j or she believes that $Pr(c_j \in L_i)= 0$. Either case’s expected utility ($V_{ij}$) of college j in $L_i$ is equal to zero. According to Assumption 3 there is a nonnegative cost of adding college in a list, and the total gain of adding a college j is negative.

Case 2: Assume that student i prefers college j over college $j^{\prime }$. Colleges $j, j^{\prime }$ are part of the student i’ consideration set, where $u_{ij}, u_{ij^{\prime }} > 0$. For simplicity, we only cover changing order of two adjacent colleges in the list. If student i lists $c_{j^{\prime }}$ before $c_j$ in a new list $L_i^{\prime }$—keeping everything else the same—the expected utility of new list $L_i^{\prime }$ should be higher than the expected utility of $L_i^*$ so that this deviation makes student i better off.

$$\begin{aligned} Pr(c_{j^{\prime }} | L_i^{\prime }) u_{ij^{\prime }} + Pr(c_{j} | L_i^{\prime }) u_{ij} > Pr(c_{j^{\prime }} | L_i^*) u_{ij^{\prime }} + Pr(c_{j} | L_i^*) u_{ij} \end{aligned}$$

(6)

where $Pr(c_{j^{\prime }} | L_i^{\prime })$ indicates the probability of assignment of college $j^{\prime }$ in the $L_i^{\prime }$. According to DA algorithm, an agent is assigned to a college in her reported list if she is rejected by all colleges that are ranked before it in the list. We know that $Pr(c_{j} | L_i^*) = Pr(t_j< p_i | L_i) > Pr(c_{j} | L_i^{\prime }) = (1- Pr(t_{j^{\prime }}< p_i | L_i) ) \times Pr(t_j < p_i | L_i)$. Also, $Pr(c_{j^{\prime }} |L_i^{\prime }) = Pr(t_{j^{\prime }}< p_i | L_i) > Pr(c_{j^{\prime }} | L_i^{*}) = (1- Pr(t_{j}< p_i | L_i) ) \times Pr(t_{j^{\prime }} < p_i | L_i)$.

Then, we can rewrite Equation 6 as

$$\begin{aligned}&(Pr(t_{j^{\prime }}< p_i | L_i) u_{ij^{\prime }} + ((1- Pr(t_{j^{\prime }}< p_i | L_i) ) \times Pr(t_j< p_i | L_i) )u_{ij} )> \\&(Pr(t_j< p_i | L_i) u_{ij} + ((1- Pr(t_{j}< p_i | L_i) ) \times Pr(t_j^{\prime }< p_i | L_i) )u_{ij^{\prime }}) \\&\quad - Pr(t_{j^{\prime }}< p_i | L_i) Pr(t_j< p_i | L_i) u_{ij^{\prime }}> - Pr(t_{j^{\prime }}< p_i | L_i) Pr(t_j < p_i | L_i) u_{ij} \\&u_{ij^{\prime }} > u_{ij} \end{aligned}$$

which contradicts with $u_{ij^{\prime }} < u_{ij}$. Hence, changing the order of colleges is not better off for student i. $\square$

1.5 Sample selection for the estimation

In our estimations, we select a sample to satisfy the identification requirements of 4 different estimation methods that we want to compare. Among 12829 students, 5495 students submitted ROLs and have a full set of variables. Among 5495 students, we restrict attention to the students who applied to only four-year degree programs. Students can apply to a two-year degree without entrance examination scores, and their assignments require additional conditions. Since admissions to two-year degree programs differ considerably, we exclude students who included two-year degree programs in order to estimate college preferences from our main specifications. This exclusion decreases the sizes of choice sets in the analysis and provides uniform priority rules among the students. Also, we drop unassigned students, students who submitted a complete ROL, and students who submitted a ROL with one college program. With these restrictions, we can satisfy a uniform estimation sample that meets estimation conditions of (i) outcome-based estimation method as in Fack et al. (2019), (ii) consideration set approach, (iii) using only ROL. In other words, we do not contaminate our comparison of estimation methods by differences in the sample.

Table 7 Summary statistics for different samples from the survey study

Table 7 presents summary statistics from different selections of the survey sample. Panels 2-4 provide information to make comparisons with the estimation sample that is used in Sect. 5 in the main text in which summary statistics are in panel 1. The selected samples differ in being admitted to a program or not, the length of the submitted ROL, and submitting a ROL.

The sample in panel 2 includes, in addition to our estimation sample, students who submitted a one-college program ROL or a complete ROL. The entrance examination and high school GPA scores, as well as observable characteristics of these two samples, are quite similar. Exclusion of students who submitted a complete ROL is a required condition for the proposed estimation strategy, and therefore, observing balanced samples mitigates bias concerns related to differences in the estimation samples.

In Panel 3, entrance examination scores differ compared to panel 1. Since the sample in panel 3 includes students who were not admitted to a four-year degree program, it is expected to observe lower examination scores. Exclusion of unassigned students who had submitted a ROL shows the importance of the proposed consideration set approach. Since the outcome-based estimation approach cannot utilize the ROL information of these students, estimation results alter just because of the differences between the selected samples.

Panel 4 presents the summary statistics of students who did not submit a ROL. Some of these students cannot qualify to submit a ROL because they do not pass the score threshold. The other students prefer not to submit a ROL even though they are eligible to so. In our data, on average, the entrance examination and high school GPA scores of the students who did not submit a ROL are lower than the scores of other selected samples. Furthermore, these students come from families of lower income and lower parental education. College preferences of these students cannot be included in the analysis because we do not have their preference lists.

We perform a t-test to investigate the difference between the lengths of submitted ROLs by whether a student is admitted to a program or not. Among all students in the survey study who submitted a ROL, we do not find any statistically significant difference between the lengths of the submitted ROLs. The average length of ROLs among the assigned students is 13.15. The average length of ROLs among unassigned students is 13.12. The difference is 0.03 with a p-value of 0.91.

Finally, to compare the frequencies of the selected number of college programs among the students in the estimation sample with all survey participants, we prepared Fig. 10. This figure shows that the selected sample does not show a different pattern compared to all survey participants that have submitted a ROL.

1.5.1 Robustness checks

While the proposed consideration sets method identifies the true preferences parameters of students under required conditions, we present in Table 11 the estimation results with other samples to make comparisons. We applied $Cons-Sets^2$ method, as in Table 5, (1) to all survey participants who submitted a ROL, (2) all admitted students, and (3) all students who submitted an incomplete ROL. Different from the estimation sample whose estimation results we presented in Table 5, two-year degree colleges and students who have applied to these programs are now in our estimation sample.³⁵

Since the samples of these estimations change, we need to be careful about interpreting and comparing the results. The magnitudes of the parameter estimates are changing across different samples, but the directions of the effects are the same. This result provides another point of robustness for at least for the directions of the estimates. Sample restrictions and/or choice set restrictions are requirements of the true preference estimation methods in the centralized school/college admission literature. However, Table 11 provides a positive result for the consideration sets method in terms of robustness in many different estimation samples.

Finally, we present estimation results for cases that use different subsamples. We estimate student preferences with students who submitted a ROL with fewer than 23 and 22 college programs and more than 2 and 3 college programs. We also exclude students whose ROLs’ including 10 college programs. These estimation results show that estimates for college preferences are stable across different subsamples, and provide additional evidence for the independence of student college preferences and submitted numbers of college programs in ROLs. Note that preference estimates are different from those in Table 5 Column 5 and Table 12 Column 1 (i.e., the original sample for the estimation) because of the differences in the empirical specifications. We omit college fixed effects from our regressions in order not to contaminate our results due to the changes in the sample sizes across estimation results in Table 12.

1.6 Welfare analysis details

The first step of performing a counterfactual admission analysis is to predict unrestricted true preference orders of students. Parameter estimates of the utility function in Equation 2 allow us to predict cardinal utilities for each student–college program pair and to create personal preference orders. When constructing the predicted true preference orders of college programs for each student, we follow the following principles to incorporate information in ROLs into our analysis:

• If a student submits an incomplete ROL, we deduce that the last choice in her ROL indicates the student’s reservation utility, and we do not add any program that has a lower expected cutoff score and predicted cardinal utility in her predicted true preference order.

• If a student submits an incomplete ROL, we include programs that have higher expected cutoff scores than the highest expected cutoff scores in her ROL, and these programs have higher predicted cardinal utilities than her selected choices in her predicted true preference order.

• If a student submits a complete ROL, in addition to programs that have higher expected cutoff scores and higher predicted cardinal utilities, we include programs that have higher expected cutoff scores than the lowest expected cutoff score in her ROL and the programs that have higher predicted cardinal utilities than the student’s selected choices in her predicted true preference order.

Table 8 Welfare analysis: rankings

After the construction of the predicted true preference orders, we perform counterfactual admission simulations with only entrance examination score-based priorities and only high school GPA score-based priorities. Since our data only include a small sample of students from a large college admission mechanism, we assume that the changes in students’ scores in our sample have negligible effects on college admission outcomes, and therefore, we can use the same cutoff scores to generate admissions with counterfactual scores. In the counterfactual admission, we also ignore:

• Capacity constraints of college programs,

• The limitation in the ROL’s length,

• Changes in the ROL construction strategies.

In the counterfactual admissions, we adopt the following rule for student assignment: a student is assigned to her most preferred college program among the programs with lower cutoff scores than the student’s score.

1.6.1 Tables and figures

Table 9 Fraction of students who include the top 5% programs in their ROLs by score percentiles

Table 10 Estimation results: interactions

Table 11 Estimation results by selected samples

Table 12 Estimation results

Fig. 5

First college choice of the top 100 students. Notes: Each bar plot shows the top 100 ranked students’ first college choices according to their education track. Quantitative, verbal, and composite education tracks are ordered from top to bottom

Fig. 6

Top, median, and bottom 100 students’ first college choices. Notes: Bar plot shows the top, median, and bottom 100 students’ first college choices from top to bottom. These bar plots are prepared from quantitative education track rankings

Fig. 7

Number of surveyed schools by city

Fig. 8

An illustration of consideration sets construction under different scenarios. Notes: In this figure, a blue box indicates the ROL of a candidate and a black box indicates the set of all college program options. The numerical orders and $C_k$ indicate the ranking in the submitted ROL (in the blue box), which is observed by the econometrician. $C^*$ is the assigned college in the admissions and is also observed by an econometrician. ${\tilde{p}}$ denotes the student’s score (i.e., priority). The cutoff score index line shows the college programs’ cutoff score orders in the program sets (i.e., black box). The curly brackets that are on the right side of the black boxes 1, 2, and 3 indicate consideration sets (P) for a student who has a score ${\tilde{p}}$ and has submitted a ROL indicated by the blue box. (Color figure online)

Fig. 9

An illustration of differences among the preference estimation methods in Table 5. Notes: From left to right, each figure illustrates the choice sets of the corresponding estimation presented in Table 5. In this figure, a blue box indicates the ROL of a candidate and a black box indicates the set of all college program options. The numerical orders and $C_k$ indicate the ranking in the submitted ROL (in the blue box), which is observed by the econometrician. $C^*$ is the assigned college in the admissions and is also observed by an econometrician. ${\tilde{p}}$ denotes student’s score (i.e., priority). The cutoff score index line shows the college programs’ cutoff score orders in the program sets (i.e., black boxes). The curly bracket that is on the right side of the 3rd black box indicates the feasible college program set (F) for a student that has a ${\tilde{p}}$ score. The curly brackets that are on the right side of the 4th and 5th black boxes indicate consideration sets (P) for a student who has a score ${\tilde{p}}$ and has submitted a ROL indicated by the blue box. (Color figure online)

Fig. 10

Frequencies of college program choices in ROLs in the estimation sample