Toward a set of measures of student learning outcomes in higher education: evidence from Brazil

Abstract

The main objective of this study was to work toward the development of a number of measures of student learning outcomes (SLOs) in higher education. Specifically, we used data from Exame Nacional de Desempenho dos Estudantes (ENADE), a college-exit examination developed and used in Brazil. The fact that Brazil administered the ENADE to both freshmen and senior students provided a unique opportunity to get a first approximation of the general and subject area knowledge gained in different programs. The results suggested that, on average, students in the three different categories of programs were gaining valuable general and subject area knowledge. The gains in the subject area were of a larger magnitude than those in the general knowledge component of the test. This study contributes to the field by providing empirical and visually compelling evidence related to SLOs gains in higher education.

Appendix: Combining effect sizes to produce program-level/engineering effect sizes

The method to combine measures of effect sizes of different fields (e.g., mathematics and computer science) into a general area (e.g., STEM) measure comes from procedures developed in the meta-analyses field.²¹ There are two approaches that are generally used to combine effect sizes: fixed and random effects models (Borenstein et al. 2007). The fixed effects model assumes that the effect size of each clinical trial, in our case each field, is an estimate of a true and unique correct effect size. Each effect size is different because of sampling error, and thus an average of the measures is a good estimate of this combined effect size. These models traditionally use a weighted average because clinical trials that involve more people should weigh more than trials with less people involved. The weight used is the inverse of the variance of the effect size, which is calculated as

$$w_{i} = {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {\sigma_{i} }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${\sigma_{i} }$}}\;\;{\text{and}}\;\;\sigma_{i}^{2} = \frac{{ne_{i} + nc_{i} }}{{ne_{i} nc_{i} }} + \frac{{d_{i}^{2} }}{{2\left( {ne_{i} + nc_{i} } \right)}}$$

where ne and nc are the number of people in the experimental and control groups and the subscript i refers to each field’s measures. The combined effect size is then calculated as

$$d = \frac{{\sum {w_{i} d_{i} } }}{{\sum {w_{i} } }}$$

The variance of the combined effect size is calculated as

$$\sigma_{i}^{2} = {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {\sum {w_{i} } }}}\right.\kern-0pt} \!\lower0.7ex\hbox{${\sum {w_{i} } }$}}$$

and from this, one can calculate a confidence interval for the resulting effect size.

The random effects model does not assume that all trials, or in our case, all field have the same “true” effect size, but instead assumes that the true effect size for each field is normally distributed around a mean true measure, with a variance, called variance between treatments, denoted by τ (tau). The statistical computations estimate not only the mean true value, but also the between-treatment variance. We refer the reader to Borenstein et al. (2007) for a more complete explanation and the specific formulas. In this study, we chose the random effect model because it is clear from Fig. 1a–c that each field has a different “true” effect sizes, sometimes statistically significantly different. Specifically, we use the DerSimonian–Laird estimation for the between-study variance as implemented in the package meta of the software R (Table 3).

Table 3 Effect sizes for the general and specific examinations, for all students, for students of high and low income, and for students in private and public universities, for all major field of study