Counseling University Instructors Based on Student Evaluations of Their Teaching Effectiveness: A Multilevel Test of its Effectiveness Under Consideration of Bias and Unfairness Variables

Abstract

Counseling instructors using evaluations made by their students has shown to be a fruitful approach to enhancing teaching quality. However, prior experimental studies are questionable in terms of external validity. Therefore, we conducted a non-experimental intervention study in which all of the courses offered by a specific department at a German university were evaluated twice with a standardized student evaluation questionnaire (HILVE-II; overall 44 instructors, 140 courses, and 2,546 student evaluations). Additionally, twelve full time instructors received counseling after the first measurement point. Long-term effects over a period of 2 years and transfer effects to other courses were analyzed using multi-level analyses with three levels. Possible influences by bias and unfairness variables were controlled for. Our results indicate a moderate to large effect of counseling on teaching quality. In conclusion, if students’ evaluations are accompanied by counseling based on the evaluation results, they present a useful method to assure and increase teaching quality in higher education.

Appendix

Estimation of the Bias Model in Four Steps

In the multi-level estimation of the distorting influences of potential bias and unfairness variables, the null model was expanded to a bias model in four steps described below. This takes into consideration bias and unfairness variables on all three levels (see Table 4) in accordance with recommendations made by Snijders and Bosker (2002, p. 86).

Step 1 (Bias Variables on the Student Level)

In the first step all six potential bias variables associated with the student level were added to the model (centered on the grand mean) and variations of the six regression coefficients between courses were allowed for (random slopes). The regression equation on the student level of the null model was therefore expanded to Y _ijk  = π_0jk  + Σ _p π _pjk ·X _pijk  + e _ijk , whereby X _p represents the pth of the six predictors. For each of the p regression weights π _pjk , the equation π _pjk  = β_p0k + r _pjk with R _p  = Var(r _pjk ) was additionally formulated on the course level in order to permit predictors to vary between courses. The estimation revealed that the effects of some variables vary between courses. The corresponding effects were retained as random effects, all remaining effects were fixed (i.e. setting r _pjk  = 0). Variables which revealed no significant relationship to the criterion and which effect did not vary between courses were removed from the model.

Step 2 (Bias Variables on the Course Level)

The second step comprised the block-wise addition of the five structural course characteristics, the four course mean scores and the two course distributions into the model as predictors of the intercept (centered on the grand mean; variations between instructors were permitted). Formally speaking, the parameters were estimated using the equation π_0jk  = β_00k  + Σ _q β_0qk ·X _qjk  + r _0jk on the course level (with X _q as the qth term of the 11 predictors) as well as the eleven equations β_0qk  = γ_0q0 + u _0qk on the instructor level with U _0q  = Var(u _0qk ). The influences of all course characteristics were constant across instructors; consequently, all course level effects were fixed (i.e. setting u _0qk  = 0). Course-level variables which did not predict the criterion variable significantly were therefore removed from the model.

Step 3 (Cross-Level Interactions Between Bias Variables on Course and Student Level)

In the third step, a test was made to determine whether the between course variations in the regression coefficients π _pjk for three student variables (gender, perceived topic relevance and prior knowledge) could be explained by characteristics of the courses themselves. To this end, the regression coefficients for the three bias variables named were treated as dependent variables (slopes as outcomes). In this case, a significant effect indicated a cross-level interaction in which the effect of a variable on the student level is moderated by a variable on the course level. Formally, these effects are represented by three regression equations π _pjk  = β_p0k + Σ _l β _plk ·X _ljk  + r _pjk for the pth of the three student bias variables, whereby X _l depicts the lth of the course characteristics included (coefficients β _plk fixed on instructor level). Results confirmed two cross-level interactions.

Step 4 (Bias Variables on the Instructor Level)

In the fourth step, a final expansion incorporated the three potential bias variables at the instructor level. They were introduced in the intercept regression equation (grand mean centered): β_00k  = γ₀₀₀ + Σ _m γ_00m ·X _mk  + u _00k with X _m as the mth of the three predictors. None of them significantly biased student evaluations of teaching quality. They were therefore removed from the model.