PISA Student Nonresponse Adjustment Procedures

Abstract

Large-scale surveys experience nonresponse that might result in biases in the results. In educational surveys, nonresponse can occur at three levels, i.e. the school, the student and the item. Traditionally, large-scale surveys in education have compensated for unit nonresponse, both at the school and at the student level, by applying survey weight adjustments.

IEA TIMSS and PIRLS surveys, for example, apply a school nonresponse adjustment calculated within the explicit stratum level and a student nonresponse adjustment calculated within each school. Since its first data collection, PISA has implemented the school nonresponse adjustment at a lower level than the explicit stratum, using combinations of the implicit stratification variables. The student nonresponse adjustment procedures have continuously evolved since the first data collection to better reflect differential response rates. In PISA 2006, the student nonresponse adjustment was computed by creating, per school nonresponse adjustment cell, four student nonresponse adjustment cells based on relative grade level, and gender.

This study compares the efficiency of the 2003 and the 2006 procedures for reducing potential bias due to differential student response rates. The comparison highlights the potential of the 2006 method for reducing bias due to differential response rates according to gender and grade.