Analyzing Large-Scale Assessment Data with Multilevel Analyses: Demonstration Using the Programme for International Student Assessment (PISA) 2018 Data

Abstract

The present chapter provides a tutorial with example demonstration designed to guide the reader through the complexities and unique challenges associated with analyzing large-scale assessment data through multilevel modeling techniques. The reader will learn how to use relevant literature to specify an appropriate model considering effects at various levels; how to address large-scale data complexities including sampling weights and plausible values; and how to estimate and interpret the results from such a model. The chapter will provide a conceptual background and short description of each of these topics, followed by an example demonstration using Programme for International Student Assessment (PISA) 2018 data. The demonstration will be used to provide a concrete example for analysis, including the process of model specification, estimation, and interpretation. In addition, annotated R syntax and R output associated with aspects of modeling will be provided so that readers will easily be able to conduct their own multilevel analyses with PISA data in order to answer their own research questions.

Appendix: R syntax

#R syntax for demonstration analysis

############################## #1. Import two packages for analysis of complex survey data and set working directory

#intsvy package will be used to import data & compute descriptive statistics library(intsvy)

#WeMix package will be used for estimating multilevel models #this package allows including of sampling weights in the model with the “mix” function library(WeMix)

#set working directory (wd) where PISA data was saved setwd("C:/Users/file path here")

############################## #2. Save variable information and import data

#prints/saves variable labels and names of countries in a text file in working directory pisa.var.label(folder=file.path(getwd(),"PISA 2018"),              school.file="CY07_MSU_SCH_QQQ.sav",              student.file="CY07_MSU_STU_QQQ.sav")

#selects & merges data; acheivement & weight variables selected by default mydata <- pisa.select.merge(folder = file.path(getwd(), "PISA 2018"),              school.file = "CY07_MSU_SCH_QQQ.sav",              student.file = "CY07_MSU_STU_QQQ.sav",              student = c("ESCS","ST004D01T","GFOFAIL","MASTGOAL"),              school = c("W_SCHGRNRABWT","SCHSIZE"),              countries = c("USA"))

############################## #3. Data preparation

mydata$SCHSIZE_TH<-mydata$SCHSIZE/1000 #Size in thousands of students mydata$Male<-mydata$ST004D01T #recode gender in direction of effect

##############################

#4. Descriptive statistics #using functions from intsvy package pisa.mean.pv(pvlabel="MATH",data=mydata) pisa.mean(variable="MASTGOAL",data=mydata) pisa.mean(variable="ESCS",data=mydata) pisa.mean(variable="Male",data=mydata) pisa.mean(variable="GFOFAIL",data=mydata) pisa.mean(variable="SCHSIZE_TH",data=mydata) pisa.table(variable="Male",data=mydata) #frequency distribution pisa.ben.pv(pvlabel="MATH",data=mydata) #percents at proficiency levels pisa.mean.pv(pvlabel="MATH",by="Male",data=mydata) #for plausible values pisa.rho(variable=c("PV1MATH","MASTGOAL","ESCS","Male","GFOFAIL","SCHSIZE_TH"),              data=mydata) #correlations pisa.rho(variable=c("PV2MATH","MASTGOAL","ESCS","Male","GFOFAIL","SCHSIZE_TH"),              data=mydata) #correlations pisa.rho(variable=c("PV3MATH","MASTGOAL","ESCS","Male","GFOFAIL","SCHSIZE_TH"),              data=mydata) #correlations

############################## #5. Estimate multilevel models

#Math DV# #listwise delete newdata<-subset(mydata, select = c(PV1MATH,PV2MATH,PV3MATH,CNTSCHID,W_FSTUWT,W_SCHGRNRABWT,              ESCS,Male,SCHSIZE_TH,GFOFAIL)) newdata <- na.omit(newdata)

#run models M1a<-mix(PV1MATH~1+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M1b<-mix(PV2MATH~1+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M1c<-mix(PV3MATH~1+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M2a<-mix(PV1MATH~1+ESCS+Male+SCHSIZE_TH+GFOFAIL+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M2b<-mix(PV2MATH~1+ESCS+Male+SCHSIZE_TH+GFOFAIL+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M2c<-mix(PV3MATH~1+ESCS+Male+SCHSIZE_TH+GFOFAIL+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT"))

#MASTGOAL DV# #listwise delete newdata<-subset(mydata, select = c(MASTGOAL,CNTSCHID,W_FSTUWT,W_SCHGRNRABWT,              ESCS,Male,SCHSIZE_TH,GFOFAIL)) newdata <- na.omit(newdata)

#run models M3<-mix(MASTGOAL~1+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT")) M4<-mix(MASTGOAL~1+ESCS+Male+SCHSIZE_TH+GFOFAIL+(1|CNTSCHID),data=newdata,              weights=c("W_FSTUWT","W_SCHGRNRABWT"))

############################## #6. Combine results from multiple models to account for plausible values

#this demonstrates the procedure for the intercept from the empty model

#average of 3 intercept estimates represents combined coefficient mean(M1a$coef,M1b$coef,M1c$coef)

#SE is computed with the following formula #M is the number of PV; 3 for demonstration; PISA 2018 has 10 PV total M<-3 mean(M1a$SE,M1b$SE,M1c$SE) + (1+(1/M))*var(c(M1a$coef,M1b$coef,M1c$coef))