Abstract
International large-scale assessments in education (ILSAs) follow complex sampling designs and ultimately create hierarchical data structures with students nested in classrooms, classrooms in schools, schools in regions, etc. To describe adequately key issues in education, such as socioeconomic gaps in academic achievement or the relations among school characteristics and student achievement using ILSA data, researchers need to consider the hierarchical data structure in statistical models. Multilevel modeling is one approach to account for such hierarchies and consider variables at different levels of analysis. This chapter provides an overview of the prominent multilevel modeling approaches to analyzing ILSA data, illustrates and discusses their strengths and weaknesses, and highlights the key methodological decisions researchers have to take in this context. The first part reviews the current practices of multilevel modeling in secondary analyses of ILSA data. This rapid systematic review is followed by a second part in which we present, illustrate, and discuss multilevel modeling approaches, including multilevel regression, multilevel structural equation models, and multilevel mixture models. Next to model estimation and fit evaluation, we review key issues associated with the multilevel modeling of ILSA data and focus on handling plausible values, multigroup and incidental multilevel data structures, and weighting. Our chapter provides worked examples showcasing the potential of multilevel modeling for ILSA data analysis.
References
Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490–1528. https://doi.org/10.1177/0149206313478188
Asparouhov, T. (2005). Sampling weights in latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 12(3), 411–434. https://doi.org/10.1207/s15328007sem1203_4
Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics – Theory and Methods, 35(3), 439–460. https://doi.org/10.1080/03610920500476598
Bellens, K., Van Damme, J., Van Den Noortgate, W., Wendt, H., & Nilsen, T. (2019). Instructional quality: Catalyst or pitfall in educational systems’ aim for high achievement and equity? An answer based on multilevel SEM analyses of TIMSS 2015 data in Flanders (Belgium), Germany, and Norway. Large-scale Assessment in Education, 7(1). https://doi.org/10.1186/s40536-019-0069-2
Berkowitz, R., Moore, H., Astor, R. A., & Benbenishty, R. (2017). A research synthesis of the associations between socioeconomic background, inequality, school climate, and academic achievement. Review of Educational Research, 87(2), 425–469. https://doi.org/10.3102/0034654316669821
Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.
Cai, T. (2012). Investigation of ways to handle sampling weights for multilevel model analyses. Sociological Methodology, 43(1), 178–219. https://doi.org/10.1177/0081175012460221
Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., … Lee, R. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. https://doi.org/10.3102/0034654308325581
Diez Roux, A. V. (2002). A glossary for multilevel analysis. Journal of Epidemiology and Community Health, 56(8), 588–594. https://doi.org/10.1136/jech.56.8.588
Else-Quest, N. M., Hyde, J. S., & Linn, M. C. (2010). Cross-national patterns of gender differences in mathematics: A meta-analysis. Psychological Bulletin, 136(1), 103–127. https://doi.org/10.1037/a0018053
Enders, C. K. (2010). Applied missing data analysis. Guilford Press.
Enders, C. K., & Mansolf, M. (2018). Assessing the fit of structural equation models with multiply imputed data. Psychological Methods, 23(1), 76–93. https://doi.org/10.1037/met0000102
Enders, C. K., Mistler, S. A., & Keller, B. T. (2016). Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation. Psychological Methods, 21(2), 222–240. https://doi.org/10.1037/met0000063
Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138. https://doi.org/10.1037/1082-989X.12.2.121
Garritty, C., Stevens, A., Gartlehner, G., King, V., Kamel, C., & On behalf of the Cochrane Rapid Reviews Methods Group. (2016). Cochrane Rapid Reviews Methods Group to play a leading role in guiding the production of informed high-quality, timely research evidence syntheses. Systematic Reviews, 5(1), 184. https://doi.org/10.1186/s13643-016-0360-z
Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
Gonzalez, E., & Rutkowski, L. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 3, 125–156. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_03_Chapter_6.pdf
Grund, S., Lüdtke, O., & Robitzsch, A. (2018). Multiple imputation of missing data for multilevel models: Simulations and recommendations. Organizational Research Methods, 21(1), 111–149. https://doi.org/10.1177/1094428117703686
Grund, S., Lüdtke, O., & Robitzsch, A. (2019). Missing data in multilevel research. In The handbook of multilevel theory, measurement, and analysis (pp. 365–386). American Psychological Association.
Heck, R. H., & Thomas, S. L. (2015). An introduction to multilevel modeling techniques: MLM and SEM approaches using Mplus (3rd ed.). Routledge.
Henry, K. L., & Muthén, B. (2010). Multilevel latent class analysis: An application of adolescent smoking typologies with individual and contextual predictors. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 193–215. https://doi.org/10.1080/10705511003659342
Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). Routledge.
Hox, J. J., van Buuren, S., & Jolani, S. (2015). Incomplete multilevel data: Problems and solutions. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 39–62). Information Age Publishing Inc..
Hsu, H.-Y., Lin, J.-H., Kwok, O.-M., Acosta, S., & Willson, V. (2017). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo Study. Educational and Psychological Measurement, 77(1), 5–31. https://doi.org/10.1177/0013164416642823
Jak, S. (2014). Testing strong factorial invariance using three-level structural equation modeling. Frontiers in Psychology, 5(745). https://doi.org/10.3389/fpsyg.2014.00745
Jak, S. (2019). Cross-level invariance in multilevel factor models. Structural Equation Modeling: A Multidisciplinary Journal, 26(4), 607–622. https://doi.org/10.1080/10705511.2018.1534205
Janis, R. A., Burlingame, G. M., & Olsen, J. A. (2016). Evaluating factor structures of measures in group research: Looking between and within. Group Dynamics: Theory, Research, and Practice, 20(3), 165–180. https://doi.org/10.1037/gdn0000043
Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). Sage.
Kaplan, D., & Su, D. (2016). On matrix sampling and imputation of context questionnaires with implications for the generation of plausible values in large-scale assessments. Journal of Educational and Behavioral Statistics, 41(1), 57–80. https://doi.org/10.3102/1076998615622221
Kelcey, B., Cox, K., & Dong, N. (2019). Croon’s bias-corrected factor score path analysis for small- to moderate-sample multilevel structural equation models. Organizational Research Methods(0), 1094428119879758. https://doi.org/10.1177/1094428119879758
Kim, E. S., Dedrick, R. F., Cao, C., & Ferron, J. M. (2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881–898. https://doi.org/10.1080/00273171.2016.1228042
Kim, J.-S., Anderson, C. J., & Keller, B. (2014). Multilevel analysis of assessment data. In L. Rutkowski, M. V. Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 390–425). CRC Press.
Klieme, E. (2013). The role of large-scale assessments in research on educational effectiveness and school development. In M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto (Eds.), The role of international large-scale assessments: Perspectives from technology, economy, and educational research (pp. 115–147). Springer.
Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). Guilford Press.
Kuger, S., & Klieme, E. (2016). Dimensions of context assessment. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning: An international perspective (pp. 3–37). Springer.
Lachowicz, M. J., Preacher, K. J., & Kelley, K. (2018). A novel measure of effect size for mediation analysis. Psychological Methods, 23(2), 244–261. https://doi.org/10.1037/met0000165
Lachowicz, M. J., Sterba, S. K., & Preacher, K. J. (2014). Investigating multilevel mediation with fully or partially nested data. Group Processes & Intergroup Relations, 18(3), 274–289. https://doi.org/10.1177/1368430214550343
Lai, M. H. C., & Kwok, O.-M. (2015). Examining the rule of thumb of not using multilevel modeling: The “Design effect smaller than two” rule. The Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229
LaRoche, S., Joncas, M., & Foy, P. (2016). Sample design in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 3.1–3.38). Boston College, TIMSS & PIRLS International Study Center.
Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics – Theory and Methods, 46(22), 11341–11357. https://doi.org/10.1080/03610926.2016.1267764
Laukaityte, I., & Wiberg, M. (2018). Importance of sampling weights in multilevel modeling of international large-scale assessment data. Communications in Statistics – Theory and Methods, 47(20), 4991–5012. https://doi.org/10.1080/03610926.2017.1383429
Little, T. D. (2013). Longitudinal structural equation modeling. The Guilford Press.
Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21–39. https://doi.org/10.1037/1082-989X.10.1.21
Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376
Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. https://doi.org/10.1037/a0012869
Lüdtke, O., Robitzsch, A., & Grund, S. (2017). Multiple imputation of missing data in multilevel designs: A comparison of different strategies. Psychological Methods, 22(1), 141–165. https://doi.org/10.1037/met0000096
Mäkikangas, A., Tolvanen, A., Aunola, K., Feldt, T., Mauno, S., & Kinnunen, U. (2018). Multilevel latent profile analysis with covariates: Identifying job characteristics profiles in hierarchical data as an example. Organizational Research Methods, 21(4), 931–954. https://doi.org/10.1177/1094428118760690
Marsh, H. W., Dowson, M., Pietsch, J., & Walker, R. (2004). Why multicollinearity matters: A reexamination of relations between self-efficacy, self-concept, and achievement. Journal of Educational Psychology, 96(3), 518–522. https://doi.org/10.1037/0022-0663.96.3.518
Marsh, H. W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A. J. S., Abduljabbar, A. S., & Köller, O. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665
Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling: A Multidisciplinary Journal, 16(2), 191–225. https://doi.org/10.1080/10705510902751010
Masyn, K. E. (2013). Latent class analysis and finite mixture modeling. In The Oxford handbook of quantitative methods: Statistical analysis (Vol. 2, pp. 551–611). Oxford University Press.
Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951–966. https://doi.org/10.1037/a0028380
McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52(2), 200–215. https://doi.org/10.1080/00273171.2016.1262236
Mislevy, R. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196. https://doi.org/10.1007/BF02294457
Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097
Morin, A. J. S., & Marsh, H. W. (2015). Disentangling shape from level effects in person-centered analyses: An illustration based on university teachers’ multidimensional profiles of effectiveness. Structural Equation Modeling, 22(1), 39–59. https://doi.org/10.1080/10705511.2014.919825
Morin, A. J. S., Marsh, H. W., Nagengast, B., & Scalas, L. F. (2014). Doubly latent multilevel analyses of classroom climate: An illustration. The Journal of Experimental Education, 82(2), 143–167. https://doi.org/10.1080/00220973.2013.769412
Muthén, B. O., & Asparouhov, T. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In Handbook for advanced multilevel analysis (pp. 15–40). Routledge/Taylor & Francis Group.
Muthén, B. O., & Asparouhov, T. (2017). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47(4), 637–664. https://doi.org/10.1177/0049124117701488
Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. V. Marsden (Ed.), Sociological methodology (pp. 267–316). Blackwell.
Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide (8th ed.). Muthén & Muthén.
Nagengast, B., & Marsh, H. W. (2011). The negative effect of school-average ability on science self-concept in the UK, the UK countries and the world: The Big-Fish-Little-Pond-Effect for PISA 2006. Educational Psychology, 31(5), 629–656. https://doi.org/10.1080/01443410.2011.586416
Nagengast, B., & Marsh, H. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104(4), 1033–1053. https://doi.org/10.1037/a0027697
Nilsen, T., Bloemeke, S., Yang Hansen, K., & Gustafsson, J.-E. (2016). Are school characteristics related to equity? The answer may depend on a country’s developmental level. IEA Policy Briefs, 10. Retrieved from https://www.iea.nl/publications/series-journals/policy-brief/april-2016-are-school-characteristics-related-equity
Nylund-Gibson, K., & Choi, A. Y. (2018). Ten frequently asked questions about latent class analysis. Translational Issues in Psychological Science, 4(4), 440–461. https://doi.org/10.1037/tps0000176
O’Connell, A. A., Yeomans-Maldonado, G., & McCoach, D. B. (2015). Residual diagnostics and model assessment in a multilevel framework: Recommendations toward best practice. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 97–135). Information Age Publishing, Inc..
OECD. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing.
OECD. (2019a). PISA 2018 results (Vol. I). OECD Publishing.
OECD. (2019b). TALIS 2018 results (Vol. I). OECD Publishing.
Preacher, K. J. (2015). Advances in mediation analysis: A survey and synthesis of new developments. Annual Review of Psychology, 66(1), 825–852. https://doi.org/10.1146/annurev-psych-010814-015258
Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2016). Multilevel structural equation models for assessing moderation within and across levels of analysis. Psychological Methods, 21(2), 189–205. https://doi.org/10.1037/met0000052
Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications.
Rohatgi, A., & Scherer, R. (2020). Identifying profiles of students’ school climate perceptions using PISA 2015 data. Large-scale Assessments in Education, 8(1), 4. https://doi.org/10.1186/s40536-020-00083-0
Rust, K. (2014). Sampling, weighting, and variance estimation in international large-scale assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117–154). CRC Taylor & Francis.
Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170
Rutkowski, L., & Zhou, Y. (2014). Using structural equation models to analyze ILSA data. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 425–450). CRC Press.
Ryu, E. (2014a). Factorial invariance in multilevel confirmatory factor analysis. British Journal of Mathematical and Statistical Psychology, 67(1), 172–194. https://doi.org/10.1111/bmsp.12014
Ryu, E. (2014b). Model fit evaluation in multilevel structural equation models. Frontiers in Psychology, 5(81). https://doi.org/10.3389/fpsyg.2014.00081
Ryu, E. (2015). Multiple group analysis in multilevel structural equation model across level 1 groups. Multivariate Behavioral Research, 50(3), 300–315. https://doi.org/10.1080/00273171.2014.1003769
Ryu, E., & Mehta, P. (2017). Multilevel factorial invariance in n-Level Structural Equation Modeling (nSEM). Structural Equation Modeling: A Multidisciplinary Journal, 24(6), 936–959. https://doi.org/10.1080/10705511.2017.1324311
Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16(4), 583–601. https://doi.org/10.1080/10705510903203466
Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference Chi-square test statistic. Psychometrika, 75(2), 243–248. https://doi.org/10.1007/s11336-009-9135-y
Scherer, R., & Gustafsson, J.-E. (2015). Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: An application of multilevel bifactor structural equation modeling. Frontiers in Psychology, 6.
Scherer, R., Nilsen, T., & Jansen, M. (2016). Evaluating individual students’ perceptions of instructional quality: An investigation of their factor structure, measurement invariance, and relations to educational outcomes. Frontiers in Psychology, 7(110). https://doi.org/10.3389/fpsyg.2016.00110
Scherer, R., Tondeur, J., Siddiq, F., & Baran, E. (2018). The importance of attitudes toward technology for pre-service teachers’ technological, pedagogical, and content knowledge: Comparing structural equation modeling approaches. Computers in Human Behavior, 80, 67–80. https://doi.org/10.1016/j.chb.2017.11.003
Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499. https://doi.org/10.3102/0034654307310317
Silva, B. C., Bosancianu, C. M., & Littvay, L. (2019). Multilevel structural equation modeling. Sage.
Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An Introduction to basic and advanced multilevel modeling (2nd ed.). Sage.
Stapleton, L. M. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 475–502. https://doi.org/10.1207/S15328007SEM0904_2
Stapleton, L. M. (2013). Multilevel structural equation modeling with complex sample data. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 521–562). Information Age Publishing, Inc.
Stapleton, L. M. (2014). Incorporating sampling weights into single- and multilevel analyses. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 363–388). CRC Taylor & Francis.
Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200
Van den Noortgate, W., Opdenakker, M.-C., & Onghena, P. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16(3), 281–303. https://doi.org/10.1080/09243450500114850
von Davier, M., Gonzalez, E., & Mislevy, R. J. (2009). What are plausible values and why are they useful? IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 9–36. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf
Wang, W., Liao, M., & Stapleton, L. (2019). Incidental second-level dependence in educational survey data with a nested data structure. Educational Psychology Review, 31(3), 571–596. https://doi.org/10.1007/s10648-019-09480-6
Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2), 114–128. https://doi.org/10.1016/j.stueduc.2005.05.005
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Section Editor information
Appendices
Appendices
Appendix A: Database Search Terms
The following strategy informed the search for relevant publications in the databases ERIC and PsycINFO through the search service Ovid (with the resultant numbers of entries in brackets; 21 November 2019):
1 | (PISA or TIMSS or PIRLS or PIAAC or ICILS or ICCS).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (5690) |
2 | (hierarchical model or hierarchical linear model or linear mixed-effects model or random coefficient model or random effects model or multilevel regression or generalized linear mixed model or GLMM or multilevel model* or multilevel analysis).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (16437) |
3 | (Multilevel path analysis or multilevel path model or multilevel mediation or multilevel factor analysis or multilevel CFA or multilevel confirmatory factor analysis or MCFA or multilevel SEM or MSEM or multilevel structural equation model* or multilevel latent covariate model or multilevel IRT or multilevel item response theory).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (1301) |
4 | 2 or 3 (17472) |
5 | 1 and 4 (227) |
6 | remove duplicates from 5 (226) |
Appendix B: PRISMA Statement
Appendix C: Description of the Variables Used in the Illustrative Examples
Variable | Original variable label (PISA 2015) | Variable label | Scale |
---|---|---|---|
Identification variables | |||
Student Identifier | CNTSTUID | Country-specific Student ID | Numeric (nominal) |
School Identifier | CNTSCHID | Country-specific School ID | Numeric (nominal) |
Country Identifier | CNTRYID | Country Identifier | 208 = Denmark |
246 = Finland | |||
352 = Iceland | |||
578 = Norway | |||
752 = Sweden | |||
Student-level variables (Cognitive tests and student questionnaire) | |||
Science achievement | PV1SCIE-PV10SCIE | Plausible values of scientific literacy | Continuous |
Socioeconomic status | ESCS | Index of economic, social, and cultural status (WLE) | Continuous |
Gender | ST004D01T | Student gender (recoded) | 0 = Male |
1 = Female | |||
Home possessions | HOMEPOS | Home possessions (WLE) | Continuous |
Parents’ occupation | HISEI | Index highest parental occupational status | |
Immigration status | IMMIG | Index Immigration status (recoded) | 0 = Native |
1 = First- or second-generation immigration | |||
Grade repetition | REPEAT | Grade repetition | 0 = No repetition |
1 = Grade repetition | |||
Parents’ education | PARED | Index highest parental education in years of schooling | Continuous (in years) |
Adaptive instruction (How often do these things happen in your lessons for this <school science> course?) | ADINST | Adaption of instruction (WLE) | Continuous |
ST107Q01 | The teacher adapts the lesson to my class needs and knowledge. | 1 = Never or almost never | |
2 = Some lessons | |||
3 = Many lessons | |||
4 = Every lesson or almost every lesson | |||
ST107Q02 | The teacher provides individual help when a student has difficulties | ||
ST107Q03 | The teacher changes the structure of the lesson on a topic | ||
Disciplinary climate | DISCLISCI | Disciplinary climate in science classes (WLE) | Continuous |
Perceived feedback | PERFEED | Perceived feedback (WLE) | Continuous |
Teacher support (How often do these things happen in your <school science> lessons?) | TEACHSUP | Teacher support in a science class of students’ choice (WLE) | Continuous |
ST100Q01 | The teacher shows interest every students’ learning. | 1 = Every lesson | |
2 = Most lessons | |||
3 = Some lessons | |||
4 = Never or hardly ever | |||
ST100Q02 | The teacher gives extra help. | ||
ST100Q03 | The teacher helps students with their learning. | ||
ST100Q04 | The teacher continues teaching\students understand. | ||
ST100Q05 | Teacher gives an opportunity to express opinions. | ||
Teachers’ unfair treatment of students | UNFAIRTEACHER | Teacher fairness (Sum) | Continuous |
Instrumental science motivation | INSTSCIE | Instrumental motivation (WLE) | Continuous |
Enjoyment of science (How much do you disagree or agree with the statements about yourself below?) | JOYSCIE | Enjoyment of science (WLE) | Continuous |
ST094Q01 | I have fun when I am learning <broad science> | 1 = Strongly disagree | |
2 = Disagree | |||
ST094Q02 | I like reading about <broad science> topics. | 3 = Agree | |
4 = Strongly agree | |||
ST094Q03 | I am happy working on <broad science> topics. | ||
ST094Q04 | I enjoy acquiring new knowledge in <broad science>. | ||
Test anxiety | ANXTEST | Test anxiety (WLE) | Continuous |
Achievement motivation (To what extent do you disagree or agree with the following statements about yourself?) | MOTIVAT | Student attitudes, preferences, and self-related beliefs: Achieving motivation (WLE) | Continuous |
ST119Q02 | I want to be able to select from among the best opportunities available when I graduate. | 1 = Strongly disagree | |
2 = Disagree | |||
3 = Agree | |||
4 = Strongly agree | |||
ST119Q03 | I want to be the best, whatever I do. | ||
ST119Q04 | I see myself as an ambitious person. | ||
ST119Q05 | I want to be one of the best students in my class. | ||
School-level variables (Principal questionnaire) | |||
School type | PRIVATE | Private school | 0 = Public school |
1 = Private independent or government-dependent school | |||
School size | SCHSIZE | School size (sum) | Continuous |
Student-teacher ratio | STRATIO | Student-teacher ratio | Continuous |
Student behavior at school (In your school, to what extent is the learning of students hindered by the following phenomena?) | STUBEHA | Student behavior hindering learning (WLE) | Continuous |
SC061Q01 | Student truancy | 1 = Not at all | |
2 = Very little | |||
3 = To some extent | |||
4 = A lot | |||
SC061Q02 | Students skipping classes | ||
SC061Q03 | Students lacking respect for teachers | ||
Teacher behavior at school | TEACHBEHA | Teacher behavior hindering learning (WLE) | Continuous |
Weights | |||
Student weights | W_FSTUWT | Final trimmed nonresponse adjusted student weight | Continuous |
School weights | W_SCHGRNRABWT | Grade nonresponse adjusted school base weight | Continuous |
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this entry
Cite this entry
Scherer, R. (2022). Analyzing International Large-Scale Assessment Data with a Hierarchical Approach. In: Nilsen, T., Stancel-Piątak, A., Gustafsson, JE. (eds) International Handbook of Comparative Large-Scale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-030-38298-8_59-1
Download citation
DOI: https://doi.org/10.1007/978-3-030-38298-8_59-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-38298-8
Online ISBN: 978-3-030-38298-8
eBook Packages: Springer Reference EducationReference Module Humanities and Social SciencesReference Module Education