Skip to main content
Book cover

International Handbook of Comparative Large-Scale Studies in Education pp 1–55Cite as

Analyzing International Large-Scale Assessment Data with a Hierarchical Approach

  • 57 Accesses

Part of the Springer International Handbooks of Education book series (SIHE)

Abstract

International large-scale assessments in education (ILSAs) follow complex sampling designs and ultimately create hierarchical data structures with students nested in classrooms, classrooms in schools, schools in regions, etc. To describe adequately key issues in education, such as socioeconomic gaps in academic achievement or the relations among school characteristics and student achievement using ILSA data, researchers need to consider the hierarchical data structure in statistical models. Multilevel modeling is one approach to account for such hierarchies and consider variables at different levels of analysis. This chapter provides an overview of the prominent multilevel modeling approaches to analyzing ILSA data, illustrates and discusses their strengths and weaknesses, and highlights the key methodological decisions researchers have to take in this context. The first part reviews the current practices of multilevel modeling in secondary analyses of ILSA data. This rapid systematic review is followed by a second part in which we present, illustrate, and discuss multilevel modeling approaches, including multilevel regression, multilevel structural equation models, and multilevel mixture models. Next to model estimation and fit evaluation, we review key issues associated with the multilevel modeling of ILSA data and focus on handling plausible values, multigroup and incidental multilevel data structures, and weighting. Our chapter provides worked examples showcasing the potential of multilevel modeling for ILSA data analysis.

Keywords

  • Hierarchical linear model (HLM)
  • Multilevel modeling
  • Multilevel structural equation modeling
  • Nested data structure
  • Random effects models

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  • Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490–1528. https://doi.org/10.1177/0149206313478188

    CrossRef  Google Scholar 

  • Asparouhov, T. (2005). Sampling weights in latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 12(3), 411–434. https://doi.org/10.1207/s15328007sem1203_4

    CrossRef  Google Scholar 

  • Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics – Theory and Methods, 35(3), 439–460. https://doi.org/10.1080/03610920500476598

    CrossRef  Google Scholar 

  • Bellens, K., Van Damme, J., Van Den Noortgate, W., Wendt, H., & Nilsen, T. (2019). Instructional quality: Catalyst or pitfall in educational systems’ aim for high achievement and equity? An answer based on multilevel SEM analyses of TIMSS 2015 data in Flanders (Belgium), Germany, and Norway. Large-scale Assessment in Education, 7(1). https://doi.org/10.1186/s40536-019-0069-2

  • Berkowitz, R., Moore, H., Astor, R. A., & Benbenishty, R. (2017). A research synthesis of the associations between socioeconomic background, inequality, school climate, and academic achievement. Review of Educational Research, 87(2), 425–469. https://doi.org/10.3102/0034654316669821

    CrossRef  Google Scholar 

  • Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

    Google Scholar 

  • Cai, T. (2012). Investigation of ways to handle sampling weights for multilevel model analyses. Sociological Methodology, 43(1), 178–219. https://doi.org/10.1177/0081175012460221

    CrossRef  Google Scholar 

  • Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., … Lee, R. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. https://doi.org/10.3102/0034654308325581

    CrossRef  Google Scholar 

  • Diez Roux, A. V. (2002). A glossary for multilevel analysis. Journal of Epidemiology and Community Health, 56(8), 588–594. https://doi.org/10.1136/jech.56.8.588

    CrossRef  Google Scholar 

  • Else-Quest, N. M., Hyde, J. S., & Linn, M. C. (2010). Cross-national patterns of gender differences in mathematics: A meta-analysis. Psychological Bulletin, 136(1), 103–127. https://doi.org/10.1037/a0018053

    CrossRef  Google Scholar 

  • Enders, C. K. (2010). Applied missing data analysis. Guilford Press.

    Google Scholar 

  • Enders, C. K., & Mansolf, M. (2018). Assessing the fit of structural equation models with multiply imputed data. Psychological Methods, 23(1), 76–93. https://doi.org/10.1037/met0000102

    CrossRef  Google Scholar 

  • Enders, C. K., Mistler, S. A., & Keller, B. T. (2016). Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation. Psychological Methods, 21(2), 222–240. https://doi.org/10.1037/met0000063

    CrossRef  Google Scholar 

  • Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138. https://doi.org/10.1037/1082-989X.12.2.121

    CrossRef  Google Scholar 

  • Garritty, C., Stevens, A., Gartlehner, G., King, V., Kamel, C., & On behalf of the Cochrane Rapid Reviews Methods Group. (2016). Cochrane Rapid Reviews Methods Group to play a leading role in guiding the production of informed high-quality, timely research evidence syntheses. Systematic Reviews, 5(1), 184. https://doi.org/10.1186/s13643-016-0360-z

    CrossRef  Google Scholar 

  • Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138

    CrossRef  Google Scholar 

  • Gonzalez, E., & Rutkowski, L. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 3, 125–156. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_03_Chapter_6.pdf

    Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2018). Multiple imputation of missing data for multilevel models: Simulations and recommendations. Organizational Research Methods, 21(1), 111–149. https://doi.org/10.1177/1094428117703686

    CrossRef  Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2019). Missing data in multilevel research. In The handbook of multilevel theory, measurement, and analysis (pp. 365–386). American Psychological Association.

    CrossRef  Google Scholar 

  • Heck, R. H., & Thomas, S. L. (2015). An introduction to multilevel modeling techniques: MLM and SEM approaches using Mplus (3rd ed.). Routledge.

    CrossRef  Google Scholar 

  • Henry, K. L., & Muthén, B. (2010). Multilevel latent class analysis: An application of adolescent smoking typologies with individual and contextual predictors. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 193–215. https://doi.org/10.1080/10705511003659342

    CrossRef  Google Scholar 

  • Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). Routledge.

    Google Scholar 

  • Hox, J. J., van Buuren, S., & Jolani, S. (2015). Incomplete multilevel data: Problems and solutions. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 39–62). Information Age Publishing Inc..

    Google Scholar 

  • Hsu, H.-Y., Lin, J.-H., Kwok, O.-M., Acosta, S., & Willson, V. (2017). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo Study. Educational and Psychological Measurement, 77(1), 5–31. https://doi.org/10.1177/0013164416642823

    CrossRef  Google Scholar 

  • Jak, S. (2014). Testing strong factorial invariance using three-level structural equation modeling. Frontiers in Psychology, 5(745). https://doi.org/10.3389/fpsyg.2014.00745

  • Jak, S. (2019). Cross-level invariance in multilevel factor models. Structural Equation Modeling: A Multidisciplinary Journal, 26(4), 607–622. https://doi.org/10.1080/10705511.2018.1534205

    CrossRef  Google Scholar 

  • Janis, R. A., Burlingame, G. M., & Olsen, J. A. (2016). Evaluating factor structures of measures in group research: Looking between and within. Group Dynamics: Theory, Research, and Practice, 20(3), 165–180. https://doi.org/10.1037/gdn0000043

    CrossRef  Google Scholar 

  • Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). Sage.

    CrossRef  Google Scholar 

  • Kaplan, D., & Su, D. (2016). On matrix sampling and imputation of context questionnaires with implications for the generation of plausible values in large-scale assessments. Journal of Educational and Behavioral Statistics, 41(1), 57–80. https://doi.org/10.3102/1076998615622221

    CrossRef  Google Scholar 

  • Kelcey, B., Cox, K., & Dong, N. (2019). Croon’s bias-corrected factor score path analysis for small- to moderate-sample multilevel structural equation models. Organizational Research Methods(0), 1094428119879758. https://doi.org/10.1177/1094428119879758

  • Kim, E. S., Dedrick, R. F., Cao, C., & Ferron, J. M. (2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881–898. https://doi.org/10.1080/00273171.2016.1228042

    CrossRef  Google Scholar 

  • Kim, J.-S., Anderson, C. J., & Keller, B. (2014). Multilevel analysis of assessment data. In L. Rutkowski, M. V. Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 390–425). CRC Press.

    Google Scholar 

  • Klieme, E. (2013). The role of large-scale assessments in research on educational effectiveness and school development. In M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto (Eds.), The role of international large-scale assessments: Perspectives from technology, economy, and educational research (pp. 115–147). Springer.

    CrossRef  Google Scholar 

  • Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). Guilford Press.

    Google Scholar 

  • Kuger, S., & Klieme, E. (2016). Dimensions of context assessment. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning: An international perspective (pp. 3–37). Springer.

    CrossRef  Google Scholar 

  • Lachowicz, M. J., Preacher, K. J., & Kelley, K. (2018). A novel measure of effect size for mediation analysis. Psychological Methods, 23(2), 244–261. https://doi.org/10.1037/met0000165

    CrossRef  Google Scholar 

  • Lachowicz, M. J., Sterba, S. K., & Preacher, K. J. (2014). Investigating multilevel mediation with fully or partially nested data. Group Processes & Intergroup Relations, 18(3), 274–289. https://doi.org/10.1177/1368430214550343

    CrossRef  Google Scholar 

  • Lai, M. H. C., & Kwok, O.-M. (2015). Examining the rule of thumb of not using multilevel modeling: The “Design effect smaller than two” rule. The Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229

    CrossRef  Google Scholar 

  • LaRoche, S., Joncas, M., & Foy, P. (2016). Sample design in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 3.1–3.38). Boston College, TIMSS & PIRLS International Study Center.

    Google Scholar 

  • Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics – Theory and Methods, 46(22), 11341–11357. https://doi.org/10.1080/03610926.2016.1267764

    CrossRef  Google Scholar 

  • Laukaityte, I., & Wiberg, M. (2018). Importance of sampling weights in multilevel modeling of international large-scale assessment data. Communications in Statistics – Theory and Methods, 47(20), 4991–5012. https://doi.org/10.1080/03610926.2017.1383429

    CrossRef  Google Scholar 

  • Little, T. D. (2013). Longitudinal structural equation modeling. The Guilford Press.

    Google Scholar 

  • Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21–39. https://doi.org/10.1037/1082-989X.10.1.21

    CrossRef  Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376

    CrossRef  Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. https://doi.org/10.1037/a0012869

    CrossRef  Google Scholar 

  • Lüdtke, O., Robitzsch, A., & Grund, S. (2017). Multiple imputation of missing data in multilevel designs: A comparison of different strategies. Psychological Methods, 22(1), 141–165. https://doi.org/10.1037/met0000096

    CrossRef  Google Scholar 

  • Mäkikangas, A., Tolvanen, A., Aunola, K., Feldt, T., Mauno, S., & Kinnunen, U. (2018). Multilevel latent profile analysis with covariates: Identifying job characteristics profiles in hierarchical data as an example. Organizational Research Methods, 21(4), 931–954. https://doi.org/10.1177/1094428118760690

    CrossRef  Google Scholar 

  • Marsh, H. W., Dowson, M., Pietsch, J., & Walker, R. (2004). Why multicollinearity matters: A reexamination of relations between self-efficacy, self-concept, and achievement. Journal of Educational Psychology, 96(3), 518–522. https://doi.org/10.1037/0022-0663.96.3.518

    CrossRef  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A. J. S., Abduljabbar, A. S., & Köller, O. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488

    CrossRef  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665

    CrossRef  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling: A Multidisciplinary Journal, 16(2), 191–225. https://doi.org/10.1080/10705510902751010

    CrossRef  Google Scholar 

  • Masyn, K. E. (2013). Latent class analysis and finite mixture modeling. In The Oxford handbook of quantitative methods: Statistical analysis (Vol. 2, pp. 551–611). Oxford University Press.

    Google Scholar 

  • Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951–966. https://doi.org/10.1037/a0028380

    CrossRef  Google Scholar 

  • McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52(2), 200–215. https://doi.org/10.1080/00273171.2016.1262236

    CrossRef  Google Scholar 

  • Mislevy, R. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196. https://doi.org/10.1007/BF02294457

    CrossRef  Google Scholar 

  • Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5

    CrossRef  Google Scholar 

  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

    CrossRef  Google Scholar 

  • Morin, A. J. S., & Marsh, H. W. (2015). Disentangling shape from level effects in person-centered analyses: An illustration based on university teachers’ multidimensional profiles of effectiveness. Structural Equation Modeling, 22(1), 39–59. https://doi.org/10.1080/10705511.2014.919825

    CrossRef  Google Scholar 

  • Morin, A. J. S., Marsh, H. W., Nagengast, B., & Scalas, L. F. (2014). Doubly latent multilevel analyses of classroom climate: An illustration. The Journal of Experimental Education, 82(2), 143–167. https://doi.org/10.1080/00220973.2013.769412

    CrossRef  Google Scholar 

  • Muthén, B. O., & Asparouhov, T. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In Handbook for advanced multilevel analysis (pp. 15–40). Routledge/Taylor & Francis Group.

    Google Scholar 

  • Muthén, B. O., & Asparouhov, T. (2017). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47(4), 637–664. https://doi.org/10.1177/0049124117701488

    CrossRef  Google Scholar 

  • Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. V. Marsden (Ed.), Sociological methodology (pp. 267–316). Blackwell.

    Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide (8th ed.). Muthén & Muthén.

    Google Scholar 

  • Nagengast, B., & Marsh, H. W. (2011). The negative effect of school-average ability on science self-concept in the UK, the UK countries and the world: The Big-Fish-Little-Pond-Effect for PISA 2006. Educational Psychology, 31(5), 629–656. https://doi.org/10.1080/01443410.2011.586416

    CrossRef  Google Scholar 

  • Nagengast, B., & Marsh, H. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104(4), 1033–1053. https://doi.org/10.1037/a0027697

    CrossRef  Google Scholar 

  • Nilsen, T., Bloemeke, S., Yang Hansen, K., & Gustafsson, J.-E. (2016). Are school characteristics related to equity? The answer may depend on a country’s developmental level. IEA Policy Briefs, 10. Retrieved from https://www.iea.nl/publications/series-journals/policy-brief/april-2016-are-school-characteristics-related-equity

  • Nylund-Gibson, K., & Choi, A. Y. (2018). Ten frequently asked questions about latent class analysis. Translational Issues in Psychological Science, 4(4), 440–461. https://doi.org/10.1037/tps0000176

    CrossRef  Google Scholar 

  • O’Connell, A. A., Yeomans-Maldonado, G., & McCoach, D. B. (2015). Residual diagnostics and model assessment in a multilevel framework: Recommendations toward best practice. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 97–135). Information Age Publishing, Inc..

    Google Scholar 

  • OECD. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing.

    Google Scholar 

  • OECD. (2019a). PISA 2018 results (Vol. I). OECD Publishing.

    CrossRef  Google Scholar 

  • OECD. (2019b). TALIS 2018 results (Vol. I). OECD Publishing.

    CrossRef  Google Scholar 

  • Preacher, K. J. (2015). Advances in mediation analysis: A survey and synthesis of new developments. Annual Review of Psychology, 66(1), 825–852. https://doi.org/10.1146/annurev-psych-010814-015258

    CrossRef  Google Scholar 

  • Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2016). Multilevel structural equation models for assessing moderation within and across levels of analysis. Psychological Methods, 21(2), 189–205. https://doi.org/10.1037/met0000052

    CrossRef  Google Scholar 

  • Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141

    CrossRef  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications.

    Google Scholar 

  • Rohatgi, A., & Scherer, R. (2020). Identifying profiles of students’ school climate perceptions using PISA 2015 data. Large-scale Assessments in Education, 8(1), 4. https://doi.org/10.1186/s40536-020-00083-0

    CrossRef  Google Scholar 

  • Rust, K. (2014). Sampling, weighting, and variance estimation in international large-scale assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117–154). CRC Taylor & Francis.

    Google Scholar 

  • Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170

    CrossRef  Google Scholar 

  • Rutkowski, L., & Zhou, Y. (2014). Using structural equation models to analyze ILSA data. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 425–450). CRC Press.

    Google Scholar 

  • Ryu, E. (2014a). Factorial invariance in multilevel confirmatory factor analysis. British Journal of Mathematical and Statistical Psychology, 67(1), 172–194. https://doi.org/10.1111/bmsp.12014

    CrossRef  Google Scholar 

  • Ryu, E. (2014b). Model fit evaluation in multilevel structural equation models. Frontiers in Psychology, 5(81). https://doi.org/10.3389/fpsyg.2014.00081

  • Ryu, E. (2015). Multiple group analysis in multilevel structural equation model across level 1 groups. Multivariate Behavioral Research, 50(3), 300–315. https://doi.org/10.1080/00273171.2014.1003769

    CrossRef  Google Scholar 

  • Ryu, E., & Mehta, P. (2017). Multilevel factorial invariance in n-Level Structural Equation Modeling (nSEM). Structural Equation Modeling: A Multidisciplinary Journal, 24(6), 936–959. https://doi.org/10.1080/10705511.2017.1324311

    CrossRef  Google Scholar 

  • Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16(4), 583–601. https://doi.org/10.1080/10705510903203466

    CrossRef  Google Scholar 

  • Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference Chi-square test statistic. Psychometrika, 75(2), 243–248. https://doi.org/10.1007/s11336-009-9135-y

    CrossRef  Google Scholar 

  • Scherer, R., & Gustafsson, J.-E. (2015). Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: An application of multilevel bifactor structural equation modeling. Frontiers in Psychology, 6.

    Google Scholar 

  • Scherer, R., Nilsen, T., & Jansen, M. (2016). Evaluating individual students’ perceptions of instructional quality: An investigation of their factor structure, measurement invariance, and relations to educational outcomes. Frontiers in Psychology, 7(110). https://doi.org/10.3389/fpsyg.2016.00110

  • Scherer, R., Tondeur, J., Siddiq, F., & Baran, E. (2018). The importance of attitudes toward technology for pre-service teachers’ technological, pedagogical, and content knowledge: Comparing structural equation modeling approaches. Computers in Human Behavior, 80, 67–80. https://doi.org/10.1016/j.chb.2017.11.003

    CrossRef  Google Scholar 

  • Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499. https://doi.org/10.3102/0034654307310317

    CrossRef  Google Scholar 

  • Silva, B. C., Bosancianu, C. M., & Littvay, L. (2019). Multilevel structural equation modeling. Sage.

    Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An Introduction to basic and advanced multilevel modeling (2nd ed.). Sage.

    Google Scholar 

  • Stapleton, L. M. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 475–502. https://doi.org/10.1207/S15328007SEM0904_2

    CrossRef  Google Scholar 

  • Stapleton, L. M. (2013). Multilevel structural equation modeling with complex sample data. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 521–562). Information Age Publishing, Inc.

    Google Scholar 

  • Stapleton, L. M. (2014). Incorporating sampling weights into single- and multilevel analyses. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 363–388). CRC Taylor & Francis.

    Google Scholar 

  • Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200

    CrossRef  Google Scholar 

  • Van den Noortgate, W., Opdenakker, M.-C., & Onghena, P. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16(3), 281–303. https://doi.org/10.1080/09243450500114850

    CrossRef  Google Scholar 

  • von Davier, M., Gonzalez, E., & Mislevy, R. J. (2009). What are plausible values and why are they useful? IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 9–36. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf

    Google Scholar 

  • Wang, W., Liao, M., & Stapleton, L. (2019). Incidental second-level dependence in educational survey data with a nested data structure. Educational Psychology Review, 31(3), 571–596. https://doi.org/10.1007/s10648-019-09480-6

    CrossRef  Google Scholar 

  • Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2), 114–128. https://doi.org/10.1016/j.stueduc.2005.05.005

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Educational Measurement at the University of Oslo (CEMO), Faculty of Educational Sciences, University of Oslo, Oslo, Norway

    Ronny Scherer

Authors
  1. Ronny Scherer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ronny Scherer .

Editor information

Editors and Affiliations

  1. University of Oslo, Blindern, Oslo, Norway

    Dr. Trude Nilsen

  2. IEA Hamburg, Hamburg, Hamburg, Germany

    Dr. Agnes Stancel-Piątak

  3. University of Gothenburg, Gothenburg, Sweden

    Prof. Dr. Jan-Eric Gustafsson

Section Editor information

  1. University of Wisconsin Madison, Madison, United States

    David Kaplan

Appendices

Appendices

Appendix A: Database Search Terms

The following strategy informed the search for relevant publications in the databases ERIC and PsycINFO through the search service Ovid (with the resultant numbers of entries in brackets; 21 November 2019):

1 (PISA or TIMSS or PIRLS or PIAAC or ICILS or ICCS).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (5690)
2 (hierarchical model or hierarchical linear model or linear mixed-effects model or random coefficient model or random effects model or multilevel regression or generalized linear mixed model or GLMM or multilevel model* or multilevel analysis).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (16437)
3 (Multilevel path analysis or multilevel path model or multilevel mediation or multilevel factor analysis or multilevel CFA or multilevel confirmatory factor analysis or MCFA or multilevel SEM or MSEM or multilevel structural equation model* or multilevel latent covariate model or multilevel IRT or multilevel item response theory).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (1301)
4 2 or 3 (17472)
5 1 and 4 (227)
6 remove duplicates from 5 (226)

Appendix B: PRISMA Statement

Fig. 11
figure 11

Flowchart indicating the search and screening processes of the systematic review (PRISMA statement; adopted from Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009)

Full size image

Appendix C: Description of the Variables Used in the Illustrative Examples

Variable Original variable label (PISA 2015) Variable label Scale
Identification variables
Student Identifier CNTSTUID Country-specific Student ID Numeric (nominal)
School Identifier CNTSCHID Country-specific School ID Numeric (nominal)
Country Identifier CNTRYID Country Identifier 208 = Denmark
246 = Finland
352 = Iceland
578 = Norway
752 = Sweden
Student-level variables (Cognitive tests and student questionnaire)
Science achievement PV1SCIE-PV10SCIE Plausible values of scientific literacy Continuous
Socioeconomic status ESCS Index of economic, social, and cultural status (WLE) Continuous
Gender ST004D01T Student gender (recoded) 0 = Male
1 = Female
Home possessions HOMEPOS Home possessions (WLE) Continuous
Parents’ occupation HISEI Index highest parental occupational status  
Immigration status IMMIG Index Immigration status (recoded) 0 = Native
1 = First- or second-generation immigration
Grade repetition REPEAT Grade repetition 0 = No repetition
1 = Grade repetition
Parents’ education PARED Index highest parental education in years of schooling Continuous (in years)
Adaptive instruction (How often do these things happen in your lessons for this <school science> course?) ADINST Adaption of instruction (WLE) Continuous
ST107Q01 The teacher adapts the lesson to my class needs and knowledge. 1 = Never or almost never
2 = Some lessons
3 = Many lessons
4 = Every lesson or almost every lesson
ST107Q02 The teacher provides individual help when a student has difficulties
ST107Q03 The teacher changes the structure of the lesson on a topic
Disciplinary climate DISCLISCI Disciplinary climate in science classes (WLE) Continuous
Perceived feedback PERFEED Perceived feedback (WLE) Continuous
Teacher support (How often do these things happen in your <school science> lessons?) TEACHSUP Teacher support in a science class of students’ choice (WLE) Continuous
ST100Q01 The teacher shows interest every students’ learning. 1 = Every lesson
2 = Most lessons
3 = Some lessons
4 = Never or hardly ever
ST100Q02 The teacher gives extra help.
ST100Q03 The teacher helps students with their learning.
ST100Q04 The teacher continues teaching\students understand.
ST100Q05 Teacher gives an opportunity to express opinions.
Teachers’ unfair treatment of students UNFAIRTEACHER Teacher fairness (Sum) Continuous
Instrumental science motivation INSTSCIE Instrumental motivation (WLE) Continuous
Enjoyment of science (How much do you disagree or agree with the statements about yourself below?) JOYSCIE Enjoyment of science (WLE) Continuous
ST094Q01 I have fun when I am learning <broad science> 1 = Strongly disagree
2 = Disagree
ST094Q02 I like reading about <broad science> topics. 3 = Agree
4 = Strongly agree
ST094Q03 I am happy working on <broad science> topics.
ST094Q04 I enjoy acquiring new knowledge in <broad science>.
Test anxiety ANXTEST Test anxiety (WLE) Continuous
Achievement motivation (To what extent do you disagree or agree with the following statements about yourself?) MOTIVAT Student attitudes, preferences, and self-related beliefs: Achieving motivation (WLE) Continuous
ST119Q02 I want to be able to select from among the best opportunities available when I graduate. 1 = Strongly disagree
2 = Disagree
3 = Agree
4 = Strongly agree
ST119Q03 I want to be the best, whatever I do.
ST119Q04 I see myself as an ambitious person.
ST119Q05 I want to be one of the best students in my class.
School-level variables (Principal questionnaire)
School type PRIVATE Private school 0 = Public school
1 = Private independent or government-dependent school
School size SCHSIZE School size (sum) Continuous
Student-teacher ratio STRATIO Student-teacher ratio Continuous
Student behavior at school (In your school, to what extent is the learning of students hindered by the following phenomena?) STUBEHA Student behavior hindering learning (WLE) Continuous
SC061Q01 Student truancy 1 = Not at all
2 = Very little
3 = To some extent
4 = A lot
SC061Q02 Students skipping classes
SC061Q03 Students lacking respect for teachers
Teacher behavior at school TEACHBEHA Teacher behavior hindering learning (WLE) Continuous
Weights
Student weights W_FSTUWT Final trimmed nonresponse adjusted student weight Continuous
School weights W_SCHGRNRABWT Grade nonresponse adjusted school base weight Continuous
  1. Note. WLE Warm’s mean weighted likelihood estimates

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Verify currency and authenticity via CrossMark

Cite this entry

Scherer, R. (2022). Analyzing International Large-Scale Assessment Data with a Hierarchical Approach. In: Nilsen, T., Stancel-Piątak, A., Gustafsson, JE. (eds) International Handbook of Comparative Large-Scale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-030-38298-8_59-1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-38298-8_59-1

  • Received:

  • Accepted:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-38298-8

  • Online ISBN: 978-3-030-38298-8

  • eBook Packages: Springer Reference EducationReference Module Humanities and Social Sciences