Skip to main content
SpringerLink
Log in

Analyzing International Large-Scale Assessment Data with a Hierarchical Approach

  • Reference work entry
  • First Online:
International Handbook of Comparative Large-Scale Studies in Education

Part of the book series: Springer International Handbooks of Education ((SIHE))

Abstract

International large-scale assessments in education (ILSAs) follow complex sampling designs and ultimately create hierarchical data structures with students nested in classrooms, classrooms in schools, schools in regions, etc. To describe adequately key issues in education, such as socioeconomic gaps in academic achievement or the relations among school characteristics and student achievement using ILSA data, researchers need to consider the hierarchical data structure in statistical models. Multilevel modeling is one approach to account for such hierarchies and consider variables at different levels of analysis. This chapter provides an overview of the prominent multilevel modeling approaches to analyzing ILSA data, illustrates and discusses their strengths and weaknesses, and highlights the key methodological decisions researchers have to take in this context. The first part reviews the current practices of multilevel modeling in secondary analyses of ILSA data. This rapid systematic review is followed by a second part in which we present, illustrate, and discuss multilevel modeling approaches, including multilevel regression, multilevel structural equation models, and multilevel mixture models. Next to model estimation and fit evaluation, we review key issues associated with the multilevel modeling of ILSA data and focus on handling plausible values, multigroup and incidental multilevel data structures, and weighting. Our chapter provides worked examples showcasing the potential of multilevel modeling for ILSA data analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 379.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aguinis, H., Gottfredson, R. K., & Culpepper, S. A. (2013). Best-practice recommendations for estimating cross-level interaction effects using multilevel modeling. Journal of Management, 39(6), 1490–1528. https://doi.org/10.1177/0149206313478188

    Article  Google Scholar 

  • Asparouhov, T. (2005). Sampling weights in latent variable modeling. Structural Equation Modeling: A Multidisciplinary Journal, 12(3), 411–434. https://doi.org/10.1207/s15328007sem1203_4

    Article  Google Scholar 

  • Asparouhov, T. (2006). General multi-level modeling with sampling weights. Communications in Statistics – Theory and Methods, 35(3), 439–460. https://doi.org/10.1080/03610920500476598

    Article  Google Scholar 

  • Bellens, K., Van Damme, J., Van Den Noortgate, W., Wendt, H., & Nilsen, T. (2019). Instructional quality: Catalyst or pitfall in educational systems’ aim for high achievement and equity? An answer based on multilevel SEM analyses of TIMSS 2015 data in Flanders (Belgium), Germany, and Norway. Large-scale Assessment in Education, 7(1). https://doi.org/10.1186/s40536-019-0069-2

  • Berkowitz, R., Moore, H., Astor, R. A., & Benbenishty, R. (2017). A research synthesis of the associations between socioeconomic background, inequality, school climate, and academic achievement. Review of Educational Research, 87(2), 425–469. https://doi.org/10.3102/0034654316669821

    Article  Google Scholar 

  • Brown, T. A. (2015). Confirmatory factor analysis for applied research (2nd ed.). The Guilford Press.

    Google Scholar 

  • Cai, T. (2012). Investigation of ways to handle sampling weights for multilevel model analyses. Sociological Methodology, 43(1), 178–219. https://doi.org/10.1177/0081175012460221

    Article  Google Scholar 

  • Dedrick, R. F., Ferron, J. M., Hess, M. R., Hogarty, K. Y., Kromrey, J. D., Lang, T. R., … Lee, R. S. (2009). Multilevel modeling: A review of methodological issues and applications. Review of Educational Research, 79(1), 69–102. https://doi.org/10.3102/0034654308325581

    Article  Google Scholar 

  • Diez Roux, A. V. (2002). A glossary for multilevel analysis. Journal of Epidemiology and Community Health, 56(8), 588–594. https://doi.org/10.1136/jech.56.8.588

    Article  Google Scholar 

  • Else-Quest, N. M., Hyde, J. S., & Linn, M. C. (2010). Cross-national patterns of gender differences in mathematics: A meta-analysis. Psychological Bulletin, 136(1), 103–127. https://doi.org/10.1037/a0018053

    Article  Google Scholar 

  • Enders, C. K. (2010). Applied missing data analysis. Guilford Press.

    Google Scholar 

  • Enders, C. K., & Mansolf, M. (2018). Assessing the fit of structural equation models with multiply imputed data. Psychological Methods, 23(1), 76–93. https://doi.org/10.1037/met0000102

    Article  Google Scholar 

  • Enders, C. K., Mistler, S. A., & Keller, B. T. (2016). Multilevel multiple imputation: A review and evaluation of joint modeling and chained equations imputation. Psychological Methods, 21(2), 222–240. https://doi.org/10.1037/met0000063

    Article  Google Scholar 

  • Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross-sectional multilevel models: A new look at an old issue. Psychological Methods, 12(2), 121–138. https://doi.org/10.1037/1082-989X.12.2.121

    Article  Google Scholar 

  • Garritty, C., Stevens, A., Gartlehner, G., King, V., Kamel, C., & On behalf of the Cochrane Rapid Reviews Methods Group. (2016). Cochrane Rapid Reviews Methods Group to play a leading role in guiding the production of informed high-quality, timely research evidence syntheses. Systematic Reviews, 5(1), 184. https://doi.org/10.1186/s13643-016-0360-z

    Article  Google Scholar 

  • Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138

    Article  Google Scholar 

  • Gonzalez, E., & Rutkowski, L. (2010). Principles of multiple matrix booklet designs and parameter recovery in large-scale assessments. IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 3, 125–156. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_03_Chapter_6.pdf

    Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2018). Multiple imputation of missing data for multilevel models: Simulations and recommendations. Organizational Research Methods, 21(1), 111–149. https://doi.org/10.1177/1094428117703686

    Article  Google Scholar 

  • Grund, S., Lüdtke, O., & Robitzsch, A. (2019). Missing data in multilevel research. In The handbook of multilevel theory, measurement, and analysis (pp. 365–386). American Psychological Association.

    Chapter  Google Scholar 

  • Heck, R. H., & Thomas, S. L. (2015). An introduction to multilevel modeling techniques: MLM and SEM approaches using Mplus (3rd ed.). Routledge.

    Book  Google Scholar 

  • Henry, K. L., & Muthén, B. (2010). Multilevel latent class analysis: An application of adolescent smoking typologies with individual and contextual predictors. Structural Equation Modeling: A Multidisciplinary Journal, 17(2), 193–215. https://doi.org/10.1080/10705511003659342

    Article  Google Scholar 

  • Hox, J. J., Moerbeek, M., & van de Schoot, R. (2018). Multilevel analysis: Techniques and applications (3rd ed.). Routledge.

    Google Scholar 

  • Hox, J. J., van Buuren, S., & Jolani, S. (2015). Incomplete multilevel data: Problems and solutions. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 39–62). Information Age Publishing Inc..

    Google Scholar 

  • Hsu, H.-Y., Lin, J.-H., Kwok, O.-M., Acosta, S., & Willson, V. (2017). The impact of intraclass correlation on the effectiveness of level-specific fit indices in multilevel structural equation modeling: A Monte Carlo Study. Educational and Psychological Measurement, 77(1), 5–31. https://doi.org/10.1177/0013164416642823

    Article  Google Scholar 

  • Jak, S. (2014). Testing strong factorial invariance using three-level structural equation modeling. Frontiers in Psychology, 5(745). https://doi.org/10.3389/fpsyg.2014.00745

  • Jak, S. (2019). Cross-level invariance in multilevel factor models. Structural Equation Modeling: A Multidisciplinary Journal, 26(4), 607–622. https://doi.org/10.1080/10705511.2018.1534205

    Article  Google Scholar 

  • Janis, R. A., Burlingame, G. M., & Olsen, J. A. (2016). Evaluating factor structures of measures in group research: Looking between and within. Group Dynamics: Theory, Research, and Practice, 20(3), 165–180. https://doi.org/10.1037/gdn0000043

    Article  Google Scholar 

  • Kaplan, D. (2009). Structural equation modeling: Foundations and extensions (2nd ed.). Sage.

    Book  Google Scholar 

  • Kaplan, D., & Su, D. (2016). On matrix sampling and imputation of context questionnaires with implications for the generation of plausible values in large-scale assessments. Journal of Educational and Behavioral Statistics, 41(1), 57–80. https://doi.org/10.3102/1076998615622221

    Article  Google Scholar 

  • Kelcey, B., Cox, K., & Dong, N. (2019). Croon’s bias-corrected factor score path analysis for small- to moderate-sample multilevel structural equation models. Organizational Research Methods(0), 1094428119879758. https://doi.org/10.1177/1094428119879758

  • Kim, E. S., Dedrick, R. F., Cao, C., & Ferron, J. M. (2016). Multilevel factor analysis: Reporting guidelines and a review of reporting practices. Multivariate Behavioral Research, 51(6), 881–898. https://doi.org/10.1080/00273171.2016.1228042

    Article  Google Scholar 

  • Kim, J.-S., Anderson, C. J., & Keller, B. (2014). Multilevel analysis of assessment data. In L. Rutkowski, M. V. Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 390–425). CRC Press.

    Google Scholar 

  • Klieme, E. (2013). The role of large-scale assessments in research on educational effectiveness and school development. In M. von Davier, E. Gonzalez, I. Kirsch, & K. Yamamoto (Eds.), The role of international large-scale assessments: Perspectives from technology, economy, and educational research (pp. 115–147). Springer.

    Chapter  Google Scholar 

  • Kline, R. B. (2015). Principles and practice of structural equation modeling (4th ed.). Guilford Press.

    Google Scholar 

  • Kuger, S., & Klieme, E. (2016). Dimensions of context assessment. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning: An international perspective (pp. 3–37). Springer.

    Chapter  Google Scholar 

  • Lachowicz, M. J., Preacher, K. J., & Kelley, K. (2018). A novel measure of effect size for mediation analysis. Psychological Methods, 23(2), 244–261. https://doi.org/10.1037/met0000165

    Article  Google Scholar 

  • Lachowicz, M. J., Sterba, S. K., & Preacher, K. J. (2014). Investigating multilevel mediation with fully or partially nested data. Group Processes & Intergroup Relations, 18(3), 274–289. https://doi.org/10.1177/1368430214550343

    Article  Google Scholar 

  • Lai, M. H. C., & Kwok, O.-M. (2015). Examining the rule of thumb of not using multilevel modeling: The “Design effect smaller than two” rule. The Journal of Experimental Education, 83(3), 423–438. https://doi.org/10.1080/00220973.2014.907229

    Article  Google Scholar 

  • LaRoche, S., Joncas, M., & Foy, P. (2016). Sample design in TIMSS 2015. In M. O. Martin, I. V. S. Mullis, & M. Hooper (Eds.), Methods and procedures in TIMSS 2015 (pp. 3.1–3.38). Boston College, TIMSS & PIRLS International Study Center.

    Google Scholar 

  • Laukaityte, I., & Wiberg, M. (2017). Using plausible values in secondary analysis in large-scale assessments. Communications in Statistics – Theory and Methods, 46(22), 11341–11357. https://doi.org/10.1080/03610926.2016.1267764

    Article  Google Scholar 

  • Laukaityte, I., & Wiberg, M. (2018). Importance of sampling weights in multilevel modeling of international large-scale assessment data. Communications in Statistics – Theory and Methods, 47(20), 4991–5012. https://doi.org/10.1080/03610926.2017.1383429

    Article  Google Scholar 

  • Little, T. D. (2013). Longitudinal structural equation modeling. The Guilford Press.

    Google Scholar 

  • Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21–39. https://doi.org/10.1037/1082-989X.10.1.21

    Article  Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 × 2 taxonomy of multilevel latent contextual models: Accuracy–bias trade-offs in full and partial error correction models. Psychological Methods, 16(4), 444–467. https://doi.org/10.1037/a0024376

    Article  Google Scholar 

  • Lüdtke, O., Marsh, H. W., Robitzsch, A., Trautwein, U., Asparouhov, T., & Muthén, B. (2008). The multilevel latent covariate model: A new, more reliable approach to group-level effects in contextual studies. Psychological Methods, 13(3), 203–229. https://doi.org/10.1037/a0012869

    Article  Google Scholar 

  • Lüdtke, O., Robitzsch, A., & Grund, S. (2017). Multiple imputation of missing data in multilevel designs: A comparison of different strategies. Psychological Methods, 22(1), 141–165. https://doi.org/10.1037/met0000096

    Article  Google Scholar 

  • Mäkikangas, A., Tolvanen, A., Aunola, K., Feldt, T., Mauno, S., & Kinnunen, U. (2018). Multilevel latent profile analysis with covariates: Identifying job characteristics profiles in hierarchical data as an example. Organizational Research Methods, 21(4), 931–954. https://doi.org/10.1177/1094428118760690

    Article  Google Scholar 

  • Marsh, H. W., Dowson, M., Pietsch, J., & Walker, R. (2004). Why multicollinearity matters: A reexamination of relations between self-efficacy, self-concept, and achievement. Journal of Educational Psychology, 96(3), 518–522. https://doi.org/10.1037/0022-0663.96.3.518

    Article  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Nagengast, B., Trautwein, U., Morin, A. J. S., Abduljabbar, A. S., & Köller, O. (2012). Classroom climate and contextual effects: Conceptual and methodological issues in the evaluation of group-level effects. Educational Psychologist, 47(2), 106–124. https://doi.org/10.1080/00461520.2012.670488

    Article  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665

    Article  Google Scholar 

  • Marsh, H. W., Lüdtke, O., Trautwein, U., & Morin, A. J. S. (2009). Classical latent profile analysis of academic self-concept dimensions: Synergy of person- and variable-centered approaches to theoretical models of self-concept. Structural Equation Modeling: A Multidisciplinary Journal, 16(2), 191–225. https://doi.org/10.1080/10705510902751010

    Article  Google Scholar 

  • Masyn, K. E. (2013). Latent class analysis and finite mixture modeling. In The Oxford handbook of quantitative methods: Statistical analysis (Vol. 2, pp. 551–611). Oxford University Press.

    Google Scholar 

  • Mathieu, J. E., Aguinis, H., Culpepper, S. A., & Chen, G. (2012). Understanding and estimating the power to detect cross-level interaction effects in multilevel modeling. Journal of Applied Psychology, 97(5), 951–966. https://doi.org/10.1037/a0028380

    Article  Google Scholar 

  • McNeish, D., & Wentzel, K. R. (2017). Accommodating small sample sizes in three-level models when the third level is incidental. Multivariate Behavioral Research, 52(2), 200–215. https://doi.org/10.1080/00273171.2016.1262236

    Article  Google Scholar 

  • Mislevy, R. J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196. https://doi.org/10.1007/BF02294457

    Article  Google Scholar 

  • Moerbeek, M. (2004). The consequence of ignoring a level of nesting in multilevel analysis. Multivariate Behavioral Research, 39(1), 129–149. https://doi.org/10.1207/s15327906mbr3901_5

    Article  Google Scholar 

  • Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), e1000097. https://doi.org/10.1371/journal.pmed.1000097

    Article  Google Scholar 

  • Morin, A. J. S., & Marsh, H. W. (2015). Disentangling shape from level effects in person-centered analyses: An illustration based on university teachers’ multidimensional profiles of effectiveness. Structural Equation Modeling, 22(1), 39–59. https://doi.org/10.1080/10705511.2014.919825

    Article  Google Scholar 

  • Morin, A. J. S., Marsh, H. W., Nagengast, B., & Scalas, L. F. (2014). Doubly latent multilevel analyses of classroom climate: An illustration. The Journal of Experimental Education, 82(2), 143–167. https://doi.org/10.1080/00220973.2013.769412

    Article  Google Scholar 

  • Muthén, B. O., & Asparouhov, T. (2011). Beyond multilevel regression modeling: Multilevel analysis in a general latent variable framework. In Handbook for advanced multilevel analysis (pp. 15–40). Routledge/Taylor & Francis Group.

    Google Scholar 

  • Muthén, B. O., & Asparouhov, T. (2017). Recent methods for the study of measurement invariance with many groups: Alignment and random effects. Sociological Methods & Research, 47(4), 637–664. https://doi.org/10.1177/0049124117701488

    Article  Google Scholar 

  • Muthén, B. O., & Satorra, A. (1995). Complex sample data in structural equation modeling. In P. V. Marsden (Ed.), Sociological methodology (pp. 267–316). Blackwell.

    Google Scholar 

  • Muthén, L. K., & Muthén, B. O. (1998–2017). Mplus user’s guide (8th ed.). Muthén & Muthén.

    Google Scholar 

  • Nagengast, B., & Marsh, H. W. (2011). The negative effect of school-average ability on science self-concept in the UK, the UK countries and the world: The Big-Fish-Little-Pond-Effect for PISA 2006. Educational Psychology, 31(5), 629–656. https://doi.org/10.1080/01443410.2011.586416

    Article  Google Scholar 

  • Nagengast, B., & Marsh, H. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104(4), 1033–1053. https://doi.org/10.1037/a0027697

    Article  Google Scholar 

  • Nilsen, T., Bloemeke, S., Yang Hansen, K., & Gustafsson, J.-E. (2016). Are school characteristics related to equity? The answer may depend on a country’s developmental level. IEA Policy Briefs, 10. Retrieved from https://www.iea.nl/publications/series-journals/policy-brief/april-2016-are-school-characteristics-related-equity

  • Nylund-Gibson, K., & Choi, A. Y. (2018). Ten frequently asked questions about latent class analysis. Translational Issues in Psychological Science, 4(4), 440–461. https://doi.org/10.1037/tps0000176

    Article  Google Scholar 

  • O’Connell, A. A., Yeomans-Maldonado, G., & McCoach, D. B. (2015). Residual diagnostics and model assessment in a multilevel framework: Recommendations toward best practice. In J. R. Harring, L. M. Stapleton, & S. N. Beretvas (Eds.), Advances in multilevel modeling for educational research: Addressing practical issues found in real-world applications (pp. 97–135). Information Age Publishing, Inc.

    Google Scholar 

  • OECD. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing.

    Google Scholar 

  • OECD. (2019a). PISA 2018 results (Vol. I). OECD Publishing.

    Book  Google Scholar 

  • OECD. (2019b). TALIS 2018 results (Vol. I). OECD Publishing.

    Book  Google Scholar 

  • Preacher, K. J. (2015). Advances in mediation analysis: A survey and synthesis of new developments. Annual Review of Psychology, 66(1), 825–852. https://doi.org/10.1146/annurev-psych-010814-015258

    Article  Google Scholar 

  • Preacher, K. J., Zhang, Z., & Zyphur, M. J. (2016). Multilevel structural equation models for assessing moderation within and across levels of analysis. Psychological Methods, 21(2), 189–205. https://doi.org/10.1037/met0000052

    Article  Google Scholar 

  • Preacher, K. J., Zyphur, M. J., & Zhang, Z. (2010). A general multilevel SEM framework for assessing multilevel mediation. Psychological Methods, 15(3), 209–233. https://doi.org/10.1037/a0020141

    Article  Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Sage Publications.

    Google Scholar 

  • Rohatgi, A., & Scherer, R. (2020). Identifying profiles of students’ school climate perceptions using PISA 2015 data. Large-scale Assessments in Education, 8(1), 4. https://doi.org/10.1186/s40536-020-00083-0

    Article  Google Scholar 

  • Rust, K. (2014). Sampling, weighting, and variance estimation in international large-scale assessments. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 117–154). CRC Taylor & Francis.

    Google Scholar 

  • Rutkowski, L., Gonzalez, E., Joncas, M., & von Davier, M. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170

    Article  Google Scholar 

  • Rutkowski, L., & Zhou, Y. (2014). Using structural equation models to analyze ILSA data. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 425–450). CRC Press.

    Google Scholar 

  • Ryu, E. (2014a). Factorial invariance in multilevel confirmatory factor analysis. British Journal of Mathematical and Statistical Psychology, 67(1), 172–194. https://doi.org/10.1111/bmsp.12014

    Article  Google Scholar 

  • Ryu, E. (2014b). Model fit evaluation in multilevel structural equation models. Frontiers in Psychology, 5(81). https://doi.org/10.3389/fpsyg.2014.00081

  • Ryu, E. (2015). Multiple group analysis in multilevel structural equation model across level 1 groups. Multivariate Behavioral Research, 50(3), 300–315. https://doi.org/10.1080/00273171.2014.1003769

    Article  Google Scholar 

  • Ryu, E., & Mehta, P. (2017). Multilevel factorial invariance in n-Level Structural Equation Modeling (nSEM). Structural Equation Modeling: A Multidisciplinary Journal, 24(6), 936–959. https://doi.org/10.1080/10705511.2017.1324311

    Article  Google Scholar 

  • Ryu, E., & West, S. G. (2009). Level-specific evaluation of model fit in multilevel structural equation modeling. Structural Equation Modeling, 16(4), 583–601. https://doi.org/10.1080/10705510903203466

    Article  Google Scholar 

  • Satorra, A., & Bentler, P. M. (2010). Ensuring positiveness of the scaled difference Chi-square test statistic. Psychometrika, 75(2), 243–248. https://doi.org/10.1007/s11336-009-9135-y

    Article  Google Scholar 

  • Scherer, R., & Gustafsson, J.-E. (2015). Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: An application of multilevel bifactor structural equation modeling. Frontiers in Psychology, 6.

    Google Scholar 

  • Scherer, R., Nilsen, T., & Jansen, M. (2016). Evaluating individual students’ perceptions of instructional quality: An investigation of their factor structure, measurement invariance, and relations to educational outcomes. Frontiers in Psychology, 7(110). https://doi.org/10.3389/fpsyg.2016.00110

  • Scherer, R., Tondeur, J., Siddiq, F., & Baran, E. (2018). The importance of attitudes toward technology for pre-service teachers’ technological, pedagogical, and content knowledge: Comparing structural equation modeling approaches. Computers in Human Behavior, 80, 67–80. https://doi.org/10.1016/j.chb.2017.11.003

    Article  Google Scholar 

  • Seidel, T., & Shavelson, R. J. (2007). Teaching effectiveness research in the past decade: The role of theory and research design in disentangling meta-analysis results. Review of Educational Research, 77(4), 454–499. https://doi.org/10.3102/0034654307310317

    Article  Google Scholar 

  • Silva, B. C., Bosancianu, C. M., & Littvay, L. (2019). Multilevel structural equation modeling. Sage.

    Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An Introduction to basic and advanced multilevel modeling (2nd ed.). Sage.

    Google Scholar 

  • Stapleton, L. M. (2002). The incorporation of sample weights into multilevel structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 9(4), 475–502. https://doi.org/10.1207/S15328007SEM0904_2

    Article  Google Scholar 

  • Stapleton, L. M. (2013). Multilevel structural equation modeling with complex sample data. In G. R. Hancock & R. O. Mueller (Eds.), Structural equation modeling: A second course (2nd ed., pp. 521–562). Information Age Publishing, Inc.

    Google Scholar 

  • Stapleton, L. M. (2014). Incorporating sampling weights into single- and multilevel analyses. In L. Rutkowski, M. von Davier, & D. Rutkowski (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 363–388). CRC Taylor & Francis.

    Google Scholar 

  • Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200

    Article  Google Scholar 

  • Van den Noortgate, W., Opdenakker, M.-C., & Onghena, P. (2005). The effects of ignoring a level in multilevel analysis. School Effectiveness and School Improvement, 16(3), 281–303. https://doi.org/10.1080/09243450500114850

    Article  Google Scholar 

  • von Davier, M., Gonzalez, E., & Mislevy, R. J. (2009). What are plausible values and why are they useful? IERI Monograph Series: Issues and Methodologies in Large-Scale Assessments, 2, 9–36. Retrieved from http://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf

    Google Scholar 

  • Wang, W., Liao, M., & Stapleton, L. (2019). Incidental second-level dependence in educational survey data with a nested data structure. Educational Psychology Review, 31(3), 571–596. https://doi.org/10.1007/s10648-019-09480-6

    Article  Google Scholar 

  • Wu, M. (2005). The role of plausible values in large-scale surveys. Studies in Educational Evaluation, 31(2), 114–128. https://doi.org/10.1016/j.stueduc.2005.05.005

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Educational Measurement at the University of Oslo (CEMO), Faculty of Educational Sciences, University of Oslo, Oslo, Norway

    Ronny Scherer

Authors
  1. Ronny Scherer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ronny Scherer .

Editor information

Editors and Affiliations

  1. Department of Teacher Education and School Research, Faculty of Educational Sciences, University of Oslo, Oslo, Norway

    Trude Nilsen

  2. IEA Hamburg, Hamburg, Hamburg, Germany

    Agnes Stancel-Piątak

  3. University of Gothenburg, Gothenburg, Sweden

    Jan-Eric Gustafsson

Section Editor information

  1. University of Wisconsin Madison, Madison, United States

    David Kaplan

Appendices

Appendices

Appendix A: Database Search Terms

The following strategy informed the search for relevant publications in the databases ERIC and PsycINFO through the search service Ovid (with the resultant numbers of entries in brackets; 21 November 2019):

1

(PISA or TIMSS or PIRLS or PIAAC or ICILS or ICCS).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (5690)

2

(hierarchical model or hierarchical linear model or linear mixed-effects model or random coefficient model or random effects model or multilevel regression or generalized linear mixed model or GLMM or multilevel model* or multilevel analysis).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (16437)

3

(Multilevel path analysis or multilevel path model or multilevel mediation or multilevel factor analysis or multilevel CFA or multilevel confirmatory factor analysis or MCFA or multilevel SEM or MSEM or multilevel structural equation model* or multilevel latent covariate model or multilevel IRT or multilevel item response theory).mp. [mp=ab, ti, hw, id, tc, ot, tm, mh] (1301)

4

2 or 3 (17472)

5

1 and 4 (227)

6

remove duplicates from 5 (226)

Appendix B: PRISMA Statement

Fig. 11
figure 11

Flowchart indicating the search and screening processes of the systematic review (PRISMA statement; adopted from Moher, Liberati, Tetzlaff, Altman, & The PRISMA Group, 2009)

Full size image

Appendix C: Description of the Variables Used in the Illustrative Examples

Variable

Original variable label (PISA 2015)

Variable label

Scale

Identification variables

Student Identifier

CNTSTUID

Country-specific Student ID

Numeric (nominal)

School Identifier

CNTSCHID

Country-specific School ID

Numeric (nominal)

Country Identifier

CNTRYID

Country Identifier

208 = Denmark

246 = Finland

352 = Iceland

578 = Norway

752 = Sweden

Student-level variables (Cognitive tests and student questionnaire)

Science achievement

PV1SCIE-PV10SCIE

Plausible values of scientific literacy

Continuous

Socioeconomic status

ESCS

Index of economic, social, and cultural status (WLE)

Continuous

Gender

ST004D01T

Student gender (recoded)

0 = Male

1 = Female

Home possessions

HOMEPOS

Home possessions (WLE)

Continuous

Parents’ occupation

HISEI

Index highest parental occupational status

 

Immigration status

IMMIG

Index Immigration status (recoded)

0 = Native

1 = First- or second-generation immigration

Grade repetition

REPEAT

Grade repetition

0 = No repetition

1 = Grade repetition

Parents’ education

PARED

Index highest parental education in years of schooling

Continuous (in years)

Adaptive instruction (How often do these things happen in your lessons for this <school science> course?)

ADINST

Adaption of instruction (WLE)

Continuous

ST107Q01

The teacher adapts the lesson to my class needs and knowledge.

1 = Never or almost never

2 = Some lessons

3 = Many lessons

 

ST107Q02

The teacher provides individual help when a student has difficulties

4 = Every lesson or almost every lesson

ST107Q03

The teacher changes the structure of the lesson on a topic

Disciplinary climate

DISCLISCI

Disciplinary climate in science classes (WLE)

Continuous

Perceived feedback

PERFEED

Perceived feedback (WLE)

Continuous

Teacher support (How often do these things happen in your <school science> lessons?)

TEACHSUP

Teacher support in a science class of students’ choice (WLE)

Continuous

ST100Q01

The teacher shows interest every students’ learning.

1 = Every lesson

2 = Most lessons

3 = Some lessons

4 = Never or hardly ever

ST100Q02

The teacher gives extra help.

ST100Q03

The teacher helps students with their learning.

ST100Q04

The teacher continues teaching\students understand.

ST100Q05

Teacher gives an opportunity to express opinions.

Teachers’ unfair treatment of students

UNFAIRTEACHER

Teacher fairness (Sum)

Continuous

Instrumental science motivation

INSTSCIE

Instrumental motivation (WLE)

Continuous

Enjoyment of science (How much do you disagree or agree with the statements about yourself below?)

JOYSCIE

Enjoyment of science (WLE)

Continuous

ST094Q01

I have fun when I am learning <broad science>

1 = Strongly disagree

2 = Disagree

ST094Q02

I like reading about <broad science> topics.

3 = Agree

4 = Strongly agree

 

ST094Q03

I am happy working on <broad science> topics.

 

ST094Q04

I enjoy acquiring new knowledge in <broad science>.

Test anxiety

ANXTEST

Test anxiety (WLE)

Continuous

Achievement motivation (To what extent do you disagree or agree with the following statements about yourself?)

MOTIVAT

Student attitudes, preferences, and self-related beliefs: Achieving motivation (WLE)

Continuous

ST119Q02

I want to be able to select from among the best opportunities available when I graduate.

1 = Strongly disagree

2 = Disagree

3 = Agree

4 = Strongly agree

ST119Q03

I want to be the best, whatever I do.

ST119Q04

I see myself as an ambitious person.

ST119Q05

I want to be one of the best students in my class.

School-level variables (Principal questionnaire)

School type

PRIVATE

Private school

0 = Public school

1 = Private independent or government-dependent school

School size

SCHSIZE

School size (sum)

Continuous

Student-teacher ratio

STRATIO

Student-teacher ratio

Continuous

Student behavior at school (In your school, to what extent is the learning of students hindered by the following phenomena?)

STUBEHA

Student behavior hindering learning (WLE)

Continuous

SC061Q01

Student truancy

1 = Not at all

2 = Very little

3 = To some extent

4 = A lot

SC061Q02

Students skipping classes

SC061Q03

Students lacking respect for teachers

Teacher behavior at school

TEACHBEHA

Teacher behavior hindering learning (WLE)

Continuous

Weights

Student weights

W_FSTUWT

Final trimmed nonresponse adjusted student weight

Continuous

School weights

W_SCHGRNRABWT

Grade nonresponse adjusted school base weight

Continuous

  1. Note. WLE Warm’s mean weighted likelihood estimates

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this entry

Check for updates. Verify currency and authenticity via CrossMark

Cite this entry

Scherer, R. (2022). Analyzing International Large-Scale Assessment Data with a Hierarchical Approach. In: Nilsen, T., Stancel-Piątak, A., Gustafsson, JE. (eds) International Handbook of Comparative Large-Scale Studies in Education. Springer International Handbooks of Education. Springer, Cham. https://doi.org/10.1007/978-3-030-88178-8_59

Download citation

Publish with us

Policies and ethics

Search

Navigation