Abstract
Interest in understanding how psychosocial environments shape youth outcomes has grown considerably. School environments are of particular interest to prevention scientists as many prevention interventions are school-based. Therefore, effective conceptualization and operationalization of the school environment is critical. This paper presents an illustration of an emerging analytic method called multilevel factor analysis (MLFA) that provides an alternative strategy to conceptualize, measure, and model environments. MLFA decomposes the total sample variance-covariance matrix for variables measured at the individual level into within-cluster (e.g., student level) and between-cluster (e.g., school level) matrices and simultaneously models potentially distinct latent factor structures at each level. Using data from 79,362 students from 126 schools in the National Longitudinal Study of Adolescent to Adult Health (formerly known as the National Longitudinal Study of Adolescent Health), we use MLFA to show how 20 items capturing student self-reported behaviors and emotions provide information about both students (within level) and their school environment (between level). We identified four latent factors at the within level: (1) school adjustment, (2) externalizing problems, (3) internalizing problems, and (4) self-esteem. Three factors were identified at the between level: (1) collective school adjustment, (2) psychosocial environment, and (3) collective self-esteem. The finding of different and substantively distinct latent factor structures at each level emphasizes the need for prevention theory and practice to separately consider and measure constructs at each level of analysis. The MLFA method can be applied to other nested relationships, such as youth in neighborhoods, and extended to a multilevel structural equation model to better understand associations between environments and individual outcomes and therefore how to best implement preventive interventions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson, C. (1982). The search for school climate: A review of the research. Review of Educational Research, 52(3), 368–420.
Aud, S., Hussar, W., Planty, M., Snyder, T., Bianco, K., Fox, M., & Drake, L. (2010). The condition of education 2010 (NCES 2010–028). Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107, 238–246.
Bradshaw, C. P., Koth, C. Q., Thornton, L. A., & Leaf, P. J. (2009). Altering school climate through school-wide positive behavioral interventions and supports: Findings from a group-randomized effectiveness trial. Prevention Science, 10, 100–115.
Brand, S., Felner, R. D., Shim, M., Seitsinger, A., & Dumas, T. (2003). Middle school improvement and reform: Development and validation of a school-level assessment of climate, cultural pluralism, and school safety. Journal of Educational Psychology, 95, 570–588.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.
Cohen, J., & Geier, V. K. (2010). School climate research summary. New York, NY: Center for Social and Emotional Education.
Cohen, J., McCabe, L., Michelli, N. M., & Pickeral, T. (2009). School climate: Research, policy, teacher education, and practice. Teachers College Record, 111(1), 180–213.
Dedrick, R. F., & Greenbaum, P. E. (2011). Multilevel confirmatory factor analysis of a scale measuring interagency collaboration of children's mental health agencies. Journal of Emotional and Behavioral Disorders, 19, 27–40.
Diez Roux, A. V. (2002). A glossary for multilevel analysis. Journal of Epidemiology and Community Health, 56, 588–594.
Dunn, E. C., Masyn, K. E., Yudron, M., Jones, S. M., & Subramanian, S. V. (2014). Translating multilevel theory into multilevel research: Challenge and opportunities for understanding the social determinants of psychiatric disorders. Social Psychiatry and Psychiatric Epidemiology, 49, 859–872.
Dyer, N. G., Hanges, P. J., & Hall, R. J. (2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. The Leadership Quarterly, 16, 149–167.
Eccles, J. S., & Roeser, R. W. (2011). School and community influences on human development. In M. H. Bornstein & M. E. Lamb (Eds.), Developmental science: An advanced textbook (6th ed., pp. 571–643). New York, NY: Psychology Press.
Flora, D. B., & Curran, P. J. (2004). An empirical evaluation of alternative methods of estimation for confirmatory factor analysis with ordinal data. Psychological Methods, 9(4), 466–491.
Greenberg, M. T., Weissberg, R. P., O'Brien, M. U., Zins, J. E., Fredericks, L., Resnik, H., & Elias, M. J. (2003). Enhancing school-based prevention and youth development through coordinated social, emotional, and academic learning. American Psychologist, 58(6–7), 466–474.
Harris, K. M. (2013). The add health study: design and accomplishments: Carolina Population Center, University of North Carolina at Chapel Hill.
Haynes, N. M., Emmons, C. L., & Ben-Avie, M. (2001). The school development program student, staff, and parent school climate surveys. New Haven, CT: Yale Child Study Center.
Hox, J. J. (2010). Multilevel analysis: Techniques and applications (2nd ed.). New York, NY: Routledge.
James, L. R., Choi, C. C., Ko, C. H. E., McNeil, P. K., Minton, M. K., Wright, M. A., & Kim, K. (2008). Organizational and psychological climate: A review of theory and research. The European Work and Organizational Psychologist, 17(1), 5–32.
Jones, S. M., Brown, J. L., & Aber, J. L. (2011). Two-year impacts of a universal school-based social-emotional and literacy intervention: An experiment in translational developmental research. Child Development, 82(2), 533–554.
Kline, P. (1994). An easy guide to factor analysis. London, England: Routledge.
Kline, R. B. (2010). Principles and practice of structural equation modeling (3rd ed.). New York, NY: Guilford Press.
Marsh, H. W., Ludtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthen, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44, 764–802.
Muthen, B. O. (1991). Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 28(4), 338–354.
Muthen, B. O. (1994). Multilevel covariance structure analysis. Sociological Methods & Research, 22, 376–398.
Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus user's guide (6th edition ed.). Los Angeles, CA: Muthén & Muthén.
National School Climate Center. The Comprehensive School Climate Inventory, from http://www.schoolclimate.org. Accessed 9 June 2013.
Reise, S. P., Ventura, J., Neuchterlein, K. H., & Kim, K. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84(2), 126–136.
Rones, M., & Hoagwood, K. (2000). School-based mental health services: A research review. Clinical Child and Family Psychology Review, 3(4), 223–241.
Shinn, M. (1990). Mixing and matching: Levels of conceptualization, measurement, and statistical analysis in community research. In P. Tolan, C. Keys, F. Chertok, & L. A. Jason (Eds.), Researching community psychology: Issues of theory and methods (pp. 111–126). Washington, DC: American Psychological Association.
Steiger, J. H. (1990). Structural model evaluation and modification: An interval estimation approach. Multivariate Behavioral Research, 25, 173–180.
Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Boston, MA: Allyn and Bacon.
Toland, M. D., & De Ayala, R. J. (2005). A multilevel factor analysis of students’ evaluations of teaching. Educational and Psychological Measurement, 65(2), 272–296.
Van Horn, M. L. (2003). Assessing the unit of measurement for school climate through psychometric and outcome analyses of the school climate survey. Educational and Psychological Measurement, 63(6), 1002–1019.
Zullig, K. J., Koopman, T. M., Patton, J. M., & Ubbes, V. A. (2010). School climate: Historical review, instrument development, and school assessment. Journal of Psychoeducational Assessment, 28(2), 139–152.
Acknowledgments
This research uses data from Add Health, a program project directed by Kathleen Mullan Harris and designed by J. Richard Udry, Peter S. Bearman, and Kathleen Mullan Harris at the University of North Carolina at Chapel Hill, and funded by grant P01-HD31921 from the Eunice Kennedy Shriver National Institute of Child Health and Human Development, with cooperative funding from 23 other federal agencies and foundations. Special acknowledgment is due Ronald R. Rindfuss and Barbara Entwisle for assistance in the original design. Information on how to obtain the Add Health data files is available on the Add Health website (http://www.cpc.unc.edu/addhealth). No direct support was received from grant P01-HD31921 for this analysis. Visit Add Health online: https://www.cpc.unc.edu/projects/addhealth. Research reported in this publication was supported by the National Institute of Mental Health under Award Number F31 MH088074 to Erin C. Dunn, ScD, MPH. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Conflict of Interest
The authors declare that they have no conflict of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Dunn, E.C., Masyn, K.E., Jones, S.M. et al. Measuring Psychosocial Environments Using Individual Responses: an Application of Multilevel Factor Analysis to Examining Students in Schools. Prev Sci 16, 718–733 (2015). https://doi.org/10.1007/s11121-014-0523-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-014-0523-x