Abstract
As colleges and universities grapple with uncertainty around current and future enrollment as well as increasingly vocal questions about the value of postsecondary education, it is critically important that institutions ascertain and invest in the elements of campus learning and engagement that add value to the undergraduate experience. This study examines the relationship between cumulative participation in high-impact practices (HIPs) and the perceived importance of postsecondary experience in preparation for adult life using the Educational Longitudinal Study of 2002 (ELS) dataset. Employing a methodology proposed and tested by Leite et al. (Struct Equ Model Multidiscip J 26(3):448–469, 2019. https://doi.org/10.1080/10705511.2018.1522591), this analysis incorporated the ability to account for self-selection into HIPs using propensity score (PS) analysis with a multiple-group structural equation model (SEM) design to examine differences between students who participated in two or more HIPs and those who did not (n = 3105). Results offered evidence of benefit to participation in two or more HIP experiences with positive and statistically significant differences in the perceived importance of postsecondary education in preparation for adult life across the analytic sample with doubly robust estimation techniques. Interaction effects for female students, students from low SES backgrounds, and students who are members of minoritized racial/ethnic populations were also identified. The findings offered evidence of post-graduation impact of cumulative participation in HIPs that can inform program development and student decision-making as well as the future use of analytic techniques such as PS analysis, doubly robust estimation, and sensitivity analysis to enhance measurement precision.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
With the exception of the demographic variables, all model variables had some missing data. The categorical nature of many covariates present in the model precluded the use of full information maximum likelihood (FIML) as a missing data treatment. With the ability to generate multiple sets of data with missing values replaced by imputed values, multiple imputation (MI) was also considered. However, while PS scores can be combined, there is little consensus on how to combine fit indices which are central to the research questions in this study. Consequently, full case analysis was employed.
Latent variables are measured indirectly using responses from a set of questions or test items (Schumacker & Lomax, 2016), differing from observed variables which can be measured directly. The use of both observed and latent variables in SEM models permits researchers to account for substantive complexity in relationships between variables (Schumacker & Lomax, 2016; Streiner, 2006; Wang & Wang, 2012).
In addition to the recommendation that all undergraduate students participate in two HIP experiences (Kuh, 2008), the grouping variable was operationally defined in this way due to the nature of the item stem in the original ELS survey which was phrased as “[Have you participated/Did you participate] in any of the following as a part of your [undergraduate/college] education?” The response scale was dichotomous with students able to indicate yes/no with no further data about the number or duration of individual experiences. Since the constituent indicators were categorical in nature with no additional qualifying information, a more conservative approach to the operationalization of the grouping variable was taken.
The F3FITSCWT sampling weight was applied to account for the complexities of the longitudinal design including unequal patterns of selection and nonparticipation from selected sample members (Ingels et al., 2014). Additionally, the F3F1T001-F3F1T200 balanced repeated replication (BRR) weight adjustments (Ingels et al., 2014) were applied to facilitate accurate estimations of standard errors (McNeish et al., 2017; Stapleton, 2008).
References
ACT. (2007). The role of nonacademic factors in college readiness and success. ACT. https://www.act.org/content/dam/act/unsecured/documents/nonacademic_factors.pdf
Adelman, C. (2006). The toolbox revisited: Paths to degree completion from high school through college. U.S. Department of Education. https://www.researchgate.net/publication/275463367_The_Toolbox_Revisited_Paths_to_Degree_Completion_From_High_School_Through_College
Allensworth, E. M., & Clark, K. (2020). High school GPAs and ACT scores as predictors of college completion: Examining assumptions about consistency across high schools. Educational Researcher, 49(3), 198–211. https://doi.org/10.3102/0013189X20902110
Asparouhov, T., Muthén, B., & Muthén, B. O. (2006). Robust chi square difference testing with mean and variance adjusted test statistics. https://www.statmodel.com/download/webnotes/webnote10.pdf
Association of American Colleges and Universities. (2007). College learning for the new global century: A report from the National Leadership Council for Liberal Education and America’s Promise. Association of American Colleges and Universities. https://www.aacu.org/sites/default/files/files/LEAP/GlobalCentury_final.pdf
Astin, A. W. (1984). Student involvement: A developmental theory for higher education. Journal of College Student Personnel, 25(4), 297–308. https://www.researchgate.net/publication/220017441_Student_Involvement_A_Development_Theory_for_Higher_Education
Astin, A. W. (1993). What matters in college? Jossey-Bass.
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. https://doi.org/10.1080/00273171.2011.568786
Bang, H., & Robins, J. M. (2005). Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4), 962–973. https://doi.org/10.1111/j.1541-0420.2005.00377.x
Beauducel, A., & Herzberg, P. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13(2), 186–203. https://doi.org/10.1207/s15328007sem1302_2
Bowman, N. A., & Holmes, J. M. (2018). Getting off to a good start? First-year undergraduate research experiences and student outcomes. Higher Education, 76(1), 17–33. https://doi.org/10.1007/s10734-017-0191-4
Brownell, J. E., & Swaner, L. E. (2010). Five high-impact practices: Research on learning outcomes, completion, and quality. Association of American Colleges and Universities.
Cheema, J. R. (2014). Some general guidelines for choosing missing data handling methods in educational research. Journal of Modern Applied Statistical Methods, 13(2), 53–75. https://doi.org/10.22237/jmasm/1414814520
Cox, B. E., McIntosh, K., Reason, R. D., & Terenzini, P. T. (2014). Working with missing data in higher education research: A primer and real-world example. The Review of Higher Education, 37(3), 377–402. https://doi.org/10.1353/rhe.2014.0026
Culver, K. C., & Bowman, N. A. (2020). Is what glitters really gold? A quasi-experimental study of first-year seminars and college student success. Research in Higher Education, 61(2), 167–196. https://doi.org/10.1007/s11162-019-09558-8
Enders, C. K., & Mansolf, M. (2018). Assessing the fit of structural equation models with multiply imputed data. Psychological Methods, 23(1), 76. https://doi.org/10.1037/met0000102
Espinosa, L. L., Turk, J. M., Taylor, M., & Chessman, H. M. (2019). Race and ethnicity in higher education: A status report. American Council on Education. https://1xfsu31b52d33idlp13twtos-wpengine.netdna-ssl.com/wp-content/uploads/2019/02/Race-and-Ethnicity-in-Higher-Education.pdf
Finley, A., & McNair, T. (2013). Assessing underserved students’ engagement in high-impact practices. Association of American Colleges and Universities. https://www.aacu.org/sites/default/files/files/assessinghips/AssessingHIPS_TGGrantReport.pdf
Grawe, N. D. (2018). Demographics and the demand for higher education. Johns Hopkins University Press. https://doi.org/10.1353/book.57044
Harring, J. R., McNeish, D. M., & Hancock, G. R. (2017). Using phantom variables in structural equation modeling to assess model sensitivity to external misspecification. Psychological Methods, 22(4), 616. https://doi.org/10.1037/met0000103
Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35(1), 1–97. https://doi.org/10.1111/j.0081-1750.2006.00164.x
Ingels, S. J., Pratt, D. J., Alexander, C. P., Jewell, D. M., Lauff, E., Mattox, T. L., & Wilson, D. (2014). Education longitudinal study of 2002 (ELS: 2002) third follow-up data file documentation. NCES 2014–364. National Center for Education Statistics. https://nces.ed.gov/pubs2014/2014364.pdf
Johnson, S. R., & Stage, F. K. (2018). Academic engagement and student success: Do high- impact practices mean higher graduation rates? The Journal of Higher Education. https://doi.org/10.1080/00221546.2018.1441107
Klebs, S., Nguyen, S., Fishman, R., & Hiler, T. (2021, June 29). One year later: COVID-19’s impact on current and future college students. Third Way. https://www.thirdway.org/memo/one-year-later-covid-19s-impact-on-current-and-future-college-students
Kilgo, C. A., Sheets, J. K. E., & Pascarella, E. T. (2015). The link between high-impact practices and student learning: Some longitudinal evidence. Higher Education, 69(4), 509–525. https://doi.org/10.1007/s10734-014-9788-z
Kuh, G. D. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Association of American Colleges and Universities.
Kuh, G. D., Kinzie, J., Cruce, T., Shoup, R., & Gonyea, R. M. (2007). Connecting the dots: Multi-faceted analyses of the relationships between student engagement results from the NSSE, and the institutional practices and conditions that foster student success. Center for Postsecondary Research. http://hdl.handle.net/2022/23684
Kuh, G. D., O’Donnell, K., & Reed, S. (2013). Ensuring quality and taking high-impact practices to scale. Association of American Colleges and Universities.
Leite, W. L., Stapleton, L. M., & Bettini, E. F. (2019). Propensity score analysis of complex survey data with structural equation modeling: A tutorial with MPlus. Structural Equation Modeling: A Multidisciplinary Journal, 26(3), 448–469. https://doi.org/10.1080/10705511.2018.1522591
Lombardi, A., Seburn, M., & Conley, D. (2011). Treatment of nonresponse items on scale validation: What “don't know” responses indicate about college readiness. https://files.eric.ed.gov/fulltext/ED518170.pdf
McDaniel, A., & Van Jura, M. (2020). High-impact practices: Evaluating their effect on college completion. Journal of College Student Retention: Research, Theory and Practice. https://doi.org/10.1177/1521025120947357
McNair, T. B., Bensimon, E. M., & Malcom-Piqueux, L. (2020). From equity talk to equity walk: Expanding practitioner knowledge for racial justice in higher education. Wiley. https://doi.org/10.1002/9781119428725
McNeish, D., Stapleton, L. M., & Silverman, R. D. (2017). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods, 22(1), 114. https://doi.org/10.1037/met0000078
Miller, A. L., Rocconi, L. M., & Dumford, A. D. (2018). Focus on the finish line: Does high-impact practice participation influence career plans and early job attainment? Higher Education, 75(3), 489–506. https://doi.org/10.1007/s10734-017-0151-z
Millsap, R. E., & Olivera-Aguilar, M. (2012). Investigating measurement invariance using confirmatory factor analysis. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 380–392). Guilford Press.
Muthén, L. K. & Muthén, B. O. (2017). MPlus user’s guide (8th ed.). https://www.statmodel.com/ugexcerpts.shtml
Myers, C. B., Myers, S. M., & Peters, M. (2018). The longitudinal connections between undergraduate high impact curriculum practices and civic engagement in adulthood. Research in Higher Education, 60, 83–110. https://doi.org/10.1007/s11162-018-9504-4
National Center for Education Statistics. (n.d.). Education Longitudinal Study of 2002 (ELS: 2002) online codebook. National Center for Education Statistics. https://nces.ed.gov/OnlineCodebook/Session/Codebook/a111f850-113c-4da9-a394-fd3f212385e7
National Student Clearinghouse Research Center. (2021). Current term enrollment estimates: Fall 2021. National Student Clearinghouse Research Center. https://nscresearchcenter.org/wp-content/uploads/CTEE_Report_Fall_2021.pdf
National Study of Student Engagement. (2007). Experiences that matter: Enhancing student learning and success. National Study of Student Engagement. https://scholarworks.iu.edu/dspace/handle/2022/23412
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: A third decade of research (Vol. 2). Jossey-Bass.
Patton, L. D., Harper, S. R., & Harris, J. (2015). Using critical race theory to (re)interpret widely studied topics related to students in US higher education. In A. M. Martínez Alemán, E. M. Bensimon & B. Pusser (Eds.), Critical approaches to the study of higher education (pp. 193–219). Johns Hopkins University Press.
Provencher, A., & Kassel, R. (2019). High-impact practices and sophomore retention: Examining the effects of selection bias. Journal of College Student Retention: Research, Theory and Practice, 21(2), 221–241. https://doi.org/10.1177/1521025117697728
Reason, R. D. (2003). Student variables that predict retention: Recent research and new developments. Journal of Student Affairs Research and Practice, 40(4), 704–723.
Reason, R. D. (2009). An examination of persistence research through the lens of a comprehensive conceptual framework. Journal of College Student Development, 50(6), 659–682. https://doi.org/10.2202/1949-6605.5022
Rosenbaum, P. R. (2002). Observational studies. Springer.
Schafer, J. L., & Kang, J. (2008). Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods, 13(4), 279. https://doi.org/10.1037/a0014268
Schumacker, R. E., & Lomax, R. G. (2016). A beginner's guide to structural equation modeling (4th ed.). Routledge. https://doi.org/10.4324/9781410610904
Seifert, T. A., Gillig, B., Hanson, J. M., Pascarella, E. T., & Blaich, C. F. (2014). The conditional nature of high impact/good practices on student learning outcomes. The Journal of Higher Education, 85(4), 531–564. https://doi.org/10.1080/00221546.2014.11777339
Stapleton, L. M. (2008). Variance estimation using replication methods in structural equation modeling with complex sample data. Structural Equation Modeling: A Multidisciplinary Journal, 15(2), 183–210. https://doi.org/10.1080/10705510801922316
Steiner, P. M., Cook, T. D., Li, W., & Clark, M. H. (2015). Bias reduction in quasi-experiments with little selection theory but many covariates. Journal of Research on Educational Effectiveness, 8(4), 552–576. https://doi.org/10.1080/19345747.2014.978058
Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250–267. https://doi.org/10.1037/a0018719
Streiner, D. L. (2006). Building a better model: An introduction to structural equation modelling. The Canadian Journal of Psychiatry, 51(5), 317–324. https://doi.org/10.1177/070674370605100507
Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 25(1), 1. https://doi.org/10.1214/09-STS313
Stuart, E. A., & Rubin, D. B. (2008). Best practices in quasi-experimental designs: Matching methods for causal inference. In J. Osbourne (Ed.), Best practices in quantitative methods (pp. 155–176). SAGE Publications. https://doi.org/10.4135/9781412995627
Terenzini, P. T., Springer, L., Yaeger, P. M., Pascarella, E. T., & Nora, A. (1996). First-generation college students: Characteristics, experiences, and cognitive development. Research in Higher Education, 37(1), 1–22. https://www.jstor.org/stable/40196208
Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46(1), 90–118. https://doi.org/10.1080/00273171.2011.540475
Tinto, V. (1993). Leaving college: Rethinking the causes and cures of student attrition (2nd ed.). University of Chicago Press.
Valentine, J., Price, D., & Yang, H. (2021). Lumina issue paper: High-impact practices and gains in student learning: Evidence from Georgia, Montana, and Wisconsin. Lumina Foundation. http://nashonline.org/wp-content/uploads/2021/03/high-impact-practices-and-gains-in-student-learning.pdf
Wang, J., & Wang, X. (2012). Structural equation modeling applications using Mplus. Wiley.
Wolniak, G. C., & Engberg, M. E. (2019). Do “high-impact” college experiences affect early career outcomes? The Review of Higher Education, 42(3), 825–858. https://doi.org/10.1353/rhe.2019.0021
Zilvinskis, J. (2019). Measuring quality in high-impact practices. Higher Education, 78(4), 687–709. https://doi.org/10.1007/s10734-019-00365-9
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
See Table 6.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Dickert, J.L., Li, J. Illuminating the Post-Graduation Impact of Undergraduate Participation in High-Impact Practices Using Propensity Score Analysis with Structural Equation Modeling. Res High Educ (2024). https://doi.org/10.1007/s11162-023-09767-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s11162-023-09767-2