Skip to main content
SpringerLink
Log in

Is What Glitters Really Gold? A Quasi-Experimental Study of First-Year Seminars and College Student Success

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

First-year seminars are frequently designed to help students adjust to and succeed in college. Although considerable literature has explored this topic, many previous studies may have notable problems with self-selection, since students who choose to participate are likely more motivated academically than those who do not. Therefore, this study used quasi-experimental analyses within a large, longitudinal, multi-institutional dataset to explore the link between seminar participation and several student success outcomes. Overall, the use of propensity score analyses substantially alters the results, such that first-year seminars are positively associated with first-year college satisfaction, but they have no effect on fourth-year satisfaction, college grades, retention, or four-year graduation within the full sample. This lack of impact is largely consistent regardless of whether the seminar is designed to engage students in academic inquiry or to promote orientation and academic success. Additional analyses observed some differential effects of first-year seminars by race/ethnicity, ACT score, and sex; the most consistent finding is that first-year seminars appear to promote the college grades and college satisfaction of Black students. Implications for future research and practice are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • ACT, Inc. (2019). ACT national ranks. Retrieved from http://www.act.org/content/act/en/products-and-services/the-act/scores/national-ranks.html.

  • Akers, A. (2010). Determination of the optimal number of strata for bias reduction in propensity score matching. Dissertation Abstracts International, 71 (08), 58A. (UMI No. 3417726)

  • An, B. P., Loes, C. N., & Trolian, T. L. (2017). Binge drinking on academic performance: Considering mediating effects of academic involvement. Journal of College Student Development,58(4), 492–508.

    Google Scholar 

  • Arpino, B., & Mealli, F. (2011). The specification of the propensity score in multilevel observational studies. Computational Statistics & Data Analysis,55(4), 1770–1780.

    Google Scholar 

  • Astin, A. W. (1970). The methodology of research on college impact, part one. Sociology of Education,43(3), 223–254.

    Google Scholar 

  • Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research,46(3), 399–424.

    Google Scholar 

  • Barefoot, B. O. (1992). 1991 National Survey of Freshman Seminar Programming: National Resource Center for The Freshman Year Experience. Columbia: University of South Carolina.

    Google Scholar 

  • Barton, A., & Donahue, C. (2009). Multiple assessments of a first-year seminar pilot. Journal of General Education,58(4), 259–278.

    Google Scholar 

  • Bean, J., & Eaton, S. (2000). A psychological model of college student retention. In J. M. Braxton (Ed.), Reworking the student departure puzzle (pp. 48–61). Nashville: Vanderbilt University Press.

    Google Scholar 

  • Bean, J. P., & Metzner, B. S. (1985). A conceptual model of nontraditional undergraduate student retention. Review of Educational Research,55(4), 485–540.

    Google Scholar 

  • Bowman, N. A., & Culver, K. C. (2018). When do honors programs make the grade? Conditional effects on college satisfaction, achievement, retention, and graduation. Research in Higher Education,59, 249–272.

    Google Scholar 

  • Bowman, N. A., Park, J. J., & Denson, N. (2015). Student involvement in ethnic student organizations: Examining civic outcomes six years after graduation. Research in Higher Education,56, 127–145.

    Google Scholar 

  • Bowman, N. A., Denson, N., & Park, J. J. (2016). Racial/cultural awareness workshops and post-college civic engagement: A propensity score matching approach. American Educational Research Journal,53, 1556–1587.

    Google Scholar 

  • Brookhart, M. A., Schneeweiss, S., Rothman, K. J., Glynn, R. J., Avorn, J., & Stürmer, T. (2006). Variable selection for propensity score models. American Journal of Epidemiology, 163(12), 1149–1156.

    Google Scholar 

  • Brooks, J. M., & Ohsfeldt, R. L. (2013). Squeezing the balloon: Propensity scores and unmeasured covariate balance. Health Services Research,48(4), 1487–1507.

    Google Scholar 

  • Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. G. (1996). Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119(2), 197.

    Google Scholar 

  • Center for Research on Undergraduate Education at the University of Iowa. (2008). Quantitative Methods section for the Wabash National Study of Liberal Arts Education. Retrieved from https://education.uiowa.edu/sites/education.uiowa.edu/files/documents/centers/crue/research_methods_draft_march2008.pdf.

  • Center of Inquiry at Wabash College. (2019a). Wabash National Study instruments. Retrieved from https://centerofinquiry.org/wabash-national-study-instruments-investigate-liberal-arts-education/.

  • Center of Inquiry at Wabash College. (2019b). Wabash National Study participating institutions. Retrieved from https://centerofinquiry.org/wabash-national-study-participants/.

  • Clark, M. H., & Cundiff, N. L. (2011). Assessing the effectiveness of a college freshman seminar using propensity score adjustments. Research in Higher Education,52(6), 616–639.

    Google Scholar 

  • Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24(2), 295.

    Google Scholar 

  • Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah: Lawrence Erlbaum.

    Google Scholar 

  • Cox, D. R. (1958). Planning of experiments. New York: Wiley.

    Google Scholar 

  • DeAngelo, L. (2014). Programs and practices that retain students from the first to second year: Results from a national study. In R. D. Padgett (Ed.), Emerging research and practices on first-year students (pp. 53–75). San Francisco: Jossey-Bass.

    Google Scholar 

  • DuGoff, E. H., Schuler, M., & Stuart, E. A. (2014). Generalizing observational study results: applying propensity score methods to complex surveys. Health Services Research,49(1), 284–303.

    Google Scholar 

  • Forsythe, D. R., Lawrence, N. K., Burnette, J. L., & Baumeister, R. F. (2007). Attempting to improve the academic performance of struggling college students by bolstering their self-esteem: An intervention that backfired. Journal of Social and Clinical Psychology,26(4), 447–459.

    Google Scholar 

  • Friedman, D. B., & Marsh, E. G. (2009). What type of first-year seminar is most effective? A comparison of thematic seminars and college transition/success seminars. Journal of the First-Year Experience & Students in Transition,21(1), 29–42.

    Google Scholar 

  • Garrido, M. M., Kelley, A. S., Paris, J., Roza, K., Meier, D. E., Morrison, R. S., & Aldridge, M. D. (2014). Methods for constructing and assessing propensity scores. Health Services Research, 49(5), 1701–1720.

    Google Scholar 

  • Godes, O., Hulleman, C. S., & Harackiewicz, J. M. (2007, April). Boosting students’ interest in math with utility value: Two experimental tests. Paper presented at the meeting of the American Educational Research Association, Chicago, IL.

  • Groves, R. M., Fowler, F. J., Jr., Couper, M. P., Lepkowski, J. M., Singer, E., & Tourangeau, R. (2009). Survey methodology (2nd ed.). Hoboken: Wiley.

    Google Scholar 

  • Guo, S., & Fraser, M. (2015). Propensity score analyses: Statistical methods and applications (2nd ed.). Thousand Oaks: SAGE Publications.

    Google Scholar 

  • Hart Research Associates (2016). Recent trends in general education design, learning outcomes, and teaching approaches: Key findings from a survey among administrators at AAC&U member institutions. Retrieved from http://www.aacu.org/sites/default/files/files/LEAP/2015_Survey_Report2_GEtrends.pdf

  • Hendel, D. D. (2007). Efficacy of participating in a first-year seminar on student satisfaction and retention. Journal of College Student Retention,8(4), 413–423.

    Google Scholar 

  • Holmes, W. M. (2013). Using propensity scores in quasi-experimental designs. Thousand Oaks (CA): SAGE Publications Inc.

    Google Scholar 

  • Hong, G., & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on children’s cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis,27, 205–224.

    Google Scholar 

  • Hulleman, C. S., & Harackiewicz, J. M. (2009). Making education relevant: Increasing interest and performance in high school science classes. Science,326, 1410–1412.

    Google Scholar 

  • Hullsiek, K. H., & Louis, T. A. (2002). Propensity score modeling strategies for the causal analysis of observational data. Biostatistics,3(2), 179–193.

    Google Scholar 

  • Hunter, M. S., & Linder, C. W. (2005). First-year seminars. In M. L. Upcraft, J. N. Gardner, & B. O. Barefoot (Eds.), Challenging and supporting the first-year student: A handbook for improving the first year of college (pp. 275–291). San Francisco: Jossey-Bass.

    Google Scholar 

  • Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics,86(1), 4–29.

    Google Scholar 

  • Jaccard, J., & Turrisi, R. (2003). Interaction effects in multiple regression. Thousand Oaks: SAGE Publications.

    Google Scholar 

  • Jamelske, E. (2009). Measuring the impact of a university first-year experience program on student GPA and retention. Higher Education,57, 373–391.

    Google Scholar 

  • Joffe, M. M., Ten Have, T. R., Feldman, H. I., & Kimmel, S. E. (2004). Model selection, confounder control, and marginal structural models: Review and new applications. The American Statistician,58, 272–279.

    Google Scholar 

  • Johnson, D. R., Wasserman, T. H., Yildirim, N., & Yonai, B. A. (2014). Examining the effects of stress and campus climate on the persistence of students of color and White students: An application of Bean and Eaton’s psychological model of retention. Research in Higher Education,55, 75–100.

    Google Scholar 

  • Keup, J. R., & Young, D. G. (2018). Investigating the first-year seminar as a high-impact practice. In R. S. Feldman (Ed.), The first year of college: Research, theory, and practice on improving the student experience and increasing retention (pp. 93–125). New York: Cambridge University Press.

    Google Scholar 

  • Kinzie, J., Gonyea, R., Kuh, G. D., Umbach, P. D., Blaich, C., & Korkmaz, A. (2007, November). The relationship between gender and student engagement in college. Paper presented at the annual meeting of the Association for the Study of Higher Education, Louisville, KY.

  • Kuh, G. D. (2008). High-impact educational practices: What they are, who has access to them, and why they matter. Washington, DC: Association of American Colleges and Universities.

    Google Scholar 

  • Kuh, G. D., & O’Donnell, K. (2013). Ensuring quality and taking high-impact practices to scale. Washington, DC: Association of American Colleges & Universities.

    Google Scholar 

  • Lang, D. J. (2007). The impact of a first-year experience course on the academic performance, persistence, and graduation rates of first-semester college students at a public research university. Journal of the First-Year Experience & Students in Transition,19(1), 9–25.

    Google Scholar 

  • Li, F., Zaslavsky, A. M., & Landrum, M. B. (2013). Propensity score weighting with multilevel data. Statistics in Medicine,32(19), 3373–3387.

    Google Scholar 

  • Lunceford, J. K., & Davidian, M. (2004). Stratification and weighting via the propensity score in estimation of causal treatment effects: A comparative study. Statistics in Medicine,23, 2937–2960.

    Google Scholar 

  • Mayhew, M. J., Rockenbach, A. N., Bowman, N. A., Seifert, T. A., & Wolniak, G. C. (2016). How college affects students (Vol. 3): 21stcentury evidence that higher education works. San Francisco, CA: Jossey-Bass.

  • Miller, J. W., Janz, J. C., & Chen, C. (2007). The retention impact of a first-year seminar on students with varying pre-college academic performance. Journal of the First-Year Experience & Students in Transition,19(1), 47–62.

    Google Scholar 

  • National Resource Center for The First-Year Experience and Students in Transition. (2006). The 2006 National Survey on First-Year Seminar Programs. Columbia: Author, University of South Carolina.

    Google Scholar 

  • National Resource Center for The First-Year Experience and Students in Transition. (2013). The 2012-2013 National Survey of First-Year Seminar Programs. Columbia: Author, University of South Carolina.

    Google Scholar 

  • Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students (Vol. 2). San Francisco: Jossey-Bass.

    Google Scholar 

  • Permzadian, V., & Credé, M. (2016). Do first-year seminars improve college grades and retention: A quantitative review of their overall effectiveness and an examination of moderators of effectiveness. Review of Educational Research,86(1), 277–316.

    Google Scholar 

  • Pintrich, P. R., & Zusho, A. (2007). Student motivation and self-regulated learning in the college classroom. In R. P. Perry & J. C. Smart (Eds.), The scholarship of teaching and learning in higher education: An evidence-based perspective (pp. 731–810). New York: Springer.

    Google Scholar 

  • Polanin, J. R., Tanner-Smith, E. E., & Hennessy, E. A. (2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research,86(1), 207–236.

    Google Scholar 

  • Porter, S. R., & Swing, R. L. (2006). Understanding how first-year seminars affect persistence. Research in Higher Education,47(1), 89–109.

    Google Scholar 

  • Rabe-Hesketh, S., & Skrondal, A. (2006). Multilevel modelling of complex survey data. Journal of the Royal Statistical Society: Series A (Statistics in Society),169(4), 805–827.

    Google Scholar 

  • Radford, A. W., Berkner, L., Wheeless, S. C., & Shepherd, B. (2010). Persistence and attainment of 2003-04 Beginning Postsecondary Students: After 6 years (NCES 2011-151). Washington, DC: U.S. Department of Education.

    Google Scholar 

  • Ridgeway, G., Kovalchik, S. A., Griffin, B. A., & Kabeto, M. U. (2015). Propensity score analysis with survey weighted data. Journal of Causal Inference,3(2), 237–249.

    Google Scholar 

  • Robbins, S. B., Oh, I.-S., Le, H., & Button, C. (2009). Intervention effects on college performance and retention as mediated by motivational, emotional, and social control factors: Integrated meta-analytic path analyses. Journal of Applied Psychology,94(5), 1163–1184.

    Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1984). Reducing bias in observational studies using subclassification on the propensity score. Journal of the American Statistical Association, 79(387), 516–524.

    Google Scholar 

  • Rubin, D. B. (1980). Comment on “Randomization analysis of experimental data in the Fisher randomization test” by D. Basu. Journal of the American Statistical Association,75, 591–593.

    Google Scholar 

  • Rubin, D. B. (1986). Which ifs have causal answers. Journal of American Statistical Association,81, 961–962.

    Google Scholar 

  • Rubin, D. B., & Thomas, N. (1996). Matching using estimated propensity scores: relating theory to practice. Biometrics, 249-264.

  • Rutschow, E. Z., Cullinan, D., & Welbeck, R. (2012). Keeping students on course: An impact study of a student success course at Guilford Technical Community College. New York: MDRC.

    Google Scholar 

  • Ryan, M. P., & Glenn, P. A. (2004). What do first-year students need most: Learning strategies instruction or academic socialization? Journal of College Reading and Learning,34(2), 4–28.

    Google Scholar 

  • Santini, F. D. O., Ladeira, W. J., Sampaio, C. H., & da Silva Costa, G. (2017). Student satisfaction in higher education: A meta-analytic study. Journal of Marketing for Higher Education,27(1), 1–18.

    Google Scholar 

  • Schnell, C. A., & Doetkott, C. D. (2003). First year seminars produce long-term impact. Journal of College Student Retention,4(4), 377–391.

    Google Scholar 

  • Schnell, C. A., Louis, K. S., & Doetkott, C. (2003). The first-year seminar as a means of improving college graduation rates. Journal of the First-Year Experience and Students in Transition,15, 53–75.

    Google Scholar 

  • Scrivener, S., & Weiss, M. J. (2013). More graduates: Two-year results from an evaluation of accelerated study in associate programs (ASAP) for developmental education students. New York: MDRC.

    Google Scholar 

  • Scrivener, S., Weiss, M. J., Ratledge, A., Rudd, T., Sommo, C., & Fresques, H. (2015). Doubling graduation rates: Three-year effects of CUNY’s Accelerated Study in Associate Programs (ASAP) for developmental education students. New York: MDRC.

    Google Scholar 

  • Shadish, W. R., Clark, M. H., & Steiner, P. M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random to nonrandom assignment. Journal of the American Statistical Association,103, 1334–1343.

    Google Scholar 

  • Sherman, D. K., Cohen, G. L., Nelson, L. D., Nussbaum, A. D., Bunyan, D. P., & Garcia, J. P. (2009). Affirmed yet unaware: Exploring the role of awareness in the process of self-affirmation. Journal of Personality and Social Psychology,97, 745–764.

    Google Scholar 

  • Singell, L. D., & Waddell, G. R. (2010). Modeling retention at a large public university: Can at-risk students be identified early enough to treat? Research in Higher Education,51(6), 546–572.

    Google Scholar 

  • Sommo, C., Cullinan, D., Manno, M., Blake, S., & Alonzo, E. (2018). Doubling graduation rates in a new state: Two-year findings from the ASAP Ohio demonstration. New York: MDRC.

    Google Scholar 

  • Steiner, P. M., & Cook, D. (2013). Matching and propensity scores. The Oxford handbook of quantitative methods in psychology,1, 237.

    Google Scholar 

  • Steiner, P. M., Cook, T. D., Li, W., & Clark, M. H. (2015). Bias reduction in quasi-experiments with little selection theory but many covariates. Journal of Research on Educational Effectiveness,8, 552–576.

    Google Scholar 

  • Swaner, L. E., & Brownell, J. E. (2010). Outcomes of high impact practices for underserved students: A review of the literature. Washington, DC: Association of American Colleges & Universities.

    Google Scholar 

  • Swing, R. L. (2002). Series of essays on the First-Year Initiative Benchmarking Study. Retrieved from http://www.sc.edu/fye/resources/assessment/essays/Swing-8.28.02.html

  • Swing, R. L. (2004). The improved learning outcomes of linked versus stand-alone first-year seminars. In J. M. Henscheid (Ed.), Integrating the first-year experience: The role of learning communities in first-year seminars (Monograph No. 39, pp. 9-15). Columbia, SC: National Resource Center for the First-Year Experience and Students in Transition, University of South Carolina.

  • Tuckman, B. W., & Kennedy, G. J. (2011). Teaching learning strategies to increase success of first-term college students. Journal of Experimental Education,79, 478–504.

    Google Scholar 

  • U.S. Department of Education. (2018). Digest of education statistics. Washington, DC: National Center for Education Statistics, U.S. Department of Education. Retrieved from https://nces.ed.gov/programs/digest/index.asp.

  • Visher, M. G., Weiss, M. J., Weissman, E., Rudd, T., & Wathington, H. D. (2012). The effects of learning communities for students in developmental education: A synthesis of findings from six community colleges. New York: MDRC.

    Google Scholar 

  • Weiss, M., Brock, T., Sommo, C., Rudd, T., & Turner, M. C. (2011). Serving community college students on probation: Four-year findings from Chaffey College’s Opening Doors program. New York: MDRC.

    Google Scholar 

  • Weissman, J., & Magill, B. A. (2008). Developing a student typology to examine the effectiveness of first-year seminars. Journal of The First-Year Experience & Students in Transition,20(2), 65–90.

    Google Scholar 

  • Xiang, Y., & Wang, S. (2013). An application of propensity score stratification using multilevel models: Do charter schools make a difference in student achievement and growth?. Portland: Northwest Evaluation Association.

    Google Scholar 

  • Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education: They’re not magic. Review of Educational Research,81, 267–301.

    Google Scholar 

  • Zhao, Z. (2005). Sensitivity of propensity score methods to the specifications. Bonn: Institute for the Study of Labor.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for Research on Undergraduate Education, University of Iowa, N438 Lindquist Center, Iowa City, IA, 52242, USA

    K. C. Culver & Nicholas A. Bowman

Authors
  1. K. C. Culver
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicholas A. Bowman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to K. C. Culver.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables 6, 7.

Table 6 Descriptive statistics for the full sample and by first-year seminar participation
Full size table
Table 7 Sample sizes for each outcome for full sample and subgroups
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Culver, K.C., Bowman, N.A. Is What Glitters Really Gold? A Quasi-Experimental Study of First-Year Seminars and College Student Success. Res High Educ 61, 167–196 (2020). https://doi.org/10.1007/s11162-019-09558-8

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-019-09558-8

Keywords

Search

Navigation