Theme Analyses for Open-Ended Survey Responses in Education Research on Summer Melt Phenomenon

  • Haiying LiEmail author
  • Joyce Zhou-Yile Schnieders
  • Becky L. Bobek
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1112)


Summer melt is a phenomenon when college-intending students fail to enroll in the fall after high school graduation. Previous research on summer melt utilized surveys, typically consisting of Likert scale questions and open-ended response questions. Open-ended responses can elicit more information from students, but they have not been fully analyzed due to the cost, time, and complexity of theme extraction with manual coding. In the present study, we applied the topic modeling approach to extract topics and relevant themes, and evaluated model performance by comparing model-generated topics and categories with the human-identified topics and themes. Results showed that the topic model allows for extracting similar topics as the survey questions that were investigated, but only extracted part of the themes classified by the human. Discussion and implications focus on potential improvements in automated topic and theme classification from open-ended survey responses.


Topic modeling Summer melt Automated theme analyses 


  1. 1.
    Bettinger, E.P., Long, B.T., Oreopoulos, P., Sanbonmatsu, L.: The role of application assistance and information in college decisions: results from the H&R Block FAFSA experiment. Q. J. Econ. 127, 1205–1242 (2012)CrossRefGoogle Scholar
  2. 2.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3(1), 993–1022 (2003)zbMATHGoogle Scholar
  3. 3.
    Bobek, B.L., Zhao, R.: Education and career navigation. In: Camara, W., O’Connor, R., Mattern, K., Hanson, M.A. (eds.) Beyond Academics: A Holistic Framework for Enhancing Education and Workplace Success, pp. 39–51. ACT Inc, Iowa City (2015)Google Scholar
  4. 4.
    Castleman, B.L., Page, L.C.: A trickle or a torrent? Understanding the extent of summer melt among college-intending high school graduates. Soc. Sci. Q. 95(1), 202–220 (2014)CrossRefGoogle Scholar
  5. 5.
    Castleman, B.L., Page, L.C., Snowdon, A.L.: Strategic data project summer melt handbook: a guide to investigating and responding to Summer Melt. Center for Education Policy Research, Harvard University (2013)Google Scholar
  6. 6.
    Li, H., Zhou, Y., Bobek, B.L.: Automated identification of open-ended survey response themes in education research: a summer melt study. Poster presented at Twenty-Ninth Annual Meeting of the Society for Text & Discourse, New York, NY (2019)Google Scholar
  7. 7.
    Lucas, C., Nielsen, R.A., Roberts, M.E., Stewart, B.M., Storer, A., Tingley, D.: Computer-assisted text analysis for comparative politics. Polit. Anal. 23(2), 254–277 (2015)CrossRefGoogle Scholar
  8. 8.
    McCallum, A.K.: MALLET: a machine learning for language toolkit (2002).
  9. 9.
    Merriam, S.B., Tisdell, E.J.: Qualitative Research: A Guide to Design and Implementation. Wiley, Hoboken (2015)Google Scholar
  10. 10.
    Oreopoulos, P., Dunn, R.: Information and college access: evidence from a randomized field experiment. Scand. J. Econ. 115, 3–26 (2013)CrossRefGoogle Scholar
  11. 11.
    Paek, P.L., Bobek, B.L.: Unpacking the factors contributing to summer melt and impacting college readiness. Paper presented at the Annual Conference for the American Research Education Association (AERA), New York, NY (2018)Google Scholar
  12. 12.
    Roberts, M.E., et al.: Structural topic models for open-ended survey responses. Am. J. Polit. Sci. 58(4), 1064–1082 (2014)CrossRefGoogle Scholar
  13. 13.
    Roberts, M.E., Stewart, B.M., Airoldi, E.M.: A model of text for experimentation in the social sciences. J. Am. Stat. Assoc. 111(515), 988–1003 (2016)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Shaffer, D.W.: Quantitative Ethnography. Cathcart Press, Madison (2017)Google Scholar
  15. 15.
    Shum, S.B.: Book review: quantitative ethnography by David Williamson Shaffer. J. Learn. Anal. 6(1), 99–101 (2019)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Trostel, P., Smith, M.C.: It’s Not Just the Money: The Benefits of College Education to Individuals and to Society. Lumina Foundation, Indianapolis (2015)Google Scholar
  17. 17.
    Vandenbroucke, G.: Lifetime benefits of an education have never been so high. Reg. Econ. 25(1), 10–11 (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Haiying Li
    • 1
    Email author
  • Joyce Zhou-Yile Schnieders
    • 1
  • Becky L. Bobek
    • 1
  1. 1.ACT, Inc.Iowa CityUSA

Personalised recommendations