Research in Higher Education

, Volume 46, Issue 3, pp 251–276

An Approach to Measuring Cognitive Outcomes Across Higher Education Institutions

  • Stephen P. Klein
  • George Kuh
  • Marc Chun
  • Laura Hamilton
  • Richard Shavelson
Article

Abstract

Over the past decade, state legislatures have experienced increasing pressure to hold higher education accountable for student learning. This pressure stems from several sources, such as increasing costs and decreasing graduation rates. To explore the feasibility of one approach to measuring student learning that emphasizes program improvement, we administered several open-ended tests to 1365 students from 14 diverse colleges. The strong correspondence between hand and computer assigned scores indicates the tests can be administered and graded cost effectively on a large scale. The scores were highly reliable, especially when the college is the unit of analysis; they were sensitive to years in college; and they correlated highly with college GPAs. We also found evidence of “value added” in that scores were significantly higher at some schools than at others after controlling on the school’s mean SAT score. Finally, the students said the tasks were interesting and engaging.

Keywords

value added assessment measuring student outcomes 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Astin, A. W. 1968Undergraduate achievement and institutional ΓÇ£excellenceΓÇ¥Science161 661668CrossRefGoogle Scholar
  2. Astin, A. W. 1977Four Critical Years: Effects of College on Beliefs, Values, and Knowledge Jossey-BassSan FranciscoGoogle Scholar
  3. Astin, A. W. 1991Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education American Council on Education/MacmillanDordrechtGoogle Scholar
  4. Astin, A. W. 1993What Matters in College? Four Critical Years Revisited Jossey-BassSan FranciscoGoogle Scholar
  5. Banta, T. W.Lund, J. P.Oblander, F. W. eds. 1996Assessment in Practice: Putting Principles to Work on College Campuses Jossey-BassSan FranciscoGoogle Scholar
  6. Benjamin, R., Hersh, R. H. 2002Measuring the difference college makes: The RAND/CAE value added assessment initiativePeer Review4 710Google Scholar
  7. Black, S. 1993Portfolio AssessmentThe Executive Educator15 2831Google Scholar
  8. Bohr, L., Pascarella, E., Nora, A., Zusman, B., Jacobs, M., Desler, M., Bulakowski, C. 1994Cognitive effects of two-year and four-year institutions: a preliminary studyCommunity College Review22411CrossRefGoogle Scholar
  9. Bransford, J. D., Brown, A. L., Cocking, L. L. 1999How People Learn: Brain, MindExperienceand School National Academy PressWashington, DCGoogle Scholar
  10. Burke, J. C., Minassians, H. 2002Performance Reporting: The Preferred ΓÇ£No CostΓÇ¥ Accountability Program (2001). The Nelson A. Rockefeller Institute of GovernmentAlbanyGoogle Scholar
  11. Callan, P. M., Finney, J. E. 2002Assessing educational capital: an imperative for policyChange342531CrossRefGoogle Scholar
  12. Carini R., Kuh G. and Klein S. 2004 Student engagement and student learning: insights from a construct validation study. San DiegoCalifornia: Paper presented at the meetings of the American Educational Research AssociationGoogle Scholar
  13. Carroll, J. B. 1993

    Human Cognitive Abilities

    A Survey of Factor-Analytic Studies Cambridge University PressCambridgeEngland
    CrossRefGoogle Scholar
  14. Chickering, A. W., Gamson, Z. F. 1987Seven principles for good practice in undergraduate educationAmerican Association for Higher Education Bulletin39 37Google Scholar
  15. Cole, J. J. K., Nettles, M. T., Sharp, S. 1997Assessment of teaching and learning for improvement and accountability: state governing, coordinating board and regional accreditation association policies and practices University of Michigan, National Center for Postsecondary ImprovementAnn ArborGoogle Scholar
  16. Cronbach, L. J. eds. 2000Remaking the Concept of Aptitude: Extending the Legacy of Richard E. Snow ErlbaumMahway, NJGoogle Scholar
  17. Dey, E., Hurtado, S., Rhee, B., Inkelas, K. K., Wimsatt, L. A., Guan, F. 1997Improving Research on Postsecondary Outcomes:A Review of the Strengths and Limitations of National Data Sources National Center for Postsecondary ImprovementStanfordCAGoogle Scholar
  18. Erwin, T. D., Schrell, K. W. 2003Assessment of critical thinking: New Jersey’s tasks in critical thinkingThe Journal of General Education525070CrossRefGoogle Scholar
  19. Ewell, P. T. 1984The Self-regarding Institution: Information for ExcellenceNational Center for Higher Education Management SystemsBoulderCOGoogle Scholar
  20. Ewell P. T. 1987. Establishing a campus-based assessment program. In D. F. Halpern. Student outcomes assessment: what institutions stand to gain. New Directions for Higher Education 59: 9--24.Google Scholar
  21. Ewell, P. T. 1988

    Outcomes, assessmentand academic improvement: In search of usable knowledge

    Smart, J. C. eds. Higher Education: Handbook of Theory and ResearchAgathon PressDordrecht53108Vol. IV
    Google Scholar
  22. Ewell, P. T. 1994A Policy Guide for Assessment: Making Good Use of the Tasks in Critical ThinkingEducational Testing ServicePrinceton, NJGoogle Scholar
  23. Flowers, L., Osterlind, S. J., Pascarella, E. T., Pierson, C. T. 2001How much do students learn in colleges?The Journal of Higher Education72565583CrossRefGoogle Scholar
  24. Fong, B. 1988

    Assessing the departmental major

    McMillan, J. H. eds. Assessing Students’ Learning. New Directions for Teaching and LearningJossey-BassSan Francisco7183Vol. 34
    Google Scholar
  25. Forrest A. 1990 Time Will Tell: Portfolio-assisted Assessment of General Education. The AAHE Assessment ForumAmerican Association for Higher EducationGoogle Scholar
  26. Gates, S. M., Augustine, C., Benjamin, R., Bikson, T., Derghazarian, E., Kaganoff, T., Levy, D., Moini, J., Zimmer, R. 2001Ensuring the quality and productivity of education and professional development activities: a review of approaches and lessons for DoDNational Defense Research InstituteRANDSanta MonicaCAGoogle Scholar
  27. Gentemann, K. M., Fletcher, J. J., Potter, D. L. 1994

    Refocusing the academic program review on student learning

    Kinnick, M. K. eds. Providing useful information for deans and department chairs, New Directions for Institutional Research No. 84Jossey-BassSan Francisco3146
    Google Scholar
  28. Gill, W. E. 1993Conversations about accreditation: Middle States Association of Colleges and Schools: Focusing on outcomes assessment in the accreditation processPaper presented at Double Feature Conference on Assessment and Continuous Quality Improvement of the American Association for Higher EducationChicagoIL(ERIC Document Reproduction Service No. ED 358 792).Google Scholar
  29. Graham, A., and Thompson, N. (2001). Broken ranks: U.S. News’ college rankings measure everything but what matters. And most universities do not seem to mind. The Washington monthly. Available at: www.washingtonmonthly.com/features/2001/0109.graham.thompson.html.Google Scholar
  30. Gustafsson, J. E., Undheim, J. O. 1996

    Individual differences in cognitive functions

    R., CalfeeD., Berliner eds. Handbook of Educational PsychologyMacmillanDordrecht186242
    Google Scholar
  31. Halpern, D. F. 1987

    Recommendations and caveats

    D. F., Halpern eds. Student Outcomes Assessment: What Institutions Stand to Gain. New directions for higher educationJossey-BassSan Francisco109111Vol. 59
    Google Scholar
  32. Hutchings P. 1989. Behind outcomes: contexts and questions. The AAHE Assessment Forum American Association for Higher Education Google Scholar
  33. Immerwahl, J. 2000Great Expectations: How Californians View Higher EducationNational Center for Public Policy and Higher Education and Public AgendaSan JoseCA(Table 3 National Column).Google Scholar
  34. Jacobi, M., Astin, A., Ayala, F. 1987College student outcomes assessment: a talent development perspectiveAssociation for the Study of Higher EducationWashington, DC(ASHE-ERIC Higher Education Report No. 7).Google Scholar
  35. Johnson, R., McCormick, R. D., Prus, J. S., Rogers, J. S.,  et al. 1993

    Assessment options for the college major

    T. W., Banta eds. Making a Difference: Outcomes of a Decade of Assessment in Higher EducationJossey-BassSan Francisco151167
    Google Scholar
  36. Klein, S. 1996The costs and benefits of performance testing on the bar examinationThe Bar Examiner651320Google Scholar
  37. Klein, S. and Hamilton L. 1998. The validity of the U.S. News and World Report ranking of ABA law schools. Report commissioned by the Association of American Law Schools (available on the web at http://www.aals.org/validity.html).Google Scholar
  38. Klein, S. 2001Rationale and plan for assessing higher education outcomes with direct constructed response measures of student skillsCouncil for Aid to Education, Higher Education Policy SeriesDordrecht, NYNumber 3.Google Scholar
  39. Klein, S. 2002Direct assessment of cumulative student learningPeer Review42628Google Scholar
  40. Klein, S., Kuh, G., Chun, M., Hamilton, L., and Shavelson R. 2003. The search for Value-Added: Assessing and validating selected higher education outcomes. Paper presented at the meetings of the American Educational Research Association. ChicagoIllinoisGoogle Scholar
  41. Klein, S., Shavelson, R., Hamilton, L. and Chun M. 2004. Characteristics of hand and machine-assigned scores to college students’ answers to open-ended tasks (unpublished report).Google Scholar
  42. Kyllonen, P.C., Shute, V.J. 1989

    A taxonomy of learning skills

    Ackerman, P.ΓÉúL.Sternberg, R. J.Glaser, R. eds. Learning and Individual Differences: Advances in Theory and ResearchFreemanDordrecht117163
    Google Scholar
  43. Kuh, G. D. 2001Assessing what really matters to student learning: inside the National Survey of Student EngagementChange33101766CrossRefGoogle Scholar
  44. Kuh, G. D. 2003What we’re learning about student engagement from NSSEChange352432CrossRefGoogle Scholar
  45. Lenning, O. T. 1988

    Use of noncognitive measures in assessment

    Banta, T. W. eds. Implementing Outcomes Assessment: Promise and Perils. New Directions for Institutional ResearchJossey-BassSan Francisco4151Vol. 59
    Google Scholar
  46. Machung, A. 1995Changes in college rankings: How real are they?Paper presented at the 35th Annual AIR ForumBoston, MAGoogle Scholar
  47. Martinez, M. E. 2000Education as the Cultivation of IntelligenceErlbaumMahway, NJGoogle Scholar
  48. McCaffrey, D. F., Lockwood, J. R., Koretz, D. M., Hamilton, L. S. 2003Evaluating Value-added Models for Teacher AccountabilityRANDSanta MonicaCACrossRefGoogle Scholar
  49. McGuire, M. D. 1995

    Validity issues for reputational studies

    Walleri, R. D.Moss, M. K. eds. Evaluating and Responding to College Guidebooks and Rankings. New Directions for Institutional ResearchJossey-BassSan FranciscoVol. 88.
    Google Scholar
  50. Messick, S. 1984The Psychology of Educational MeasurementJournal of Educational Measurement21215237CrossRefGoogle Scholar
  51. Muffo, J. A., Bunda, M. A.,  et al. 1993

    Attitude and Opinion Data

    Banta, T eds. Making a difference: Outcomes of a decade of assessment in higher educationJossey-BassSan Francisco139150
    Google Scholar
  52. National Center for Higher Education Management Systems (NCEMS)1994A preliminary study of the feasibility and utility for national policy of instructional and good practice indicators in undergraduate educationNational Center for Higher Education Management SystemsBoulderCOContractor Report for the National Center for Education Statistics.Google Scholar
  53. National Center for Higher Education Management Systems (NCEMS)1996The National Assessment of College Student Learning: An Inventory of State-level Assessment ActivitiesNational Center for Higher Education Management SystemsBoulderCOGoogle Scholar
  54. National Opinion Research Center. 1997. A review of the methodology for the U.S. News and World Report’s rankings of undergraduate colleges and universities. Report by the National Opinion Research CenterGoogle Scholar
  55. National Postsecondary Education Cooperative. 2000a The NPEC sourcebook on assessmentvolume 1: Definitions and assessment methods for critical thinking, problem solving, and writing. Center for Assessment and Research Studies, James Madison University, Harrisonburg, VAunder the sponsorship of the National Center for Education Statistics, U.S. Department of EducationGoogle Scholar
  56. National Postsecondary Education Cooperative. 2000b. The NPEC sourcebook on assessmentvolume 2: Selected institutions utilizing assessment results. Center for Assessment and Research Studies, James Madison University, Harrisonburg, VAunder the sponsorship of the National Center for Education Statistics, U.S. Department of EducationGoogle Scholar
  57. Naughton, B. A., Suen, A. Y., Shavelson, R. J. 2003Accountability for what? Understanding the learning objectives in state higher education accountability programsPaper presented at the annual meetings of the American Educational Research AssociationChicagoGoogle Scholar
  58. Obler, S. S., Slark, J., Umbdenstock, L.,  et al. 1993

    Classroom assessment

    Banta, T. W. eds. Making a difference: Outcomes of a decade of assessment in higher educationJossey-BassSan Francisco211226
    Google Scholar
  59. Pace, C. R. 1990The undergraduates; A report of their activities and progress in college in the 1980’sCenter for the Study of Evaluation, University of CaliforniaLos AngelesLos AngelesGoogle Scholar
  60. Palomba, C. A., Banta, T. W. 1999Assessment essentials: Planning, implementing, and improving assessment in higher educationJossey-BassSan FranciscoGoogle Scholar
  61. Pascarella, E. T., Terenzini, P. T. 1991How college affects students: Findings and insights from twenty years of researchJossey-BassSan FranciscoGoogle Scholar
  62. Pascarella, E. T., Bohr, L., Nora, A., Terenzini, P.T. 1996ΓÇ£Is differential exposure to college linked to the development of critical thinking?ΓÇ¥Research in Higher Education37159174CrossRefGoogle Scholar
  63. Pascarella, E. T. 2001Cognitive growth in collegeChange332127CrossRefGoogle Scholar
  64. Pellegrino, J. W.Chudowsky, N.Glaser, R. eds. 2001Knowing What Students Know: The Science and Design of Educational Assessment National Academy PressWashington, DCGoogle Scholar
  65. Ratcliff, J. L., Jones, E. A., Guthrie, D. S., Oehler, D. 1991The effect of coursework patterns, advisementand course selection on the development of generalΓÉúlearned abilities of college graduatesThe Pennsylvania StateΓÉúUniversity, National Center on Postsecondary Teaching, Learning, and AssessmentUniversity ParkGoogle Scholar
  66. Ratcliff, J. L., Jones, E. A.,  et al. 1997Turning results into improvement strategiesThe Pennsylvania State University, National Center on Postsecondary Teaching, Learning, and AssessmentUniversity ParkGoogle Scholar
  67. Riggs, M. L., and Worthley, J. S. (1992). Baseline Characteristics of Successful Program of Student Outcomes AssessmentERIC document ED353285.Google Scholar
  68. Shavelson, R.J., Roeser, R.W., Kupermintz, H., Lau, S., Ayala, C., Haydel, A., Schultz, S., Quihuis, G., Gallagher, L. 2002Richard E. Snow’s remaking of the concept of aptitude and multidimensional test validity: introduction to the special issueEducational Assessment877100CrossRefGoogle Scholar
  69. Shavelson, R.J., Huang, L. 2003Responding responsibly to the frenzy to assess learning in higher educationChange351019CrossRefGoogle Scholar
  70. Smith, M. K., Bradley, J. L., Draper, G. F. 1993A National Survey on Assessment PracticesUniversity of TennesseeKnoxvilleClearinghouse for Higher Education Assessment InstrumentsKnoxvilleTNGoogle Scholar
  71. Snow, R. E. 1994

    Abilities in Academic Tasks

    Sternberg, R. J.Wagner, R. K. eds. Mind in Context: Interactionist Perspectives on Human IntelligenceCambridge University PressCambridgeEngland337
    Google Scholar
  72. Snow, R. E., Lohman, D. F. 1989

    Implications of cognitive psychology for educational measurement

    Linn, R. eds. Educational Measurement3rd ed.MacmillanDordrecht263331
    Google Scholar
  73. Steele, J. M., Lutz, D. A. 1995Report of ACT’s research on postsecondary assessment needsAmerican College Testing ProgramIowa City, IAGoogle Scholar
  74. Suen H. K., Parkes J. 1996. Challenges and opportunities for student assessment in distance education. Distance Education Online Symposium. 6(7): [On-line serial]. Available: Internet: ACSDE@PSUVM.PSU.EDUGoogle Scholar
  75. Terenzini, P. T., Wright, T. 1987Influences on students’ academic growth during four years of collegeResearch in Higher Education26161179CrossRefGoogle Scholar
  76. Terenzini, P. T. 1989Assessment with open eyes: pitfalls in studying student outcomesJournal of Higher Education60644664CrossRefGoogle Scholar
  77. Vandament W. E. 1987. A state university perspective on student outcomes assessment In Halpern, D. F. Student outcomes assessment: what institutions stand to gain. New Directions for Higher Education 59: 25--28Google Scholar
  78. Waluconis, C. J. 1993

    Student self-evaluation

    Trudy, B. eds. Making a difference: Outcomes of a decade of assessment in higher educationJossey-BassSan Francisco244255
    Google Scholar
  79. Winter, D. G., McClelland, D. C., Stewart, A. J. 1981A new case for the liberal artsJossey-BassSan FranciscoGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2005

Authors and Affiliations

  • Stephen P. Klein
    • 1
  • George Kuh
    • 2
  • Marc Chun
    • 3
  • Laura Hamilton
    • 4
  • Richard Shavelson
    • 5
  1. 1.The RAND CorporationSanta MonicaUSA.
  2. 2.Indiana University Center for Postsecondary ResearchBloomington
  3. 3.Council for Aid to EducationNew York City
  4. 4.The RAND CorporationPittsburgh
  5. 5.Graduate School of Education and Department of PsychologyStanford UniversityStanford

Personalised recommendations