Skip to main content

Advertisement

Log in

An Approach to Measuring Cognitive Outcomes Across Higher Education Institutions

  • Published:
Research in Higher Education Aims and scope Submit manuscript

Abstract

Over the past decade, state legislatures have experienced increasing pressure to hold higher education accountable for student learning. This pressure stems from several sources, such as increasing costs and decreasing graduation rates. To explore the feasibility of one approach to measuring student learning that emphasizes program improvement, we administered several open-ended tests to 1365 students from 14 diverse colleges. The strong correspondence between hand and computer assigned scores indicates the tests can be administered and graded cost effectively on a large scale. The scores were highly reliable, especially when the college is the unit of analysis; they were sensitive to years in college; and they correlated highly with college GPAs. We also found evidence of “value added” in that scores were significantly higher at some schools than at others after controlling on the school’s mean SAT score. Finally, the students said the tasks were interesting and engaging.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • A. W. Astin (1968) ArticleTitleUndergraduate achievement and institutional ΓÇ£excellenceΓÇ¥ Science 161 661–668 Occurrence Handle10.1126/science.161.3842.661

    Article  Google Scholar 

  • A. W. Astin (1977) Four Critical Years: Effects of College on Beliefs, Values, and Knowledge Jossey-Bass San Francisco

    Google Scholar 

  • A. W. Astin (1991) Assessment for Excellence: The Philosophy and Practice of Assessment and Evaluation in Higher Education American Council on Education/Macmillan Dordrecht

    Google Scholar 

  • A. W. Astin (1993) What Matters in College? Four Critical Years Revisited Jossey-Bass San Francisco

    Google Scholar 

  • T. W. Banta J. P. Lund F. W. Oblander (Eds) (1996) Assessment in Practice: Putting Principles to Work on College Campuses Jossey-Bass San Francisco

    Google Scholar 

  • R. Benjamin R. H. Hersh (2002) ArticleTitleMeasuring the difference college makes: The RAND/CAE value added assessment initiative Peer Review 4 7–10

    Google Scholar 

  • S. Black (1993) ArticleTitlePortfolio Assessment The Executive Educator 15 28–31

    Google Scholar 

  • L. Bohr E. Pascarella A. Nora B. Zusman M. Jacobs M. Desler C. Bulakowski (1994) ArticleTitleCognitive effects of two-year and four-year institutions: a preliminary study Community College Review 22 IssueID1 411 Occurrence Handle10.1177/009155219402200102

    Article  Google Scholar 

  • J. D. Bransford A. L. Brown L. L. Cocking (1999) How People Learn: Brain, MindExperienceand School National Academy Press Washington, DC

    Google Scholar 

  • J. C. Burke H. Minassians (2002) Performance Reporting: The Preferred ΓÇ£No CostΓÇ¥ Accountability Program (2001). The Nelson A. Rockefeller Institute of Government Albany

    Google Scholar 

  • P. M. Callan J. E. Finney (2002) ArticleTitleAssessing educational capital: an imperative for policy Change 34 25–31 Occurrence Handle10.1080/00091380209604732

    Article  Google Scholar 

  • Carini R., Kuh G. and Klein S. 2004 Student engagement and student learning: insights from a construct validation study. San DiegoCalifornia: Paper presented at the meetings of the American Educational Research Association

  • J. B. Carroll (1993) Human Cognitive Abilities A Survey of Factor-Analytic Studies Cambridge University Press CambridgeEngland Occurrence Handle10.1017/CBO9780511571312

    Chapter  Google Scholar 

  • A. W. Chickering Z. F. Gamson (1987) ArticleTitleSeven principles for good practice in undergraduate education American Association for Higher Education Bulletin 39 IssueID7 3–7

    Google Scholar 

  • J. J. K. Cole M. T. Nettles S. Sharp (1997) Assessment of teaching and learning for improvement and accountability: state governing, coordinating board and regional accreditation association policies and practices University of Michigan, National Center for Postsecondary Improvement Ann Arbor

    Google Scholar 

  • L. J. Cronbach (Eds) (2000) Remaking the Concept of Aptitude: Extending the Legacy of Richard E. Snow Erlbaum Mahway, NJ

    Google Scholar 

  • E. Dey S. Hurtado B. Rhee K. K. Inkelas L. A. Wimsatt F. Guan (1997) Improving Research on Postsecondary Outcomes:A Review of the Strengths and Limitations of National Data Sources National Center for Postsecondary Improvement StanfordCA

    Google Scholar 

  • T. D. Erwin K. W. Schrell (2003) ArticleTitleAssessment of critical thinking: New Jersey’s tasks in critical thinking The Journal of General Education 52 50–70 Occurrence Handle10.1353/jge.2003.0019

    Article  Google Scholar 

  • P. T. Ewell (1984) The Self-regarding Institution: Information for Excellence National Center for Higher Education Management Systems BoulderCO

    Google Scholar 

  • Ewell P. T. 1987. Establishing a campus-based assessment program. In D. F. Halpern. Student outcomes assessment: what institutions stand to gain. New Directions for Higher Education 59: 9--24.

  • P. T. Ewell (1988) Outcomes, assessmentand academic improvement: In search of usable knowledge J. C. Smart (Eds) Higher Education: Handbook of Theory and Research Agathon Press Dordrecht 53–108

    Google Scholar 

  • P. T. Ewell (1994) A Policy Guide for Assessment: Making Good Use of the Tasks in Critical Thinking Educational Testing Service Princeton, NJ

    Google Scholar 

  • L. Flowers S. J. Osterlind E. T. Pascarella C. T. Pierson (2001) ArticleTitleHow much do students learn in colleges? The Journal of Higher Education 72 IssueID5 565–583 Occurrence Handle10.2307/2672881

    Article  Google Scholar 

  • B. Fong (1988) Assessing the departmental major J. H. McMillan (Eds) Assessing Students’ Learning. New Directions for Teaching and Learning Jossey-Bass San Francisco 71–83

    Google Scholar 

  • Forrest A. 1990 Time Will Tell: Portfolio-assisted Assessment of General Education. The AAHE Assessment ForumAmerican Association for Higher Education

  • S. M. Gates C. Augustine R. Benjamin T. Bikson E. Derghazarian T. Kaganoff D. Levy J. Moini R. Zimmer (2001) Ensuring the quality and productivity of education and professional development activities: a review of approaches and lessons for DoD National Defense Research InstituteRAND Santa MonicaCA

    Google Scholar 

  • K. M. Gentemann J. J. Fletcher D. L. Potter (1994) Refocusing the academic program review on student learning M. K. Kinnick (Eds) Providing useful information for deans and department chairs, New Directions for Institutional Research No. 84 Jossey-Bass San Francisco 31–46

    Google Scholar 

  • W. E. Gill (1993) Conversations about accreditation: Middle States Association of Colleges and Schools: Focusing on outcomes assessment in the accreditation process Paper presented at Double Feature Conference on Assessment and Continuous Quality Improvement of the American Association for Higher Education ChicagoIL

    Google Scholar 

  • Graham, A., and Thompson, N. (2001). Broken ranks: U.S. News’ college rankings measure everything but what matters. And most universities do not seem to mind. The Washington monthly. Available at: www.washingtonmonthly.com/features/2001/0109.graham.thompson.html.

  • J. E. Gustafsson J. O. Undheim (1996) Individual differences in cognitive functions Calfee R. Berliner D. (Eds) Handbook of Educational Psychology Macmillan Dordrecht 186–242

    Google Scholar 

  • D. F. Halpern (1987) Recommendations and caveats Halpern D. F. (Eds) Student Outcomes Assessment: What Institutions Stand to Gain. New directions for higher education Jossey-Bass San Francisco 109–111

    Google Scholar 

  • Hutchings P. 1989. Behind outcomes: contexts and questions. The AAHE Assessment Forum American Association for Higher Education

  • J. Immerwahl (2000) Great Expectations: How Californians View Higher Education National Center for Public Policy and Higher Education and Public Agenda San JoseCA

    Google Scholar 

  • M. Jacobi A. Astin F. Ayala (1987) College student outcomes assessment: a talent development perspective Association for the Study of Higher Education Washington, DC

    Google Scholar 

  • R. Johnson R. D. McCormick J. S. Prus J. S. Rogers (1993) Assessment options for the college major Banta T. W. (Eds) et al. Making a Difference: Outcomes of a Decade of Assessment in Higher Education Jossey-Bass San Francisco 151–167

    Google Scholar 

  • S. Klein (1996) ArticleTitleThe costs and benefits of performance testing on the bar examination The Bar Examiner 65 IssueID3 13–20

    Google Scholar 

  • Klein, S. and Hamilton L. 1998. The validity of the U.S. News and World Report ranking of ABA law schools. Report commissioned by the Association of American Law Schools (available on the web at http://www.aals.org/validity.html).

  • S. Klein (2001) Rationale and plan for assessing higher education outcomes with direct constructed response measures of student skills Council for Aid to Education, Higher Education Policy Series Dordrecht, NY

    Google Scholar 

  • S. Klein (2002) ArticleTitleDirect assessment of cumulative student learning Peer Review 4 26–28

    Google Scholar 

  • Klein, S., Kuh, G., Chun, M., Hamilton, L., and Shavelson R. 2003. The search for Value-Added: Assessing and validating selected higher education outcomes. Paper presented at the meetings of the American Educational Research Association. ChicagoIllinois

  • Klein, S., Shavelson, R., Hamilton, L. and Chun M. 2004. Characteristics of hand and machine-assigned scores to college students’ answers to open-ended tasks (unpublished report).

  • P.C. Kyllonen V.J. Shute (1989) A taxonomy of learning skills P.ΓÉúL. Ackerman R. J. Sternberg R. Glaser (Eds) Learning and Individual Differences: Advances in Theory and Research Freeman Dordrecht 117–163

    Google Scholar 

  • G. D. Kuh (2001) ArticleTitleAssessing what really matters to student learning: inside the National Survey of Student Engagement Change 33 IssueID3 10–17 Occurrence Handle10.1080/00091380109601795

    Article  Google Scholar 

  • G. D. Kuh (2003) ArticleTitleWhat we’re learning about student engagement from NSSE Change 35 IssueID2 24–32 Occurrence Handle10.1080/00091380309604090

    Article  Google Scholar 

  • O. T. Lenning (1988) Use of noncognitive measures in assessment T. W. Banta (Eds) Implementing Outcomes Assessment: Promise and Perils. New Directions for Institutional Research Jossey-Bass San Francisco 41–51

    Google Scholar 

  • A. Machung (1995) Changes in college rankings: How real are they? Paper presented at the 35th Annual AIR Forum Boston, MA

    Google Scholar 

  • M. E. Martinez (2000) Education as the Cultivation of Intelligence Erlbaum Mahway, NJ

    Google Scholar 

  • D. F. McCaffrey J. R. Lockwood D. M. Koretz L. S. Hamilton (2003) Evaluating Value-added Models for Teacher Accountability RAND Santa MonicaCA Occurrence Handle10.1037/e658712010-001

    Book  Google Scholar 

  • M. D. McGuire (1995) Validity issues for reputational studies R. D. Walleri M. K. Moss (Eds) Evaluating and Responding to College Guidebooks and Rankings. New Directions for Institutional Research Jossey-Bass San Francisco

    Google Scholar 

  • S. Messick (1984) ArticleTitleThe Psychology of Educational Measurement Journal of Educational Measurement 21 IssueID3 215–237 Occurrence Handle10.1111/j.1745-3984.1984.tb01030.x

    Article  Google Scholar 

  • J. A. Muffo M. A. Bunda (1993) Attitude and Opinion Data T Banta (Eds) et al. Making a difference: Outcomes of a decade of assessment in higher education Jossey-Bass San Francisco 139–150

    Google Scholar 

  • InstitutionalAuthorNameNational Center for Higher Education Management Systems (NCEMS) (1994) A preliminary study of the feasibility and utility for national policy of instructional and good practice indicators in undergraduate education National Center for Higher Education Management Systems BoulderCO

    Google Scholar 

  • InstitutionalAuthorNameNational Center for Higher Education Management Systems (NCEMS) (1996) The National Assessment of College Student Learning: An Inventory of State-level Assessment Activities National Center for Higher Education Management Systems BoulderCO

    Google Scholar 

  • National Opinion Research Center. 1997. A review of the methodology for the U.S. News and World Report’s rankings of undergraduate colleges and universities. Report by the National Opinion Research Center

  • National Postsecondary Education Cooperative. 2000a The NPEC sourcebook on assessmentvolume 1: Definitions and assessment methods for critical thinking, problem solving, and writing. Center for Assessment and Research Studies, James Madison University, Harrisonburg, VAunder the sponsorship of the National Center for Education Statistics, U.S. Department of Education

  • National Postsecondary Education Cooperative. 2000b. The NPEC sourcebook on assessmentvolume 2: Selected institutions utilizing assessment results. Center for Assessment and Research Studies, James Madison University, Harrisonburg, VAunder the sponsorship of the National Center for Education Statistics, U.S. Department of Education

  • B. A. Naughton A. Y. Suen R. J. Shavelson (2003) Accountability for what? Understanding the learning objectives in state higher education accountability programs Paper presented at the annual meetings of the American Educational Research Association Chicago

    Google Scholar 

  • S. S. Obler J. Slark L. Umbdenstock (1993) Classroom assessment T. W. Banta (Eds) et al. Making a difference: Outcomes of a decade of assessment in higher education Jossey-Bass San Francisco 211–226

    Google Scholar 

  • C. R. Pace (1990) The undergraduates; A report of their activities and progress in college in the 1980’s Center for the Study of Evaluation, University of CaliforniaLos Angeles Los Angeles

    Google Scholar 

  • C. A. Palomba T. W. Banta (1999) Assessment essentials: Planning, implementing, and improving assessment in higher education Jossey-Bass San Francisco

    Google Scholar 

  • E. T. Pascarella P. T. Terenzini (1991) How college affects students: Findings and insights from twenty years of research Jossey-Bass San Francisco

    Google Scholar 

  • E. T. Pascarella L. Bohr A. Nora P.T. Terenzini (1996) ArticleTitleΓÇ£Is differential exposure to college linked to the development of critical thinking?ΓÇ¥ Research in Higher Education 37 159–174 Occurrence Handle10.1007/BF01730114

    Article  Google Scholar 

  • E. T. Pascarella (2001) ArticleTitleCognitive growth in college Change 33 21–27 Occurrence Handle10.1080/00091380109601806

    Article  Google Scholar 

  • J. W. Pellegrino N. Chudowsky R. Glaser (Eds) (2001) Knowing What Students Know: The Science and Design of Educational Assessment National Academy Press Washington, DC

    Google Scholar 

  • J. L. Ratcliff E. A. Jones D. S. Guthrie D. Oehler (1991) The effect of coursework patterns, advisementand course selection on the development of generalΓÉúlearned abilities of college graduates The Pennsylvania StateΓÉúUniversity, National Center on Postsecondary Teaching, Learning, and Assessment University Park

    Google Scholar 

  • J. L. Ratcliff E. A. Jones et al. (1997) Turning results into improvement strategies The Pennsylvania State University, National Center on Postsecondary Teaching, Learning, and Assessment University Park

    Google Scholar 

  • Riggs, M. L., and Worthley, J. S. (1992). Baseline Characteristics of Successful Program of Student Outcomes AssessmentERIC document ED353285.

  • R.J. Shavelson R.W. Roeser H. Kupermintz S. Lau C. Ayala A. Haydel S. Schultz G. Quihuis L. Gallagher (2002) ArticleTitleRichard E. Snow’s remaking of the concept of aptitude and multidimensional test validity: introduction to the special issue Educational Assessment 8 IssueID2 77–100 Occurrence Handle10.1207/S15326977EA0802_01

    Article  Google Scholar 

  • R.J. Shavelson L. Huang (2003) ArticleTitleResponding responsibly to the frenzy to assess learning in higher education Change 35 IssueID1 10–19 Occurrence Handle10.1080/00091380309604739

    Article  Google Scholar 

  • M. K. Smith J. L. Bradley G. F. Draper (1993) A National Survey on Assessment Practices University of TennesseeKnoxvilleClearinghouse for Higher Education Assessment Instruments KnoxvilleTN

    Google Scholar 

  • R. E. Snow (1994) Abilities in Academic Tasks R. J. Sternberg R. K. Wagner (Eds) Mind in Context: Interactionist Perspectives on Human Intelligence Cambridge University Press CambridgeEngland 337

    Google Scholar 

  • R. E. Snow D. F. Lohman (1989) Implications of cognitive psychology for educational measurement R. Linn (Eds) Educational Measurement EditionNumber3rd ed. Macmillan Dordrecht 263–331

    Google Scholar 

  • J. M. Steele D. A. Lutz (1995) Report of ACT’s research on postsecondary assessment needs American College Testing Program Iowa City, IA

    Google Scholar 

  • Suen H. K., Parkes J. 1996. Challenges and opportunities for student assessment in distance education. Distance Education Online Symposium. 6(7): [On-line serial]. Available: Internet: ACSDE@PSUVM.PSU.EDU

  • P. T. Terenzini T. Wright (1987) ArticleTitleInfluences on students’ academic growth during four years of college Research in Higher Education 26 161–179 Occurrence Handle10.1007/BF00992027

    Article  Google Scholar 

  • P. T. Terenzini (1989) ArticleTitleAssessment with open eyes: pitfalls in studying student outcomes Journal of Higher Education 60 644–664 Occurrence Handle10.2307/1981946

    Article  Google Scholar 

  • Vandament W. E. 1987. A state university perspective on student outcomes assessment In Halpern, D. F. Student outcomes assessment: what institutions stand to gain. New Directions for Higher Education 59: 25--28

  • C. J. Waluconis (1993) Student self-evaluation B. Trudy (Eds) Making a difference: Outcomes of a decade of assessment in higher education Jossey-Bass San Francisco 244–255

    Google Scholar 

  • D. G. Winter D. C. McClelland A. J. Stewart (1981) A new case for the liberal arts Jossey-Bass San Francisco

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Stephen P. Klein.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Klein, S.P., Kuh, G., Chun, M. et al. An Approach to Measuring Cognitive Outcomes Across Higher Education Institutions. Res High Educ 46, 251–276 (2005). https://doi.org/10.1007/s11162-004-1640-3

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11162-004-1640-3

Keywords

Navigation