Abstract
An important issue in national assessment efforts is how best to measure the outcomes of college. While initial discussions about a national collegiate assessment focused on the reliability, validity, and feasibility of using achievement tests to measure student learning, subsequent discussions have raised the possibility of using students' self-reports of academic development as proxies for achievement test scores. The present study examines the stability of the relationships among self-reports and test scores across samples of two- and four-year colleges and universities. Multitrait-multimethod analyses indicated that self-reports and test scores developed from the same set of test specifications do measure the same constructs, although the scores from one type of measurement may not be “substitutable” for scores from the other type of measurement. In addition, the analyses produced ambiguous results concerning the stability of relationships across different types of institutions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anaya, G. (1992). Cognitive development among college undergraduates. Unpublished doctoral dissertation, University of California, Los Angeles.
Armor, D. J. (1974). Theta reliability and factor scaling. In H. L. Costner (ed.),Sociological Methodology 1973–1974 (pp. 17–50). San Francisco: Jossey-Bass.
Astin, A. W. (1993).What Matters in College: Four Critical Years Revisited. San Francisco: Jossey-Bass.
Baird, L. L. (1976).Using Self Reports to Predict Student Performance. New York: College Entrance Examination Board.
Banta, T. W. (1991). Toward a plan for using national assessment to ensure continuous improvement in higher education. Unpublished manuscript, Center for Assessment Research and Development, Knoxville, TN. ERIC Document Reproduction Service No. ED 340 753.
Berdie, R. F. (1971). Self-claimed and tested knowledge.Educational and Psychological Measurement 31: 629–636.
Browne, M. W., and Cudeck, R. (1989). Single sample cross-validation indices for covariance structures.Multivariate Behavioral Research 24: 445–455.
Byrne, B. M. (1989). Multigroup comparisons and the assumption of equivalent construct validity across groups: Methodological and substantive issues.Multivariate Behavioral Research 24: 503–523.
Byrne, B. M. (1993).Structural Equation Modeling with EQS and EQS/Windows. Thousand Oaks, CA: Sage.
Campbell, D. T., and Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix.Psychological Bulletin 56: 81–105.
College Entrance Examination Board (1983).Academic Preparation for College: What Examinees Need to Know and Be Able to Do. New York: Author.
Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests.Psychological Bulletin 52: 281–302.
Daly, J. A. (1994). Assessing speaking and listening: Preliminary considerations for a national assessment. In A. Greenwood (ed.),The National Assessment of College Student Learning: Identification of the Skills to Be Taught, Learned, and Assessed (pp. 113–161). Washington, DC: U.S. Government Printing Office, NCES 94-286.
Dumont, R. G., and Troelstrup, R. L. (1980). Exploring relationships between objective and subjective measures of instructional outcomes.Research in Higher Education 12: 37–51.
Dunbar, S. (1991).On the Development of a National Assessment of College Student Learning: Measurement Policy and Practice in Perspective. University of Iowa, Iowa City, IA. ERIC Document Reproduction Service No. ED 340 755.
Elliott, E. (1991). Charge to participants. In A. Greenwood (ed.),National Assessment of College Student Learning: Issues and Concerns (pp. 24–31). Washington, DC: U.S. Government Printing Office.
Ewell, P. T. (1991). To capture the ineffable: New forms of assessment in higher education. In G. Grant (ed.),Review of Research in Education (vol. 17). Washington, DC: American Educational Research Association.
Ewell, P. T. (1994). A matter of integrity: Accountability and the future of self-regulation.Change 26: 25–29.
Ewell, P. T., Lovell, C. D., Dressler, P., and Jones, D. P. (1994).A Preliminary Study of the Feasibility and Utility for National Policy of Instructional “Good Practice” Indicators in Undergraduate Education. Washington, DC: U.S. Government Printing Office. NCES 94-437.
Fiske, D. W. (1982). Convergent-discriminant validation of measurements in research strategies. In D. Brinberg and L. Kidder (eds.),Forms of Validity in Research (New Directions for the Methodology of Social and Behavioral Science Series, No. 12, pp. 77–92). San Francisco: Jossey-Bass.
Halpern, D. F. (1994). A national assessment of critical thinking skills in adults: Taking steps toward the goal. In A. Greenwood (ed.),The National Assessment of College Student Learning: Identification of the Skills to Be Taught, Learned, and Assessed (pp. 24–64). Washington, DC: U.S. Government Printing Office. NCES 94-286.
House, E. R. (1993).Professional Evaluation: Social Impact and Political Consequences. Newbury Park, CA: Sage.
Jöreskog, K. G. (1971a). Simultaneous factor analysis in several populations.Psychometrika 35: 409–426.
Jöreskog, K. G. (1971b). Statistical analysis of sets of congeneric tests.Psychometrika 36: 109–133.
Jöreskog, K. G., and Sörbom, D. (1993).LISREL 8. Chicago: Scientific Software.
Marsh, H. W. (1994). Confirmatory factor analysis models of factorial invariance: A multifaceted approach.Structural Equation Modeling 1: 5–34.
Marsh, H. W., and Hocevar, D. (1985). The application of confirmatory factor analysis to the study of self concept: First and higher order factor structures and their invariance across age groups.Psychological Bulletin 97: 562–582.
McClenney, K. (1993). Assessment in an era of empowerment.Assessment Update: Progress, Trends, and Practices in Higher Education 5(1): 1–2, 4–6.
National Education Goals Panel Resource Group on Adult Literacy and Lifelong Learning (1991). Adult literacy and lifelong learning. In National Education Goals Panel,Measuring Progress Toward the National Education Goals: Potential Indicators and Measurement Strategies (pp. 81–98). Washington, DC: U.S. Government Printing Office.
Osterlind, S. J. (1989).College BASE: Guide to Test Content. Chicago: Riverside.
Osterlind, S. J., and Merz, W. R. (1992).College BASE Technical Manual. University of Missouri-Columbia: Center for Educational Assessment.
Pace, C. R. (1987).CSEQ Test Manual and Norms. Los Angeles: Center for the Study of Evaluation.
Pascarella, E. T., and Terenzini, P. T. (1991).How College Affects Students: Findings and Insights from Twenty Years of Research. San Francisco: Jossey-Bass.
Perkins, D., Jay, E., and Tishman, S. (1994). Assessing thinking: A framework for measuring critical thinking and problem-solving skills at the college level. In A. Greenwood (ed.),The National Assessment of College Student Learning: Identification of the Skills to Be Taught, Learned, and Assessed (pp. 65–111). Washington, DC: U.S. Government Printing Office. NCES 94-286.
Pike, G. R. (1992a).A Generalizability Analysis of the College Basic Academic Subjects Examination. Knoxville, TN: Center for Assessment Research and Development, University of Tennessee.
Pike, G. R. (1992b). The components of construct validity: A comparison of two measures of general education outcomes.Journal of General Education 41: 130–159.
Pike, G. R. (1994, November). The relationship between self-report and objective measures of student achievement. Paper presented at the annual meeting of the Association for the Study of Higher Education, Tucson, AZ.
Pike, G. R. (1995). The relationship between self reports of college experiences and achievement test scores.Research in Higher Education 36: 1–22.
Pohlmann, J. T., and Beggs, D. L. (1974). A study of the validity of self-reported measures of academic growth.Journal of Educational Measurement 11: 115–119.
Ratcliff, J. L. (1991).What Type of National Assessment Fits American Higher Education? National Center for Teaching, Learning, and Assessment, Pennsylvania State University, University Park, PA. ERIC Document Reproduction Service No. ED 340 763.
Schmitt, N., and Stults, D. M. (1986). Methodology review: Analysis of multitrait-multimethod matrices.Applied Psychological Measurement 10: 1–22.
Sugawara, N., and MacCallum, R. C. (1993). Effect of estimation method on incremental fit indexes for covariance structure models.Applied Psychological Measurement 17: 365–378.
Widaman, K. F. (1985). Hierarchically nested covariance structure models for multitrait-multimethod data.Applied Psychological Measurement 9: 1–26.
Williams, L. J., and Holahan, P. J. (1994). Parsimony-based fit indices for multiple-indicator models: Do they work?Structural Equation Modeling 1: 161–189.
Wingspread Group on Higher Education (1993).An American Imperative: Higher Expectations for Higher Education. The Johnson Foundation.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Pike, G.R. Limitations of using students' self-reports of academic development as proxies for traditional achievement measures. Res High Educ 37, 89–114 (1996). https://doi.org/10.1007/BF01680043
Issue Date:
DOI: https://doi.org/10.1007/BF01680043