Manual and Automatic Estimates of Growth and Gain Across Year Levels: How Close is Close?
Users of statistical software are frequently unaware of the calculations underlying the routines that they use. Indeed, users—particularly in the social sciences—are often somewhat adverse towards the underlying mathematics, Yet, in order to appreciate the thrust of certain routines, it is beneficial to understand the way in which a program arrives at a particular solution. Based on data from the Economic Literacy Study conducted at year 11 and 12 level across Queensland in 1998, this article renders explicit the steps involved in calculating growth and gain estimates in student performance. To this end, the first part of the article describes the Omanual calculation of such estimates using the Rasch estimates of item thresholds of common items at the different year levels produced by Quest (Adams & Khoo, 1993) as a starting point for the subsequent calibrating, scoring and equating. In the second part of the chapter, we explore the extent to which estimates of change in performance across year levels that are calculated with ConQuest (Wu, Adams & Wilson, 1997).
The article shows that the manual and automatic way of calculating growth and gain estimates produce nearly identical results. This is not only reassuring from a technical point of view but also from an educational point of view as this means that the reader of the non-mathematical discussion of the manual calculation procedure will develop a better understanding of the processes involved in calculating growth and gain estimates.
Key wordsunidimensional latent regression gain calibrate scoring equating across year levels test performance economic literacy
Unable to display preview. Download preview PDF.
- Adams RJ & Khoo SK 1993 Quest — The interactive test analysis system. Hawthorn, Vic.: Australian Council for Educational Research.Google Scholar
- Adams RJ, Wilson M & Wu M 1997 Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), pp. 47–76.Google Scholar
- Australian Bureau of Statistics (ABS) 1999 Census Update. http://www.abs.gov.au/websitedbs/D31 10129.NSF.Google Scholar
- Beck K & Krumm V 1989 Economic literacy in German speaking countries and the United States. First steps to a comparative study. Paper presented at the annual meeting of AERA, San Francisco.Google Scholar
- Elley WB 1992 How in the world do students read? The Hague: IEA.Google Scholar
- Harmon M, Smith TA, Martin MO, Kelly DL, Beaton AE, Mullis IVS, Gonzalez EJ & Orpwood G 1997 Performance Assessment in IEA’s Third International Mathematics and Science Study. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College & Amsterdam: IEA.Google Scholar
- Keeves JP & Kotte D 1996 The measurement and reporting of key competencies. The Flinders University of South Australia, Adelaide.Google Scholar
- Keeves JP 1992 Learning Science in a Changing World. Cross-national studies of Science Achievement: 1970 to 1984. The Hague: IEA.Google Scholar
- Keeves JP 1996 The world of school learning. Selected key findings from 35 years of IEA research. The Hague: IEA.Google Scholar
- Kotte D & Lietz P 1998 Welche Faktoren beeinflussen die Leistung in Wirtschaftskunde? Zeitschrift für Berufs-und Wirtschaftspädagogik, Vol. 94, No. X, pp. 421–434.Google Scholar
- Lietz P & Kotte D 1997 Economic literacy in Central Queensland: Results of a pilot study. Paper presented at the Australian Association for Research in Education (AARE) annual meeting, Brisbane, 1–4 December, 1997.Google Scholar
- Lietz P 1996 Reading comprehension across cultures and over time. Münster/New York: Waxmann.Google Scholar
- Loehlin JC 1998 Latent variable models (3rd Ed.). Mahwah, NJ: Erlbaum.Google Scholar
- Lokan J, Ford P & Greenwood L 1996 Maths & Science on the Line: Australian junior secondary students’ performance in the Third International Mathematics and Science Study. Melbourne: Australian Council for Educational Research.Google Scholar
- Lokan J, Ford P & Greenwood L 1997 Maths & Science on the Line: Australian middle primary students’ performance in the Third International Mathematics and Science Study. Melbourne: Australian Council for Educational Research.Google Scholar
- Martin MO & Kelly DA (eds) 1996 Third International Mathematics and Science Study Technical Report, Volume II: Design and Development. Primary and Middle School Years. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College & Amsterdam: International Association for the Evaluation of Educational Achievement (IEA).Google Scholar
- OECD 1998 The PISA Assessment Framework — An Overview. September 1998: Draft of the PISA Project Consortium. Paris: OECD.Google Scholar
- Postlethwaite TN & Ross KN 1992 Effective schools in reading. Implications for educational planners. The Hague: IEA.Google Scholar
- Postlethwaite TN & Wiley DE 1991 Science Achievement in Twenty-Three Countries. Oxford: Pergamon Press.Google Scholar
- Shen R & Shen TY 1993 Economic thinking in China: Economic knowledge and attitudes of high school students. Journal of Economic Education, Vol. 24, pp. 70–84.Google Scholar
- Soper JC & Walstad WB 1987 Test of economic literacy. Examiner’s manual 2nd ed. New York: Joint Council on Economic Education (now the National Council on Economic Education).Google Scholar
- Walstad WB & Robson D 1997 Differential item functioning and male-female differences on multiple-choice tests in economics. Journal of Economic Education, Spring, pp. 155–171.Google Scholar
- Walstad WB & Soper JC 1987 A report card on the economic literacy of U.S. High school students. American Economic Review, Vol. 78, pp. 251–256.Google Scholar
- Wang W, Wilson M & Adams RJ 1997 Rasch models for multidimensionality between and within items. In: Wilson M, Engelhard G & Draney K (eds), Objective measurement IV: Theory into practice. Norwood, NJ: Ablex.Google Scholar
- Whitehead DJ & Halil T 1991 Economic literacy in the United Kingdom and the United States: A comparative study. Journal of Economic Education, Spring, pp. 101–110.Google Scholar
- Wu M, Adams RJ & Wilson MR 1996 ConQuest: Generalised Item Response Modelling Software. Draft Version 1. Camberwell: ACER.Google Scholar
- Wu M, Adams RJ & Wilson MR 1997 ConQuest: Generalised Item Response Modelling Software. Camberwell: ACER.Google Scholar