Abstract
A test theory using only ordinal assumptions is presented. It is based on the idea that the test items are a sample from a universe of items. The sum across items of the ordinal relations for a pair of persons on the universe items is analogous to a true score. Using concepts from ordinal multiple regression, it is possible to estimate the tau correlations of test items with the universe order from the taus among the test items. These in turn permit the estimation of the tau of total score with the universe. It is also possible to estimate the odds that the direction of a given observed score difference is the same as that of the true score difference. The estimates of the correlations between items and universe and between total score and universe are found to agree well with the actual values in both real and artificial data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Birnbaum, A. (1968). Some latent trait models and their uses in inferring an examinee's ability. In F. M. Lord & M. R. Novick (Eds.),Statistical theories of mental test scores (pp. 397–472). Reading, MA: Addison-Wesley.
Charlin, V. L. (1987).Implementation and inferential issues of a multiple regression system with ordinal variables. Unpublished doctoral dissertation, University of Southern California.
Cliff, N. (1977). A theory of consistency of ordering generalizable to tailored testing.Psychometrika, 42, 375–401.
Cliff, N. (1979). Test theory without true scores?Psychometrika, 44, 373–393.
Cliff, N. (1986). Ordinal analogues of multiple regression.1986 Proceedings of the Social Statistics Section of the American Statistical Association. (pp. 320–324). Washington, DC: American Statistical Association.
Cliff, N. (1989a). “Multiple regression” with ordinal variables. Unpublished manuscript.
Cliff, N. (1989b). Ordinal consistency and ordinal true scores.Psychometrika, 54, 75–91.
Cliff, N., & Charlin, V. L. (In press). Variances and covariances of Kendall's tau and their estimation.Multivariate Behavior Research.
Donoghue, J. R. (1990).Properties of ability estimates derived from logistic item response theory and ordinal item sampling test theory. Unpublished doctoral dissertation, University of Southern California.
Donoghue, J. R., & Cliff, N. (In press). An investigation of ordinal true score test theory.Applied Psychological Measurement.
Douglas, G. A., & Wright, B. D. (1986).The two-category model for objective measurement (Research Memorandum No. 34). Chicago: University of Chicago, Department of Education, MESA Psychometric Laboratory.
Gulliksen, H. (1950).Theory of mental tests. New York: Wiley.
Guttman, L. (1944). A basis for scaling qualitative data.American Sociological Review, 9, 139–150.
Guttman, L. (1950). The basis for scalogram analysis. In S.A. Stouffer (Ed.),Measurement and Prediction (pp. 60–90). Princeton, NJ: Princeton University.
Hawkes, R. J. (1971). The multivariate analysis of ordinal data.American Journal of Sociology, 76, 908–926.
Hettmansperger, T. P. (1984).Statistical inference based on ranks. New York: Wiley.
Holland, P. W. (1990). The Dutch identity: A new tool for understanding item response theory.Psychometrika, 55, 5–18.
Kendall, M. G. (1970)Rank correlation methods (3rd ed.). New York: Hafner.
Loevinger, J. A. (1947). A systematic approach to the construction and evaluation of tests of ability.Psychological Monographs, 61, (Whole No. 285).
Loevinger, J. A. (1948). A technique of homogeneous test evaluation compared with some aspects of “scale analysis”.Psychological Bulletin, 45, 507–529.
Lord, F. M. (1968). An analysis of the verbal Scholastic Aptitude Test using Birnbaum's three-parameter logistic model.Educational and Psychological Measurement, 28, 989–1020.
Lord, F. M. (1975). The “ability” scale in item response theory.Psychometrika, 40, 205–217.
Lord, F. M., & Novick, M. R. (1968).Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
Mislevy, R. J. (1987). Recent developments in item response theory with implications for teacher certification.Review of Educational Research, 14, 239–275.
Mokken, R. J. (1971).A theory and procedure of scale analysis. Hawthorne, NY: Mouton & Co.
Mokken, R. J. & Lewis, C. (1982). A nonparametric approach to the analysis of dichotomous item responses.Applied Psychological Measurement, 6, 417–430.
Reckase, M. D. (1985). The difficulty of test items that measure more than one ability.Applied Psychological Measurement, 9, 401–412.
Reynolds, T. J., & Sutrick, K. H. (1986). Assessing the correspondence of one or more vectors to a symmetric matrix.Psychometrika, 51, 101–112.
Roskam, E. E., & Jansen, P. G. W. (1984). A new derivation of the Rasch Model. In E. deGreef & J. Buggenhaut (Eds.),Trends in mathematical psychology (pp. 293–307). Amsterdam: Elsevier.
Samejima, F. (1979).A new family of models for the multiple-choice item (ONR Research Report 79-4). Knoxville, TN: University of Tennessee.
Schulman, R. S., & Haden, R. L. (1975). A test theory for ordinal measurements.Psychometrika, 40, 329–340.
Smith, R. B. (1974). Continuities in ordinal path analysis.Social Forces, 53, 200–229.
Somers, R. H. (1974). Analysis of partial rank correlation measures based on the product-moment model: Part One.Social Forces, 53, 229–246.
Stout, W. (1987). A nonparametric approach for assessing latent dimensionality.Psychometrika, 52, 589–617.
Stout, W. (1988).A nonparametric multidimensional IRT approach with applications to ability estimation and test bias (ONR Research Report 88-1). Champaign, IL: University of Illinois.
Stout, W. (1990). A new item-response theory modeling approach with applications to unidimensionality assessment and ability estimation.Psychometrika, 55, 293–325.
Thissen, D., & Steinberg, L. (1984). A response model for multiple choice items.Psychometrika, 47, 501–519.
Thissen, D., & Wainer, H. (1982). Some standard errors in item response theory.Psychometrika, 45, 397–412.
Yen, W. (1986). The choice of scale for educational measurement: An IRT perspective.Journal of Educational Measurement, 23, 299–325.
Author information
Authors and Affiliations
Additional information
Part of this paper was presented at the June, 1989, Meeting of the Psychometric Society. The authors wish to thank several reviewers for their suggestions. This research was mainly done while the second author was a University Fellow at the University of Southern California.
Rights and permissions
About this article
Cite this article
Cliff, N., Donoghue, J.R. Ordinal test fidelity estimated by an item sampling model. Psychometrika 57, 217–236 (1992). https://doi.org/10.1007/BF02294506
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02294506