, Volume 44, Issue 4, pp 373–393 | Cite as

Test theory without true scores?

  • Norman Cliff


This paper traces the course of the consequences of viewing test responses as simply providing dichotomous data concerning ordinal relations. It begins by proposing that the score matrix is best considered to be items-plus-persons by items-plus-persons, and recording the wrongs as well as the rights. This shows how an underlying order is defined, and was used to provide the basis for a tailored testing procedure. It also was used to define a number of measures of test consistency. Test items provide person dominance relations, and the relations provided by one item can be in one of three relations with a second one: redundant, contradictory, or unique. Summary statistics concerning the number of relations of each kind are easy to get and provide useful information about the test, information which is related to but different from the usual statistics. These concepts can be extended to form the basis of a test theory which is based on ordinal statistics and frequency counts and which invokes the concept of true scores only in a limited sense.

Key words

test theory consistency true scores ordinal measures tailored testing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Reference notes

  1. 1.
    McCormick, D.Tailor-APL: An interactive computer program for individual tailored testing. (Technical Report No. 5) Los Angeles, CA: University of Southern California, Department of Psychology, 1978.Google Scholar
  2. 2.
    Cliff, N., Cudeck, R., & McCormick, D.Evaluations of implied orders as a basis for tailored testing. (Technical Report No. 4) Los Angeles, CA: University of Southern California, Department of Psychology, 1977.Google Scholar
  3. 3.
    Cliff, N.Psychological scaling. In preparation.Google Scholar
  4. 4.
    Cliff, N., Cudeck, R., & McCormick, D.Implied orders as a basis for tailored testing: Final report. (Technical Report No. 6) Los Angeles, CA: University of Southern California, Department of Psychology, 1978.Google Scholar
  5. 5.
    Reynolds, T. J.The analysis of dominance matrices. Extraction of unidimensional orders within a multidimensional context. (Technical Report No. 3) Los Angeles, CA: University of Southern California, Department of Psychology, 1976.Google Scholar


  1. 6.
    Airasian, P., & Bart, W. Ordering theory: A new and useful measurement model.Educational Psychology, 1973,13, 56–60.Google Scholar
  2. 7.
    Bart, W., & Krus, D. An ordering theoretic method to determine hierarchies among items.Educational and Psychological Measurement, 1973,33, 291–300.Google Scholar
  3. 8.
    Brennan, R. L., & Kane, M. T. Signal/noise ratios for domain referenced tests.Psychometrika, 1977,42, 609–630.CrossRefGoogle Scholar
  4. 9.
    Cliff, N. Scaling.Annual Review of Psychology, 1974,24, 473–506.Google Scholar
  5. 10.
    Cliff, N. Complete orders from incomplete data: Interactive ordering and tailored testing.Psychological Bulletin, 1975,82, 289–302.Google Scholar
  6. 11.
    Cliff, N. A theory of consistency of ordering generalizable to tailored testing.Psychometrika, 1977,42, 375–401.CrossRefGoogle Scholar
  7. 12.
    Cliff, N. What is and isn't measurement? In Gideon Keren (Ed.)Statistical and methodological issues in psychological and social science research. New York: Erlbaum, in press.Google Scholar
  8. 13.
    Coombs, C. A theory of psychological scaling.Engineering Research Bulletin, No. 34. Ann Arbor: University of Michigan Press, 1952.Google Scholar
  9. 14.
    Coombs, C.A theory of data. New York: Wiley, 1964.Google Scholar
  10. 15.
    Cudeck, R., Cliff, N., & Kehoe, J. TAILOR: A FORTRAN procedure for interactive tailored testing.Educational and Psychological Measurement, 1977,37, 767–769.Google Scholar
  11. 16.
    Cudeck, R., McCormick, D., & Cliff, N. Monte Carlo evaluation of implied orders as a basis for tailored testing.Applied Psychological Measurement, 1979,3, 65–74.Google Scholar
  12. 17.
    Ducamp, A., & Falmagne, J. C. Composite measurement.Journal of Mathematical Psychology, 1969,6, 359–390.CrossRefGoogle Scholar
  13. 18.
    Freeman, L. C.Elementary applied statistics. New York: Wiley, 1965.Google Scholar
  14. 19.
    Glaser, R. Instructional technology and the measurement of learning outcomes: Some questions.American Psychologist, 1963,18, 519–521.Google Scholar
  15. 20.
    Guttman, L. The quantification of a class of attributes: A theory and method of scale construction. In P. Horst (Ed.)The prediction of peronsal adjustment. New York: Social Science Research Council, 1941.Google Scholar
  16. 21.
    Hubert, L. A note on Freeman's measure of association for relating an ordered to an unordered factor.Psychometrika, 1974,39, 517–520.Google Scholar
  17. 22.
    Humpheys, L. G. The normal curve and the attenuation paradox in test theory.Psychological Bulletin, 1956,53, 472–476.Google Scholar
  18. 23.
    Keats, J. A.Statistical theory of objective test scores. Melbourne: Australian Council for Educational Research, 1951.Google Scholar
  19. 24.
    Krus, D. Order analysis: An inferential model of dimensional analysis and scaling.Educational and Psychological Measurement, 1977,37, 587–601.Google Scholar
  20. 25.
    Krus, D., & Bart, W. An ordering-theoretic method of multidimensional scaling of items.Educational and Psychological Measurement, 1974, 34, 525–535.Google Scholar
  21. 26.
    Loevinger, J. A. A systematic approach to the construction and evaluation of tests of ability.Psychological Monographs, 1947,61, (4, Whole No. 285).Google Scholar
  22. 27.
    Loevinger, J. A. The technique of homogenous tests compared with some aspects of “scale analysis” and factor analysis.Psychological Bulletin, 1948,45, 507–529.Google Scholar
  23. 28.
    Loevinger, J. The attenuation paradox in test theory.Psychological Bulletin, 1954,51, 493–504.PubMedGoogle Scholar
  24. 29.
    McCormick, D., & Cliff, N. TAILOR-APL: An interactive computer program for individual tailored testing.Educational and Psychological Measurement, 1977,37, 771–774.Google Scholar
  25. 30.
    McNemar, Q.Psychological statistics. New York: Wiley, 1949.Google Scholar
  26. 31.
    Reynolds, T. J. The logical fallacy of order analysis.Multivariate Behavioral Research, in press.Google Scholar
  27. 32.
    Sato, T.S-P Table Analysis-Analysis and interpretation of test scores. Tokyo: Meiji-Tosho Publishing, 1975.Google Scholar
  28. 33.
    Sato, T., & Kurata, M. Basic S-P score table characteristics.Nippon Electric Co. Research & Development, 1977 (Whole No. 47), 64–71.Google Scholar
  29. 34.
    Schulman, R. S. Correlation and prediction in ordinal test theory.Psychometrika, 1976,41, 19–29.CrossRefGoogle Scholar
  30. 35.
    Schulman, R. S. Individual distributions under ordinal measurement.Psychometrika, 1978,43, 19–29.CrossRefGoogle Scholar
  31. 36.
    Schulman, R. S., & Haden, R. S. A test theory model for ordinal measurements.Psychometrika, 1975,40, 455–472.CrossRefGoogle Scholar
  32. 37.
    Wherry, R. J., & Gaylord, R. H. Factor pattern of test items and tests as a function of the correlation coefficient: Content, validity, and constant error factors.Psychometrika, 1944,9, 237–244.Google Scholar

Copyright information

© The Psychometric Society 1979

Authors and Affiliations

  • Norman Cliff
    • 1
  1. 1.Department of PsychologyUniversity of Southern CaliforniaLos Angeles

Personalised recommendations