, Volume 36, Issue 3, pp 261–288 | Cite as

Bayesian inference and the classical test theory model: Reliability and true scores

  • Melvin R. Novick
  • Paul H. Jackson
  • Dorothy T. Thayer


A general one-way analysis of variance components with unequal replication numbers is used to provide unbiased estimates of the true and error score variance of classical test theory. The inadequacy of the ANOVA theory is noted and the foundations for a Bayesian approach are detailed. The choice of prior distribution is discussed and a justification for the Tiao-Tan prior is found in the particular context of the “n-split” technique. The posterior distributions of reliability, error score variance, observed score variance and true score variance are presented with some extensions of the original work of Tiao and Tan. Special attention is given to simple approximations that are available in important cases and also to the problems that arise when the ANOVA estimate of true score variance is negative. Bayesian methods derived by Box and Tiao and by Lindley are studied numerically in relation to the problem of estimating true score. Each is found to be useful and the advantages and disadvantages of each are discussed and related to the classical test-theoretic methods. Finally, some general relationships between Bayesian inference and classical test theory are discussed.


Variance Component Posterior Distribution Prior Distribution Bayesian Inference Bayesian Approach 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Box, G. E. P. and Tiao, G. C. (1964a). A Bayesian approach to the importance of assumptions applied to the comparison of variances.Biometrika,51, 153–167.Google Scholar
  2. Box, G. E. P. and Tiao, G. C. (1964b). A note on criterion robustness and inference robustness.Biometrika,51, 169–173.Google Scholar
  3. Box, G. E. P. and Tiao, G. C. (1968). Bayesian estimation of means for the random effect model.J. Amer. Statist. Assoc.,63, 174–181.Google Scholar
  4. Davies, O. L. et al. (1961).Statistical Methods in Research and Production, 3rd edition. Edinburgh: Oliver and Boyd.Google Scholar
  5. Hill, B. M. (1965). Inference about variance components in the one-way model.J. Amer. Statist. Assoc.,60, 806–825.Google Scholar
  6. Hill, B. M. (1967). Correlated errors in the random model.J. Amer. Statist. Assoc.,62, 1385.Google Scholar
  7. James, W. and Stein, C. (1961). Estimation with quadratic loss. In J. Neyman (Ed.),Proceedings of the Fourth Berkeley Symposium on Probability and Statistics. Vol. I. Berkeley: University of California Press.Google Scholar
  8. Jeffreys, H. (1961).Theory of Probability. 3rd edition. Oxford: The Clarendon Press.Google Scholar
  9. Kelley, T. L. (1927).Fundamentals of Statistics. Cambridge: Harvard University Press.Google Scholar
  10. Klotz, J. H., Milton, R. C., and Zacks, S. (1969). Mean square efficiency of estimators of variance components.J. Amer. Stat. Assoc.,64, 1383–1402.Google Scholar
  11. Kristof, W. (1963). The statistical theory of stepped-up reliability coefficients when a test has been divided into several equivalent parts.Psychometrika,28, 221–238.Google Scholar
  12. Kristof, W. (1969b). Estimation of true score and error variance for tests under various equivalence assumptions.Psychometrika,34, 489–508.CrossRefGoogle Scholar
  13. Lindley, D. V. (1965).Introduction to Probability and Statistics, Part 2. Cambridge: University Press.Google Scholar
  14. Novick, M. R. (1969). Multiparameter Bayesian indifference procedures.J. Royal Statist. Soc.: Series B,31, 29–64 (with discussion).Google Scholar
  15. Pearson, K. et al. (1968).Tables of the Incomplete Beta-Function. Cambridge: University Press.Google Scholar
  16. Stein, C. M. (1962). Confidence sets for the mean of a multivariate normal distribution.J. Royal Statist. Soc.: Series B,24, 265–296.Google Scholar
  17. Tiao, G. C. and Tan, W. Y. (1965). Bayesian analysis of random-effect models in the analysis of variance. I. Posterior distribution of variance-components.Biometrika,52, 37–53.Google Scholar

Copyright information

© Psychometric Society 1971

Authors and Affiliations

  • Melvin R. Novick
    • 1
  • Paul H. Jackson
    • 1
  • Dorothy T. Thayer
    • 1
  1. 1.Educational Testing ServiceUSA

Personalised recommendations