, Volume 68, Issue 3, pp 453–471 | Cite as

Using the conditional grade-of-membership model to assess judgment accuracy

  • Bruce CooilEmail author
  • Sajeev Varki


Consider the case whereJ instruments are used to classify each ofI objects relative toK nominal categories. The conditional grade-of-membership (GoM) model provides a method of estimating the classification probabilities of each instrument (or “judge”) when the objects being classified consist of both pure types that lie exclusively in one ofK nominal categories, and mixtures that lie in more than one category. Classification probabilities are identifiable whenever the sample of GoM vectors includes pure types from each category. When additional, relatively mild, assumptions are made about judgment accuracy, the identifiable correct classification probabilities are the greatest lower bounds among all solutions that might correspond to the observed multinomial process, even when the unobserved GoM vectors do not include pure types from each category. Estimation using the conditional GoM model is illustrated on a simulated data set. Further simulations show that the estimates of the classification probabilities are relatively accurate, even when the sample contains only a small percentage of approximately pure objects.

Key words

nominal classification incidental parameters extreme profiles mixtures 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aptech Systems. (1995).Constrained maximum likelihood. Kent, WA: Author.Google Scholar
  2. Batchelder, W.H., & Romney, A.K. (1986). The statistical analysis of a general Condorcet model for dichotomous choice situations. In B. Grofman & G. Owen (Eds.),Information pooling and group decision making (pp. 103–112). Greenwich, CN: JAI Press.Google Scholar
  3. Batchelder, W.H., & Romney, A.K. (1988). Test theory without an answer key.Psychometrika, 53, 193–224.Google Scholar
  4. Batchelder, W.H., & Romney, A.K. (1989). New results in test theory without an answer key. In Edward E. Roskam (Ed.),Mathematical psychology in progress (pp. 229–248). Berlin, Heidelberg, New York: Springer-Verlag.Google Scholar
  5. Berkman, L., Singer, B., & Manton, K.G. (1989). Black/white differences in health status and mortality among the elderly.Demography, 26, 661–678.Google Scholar
  6. Blazer, D., Woodbury, M.A., Hughes, D., George, L.K., Manton, K.G., Bachar, J.R., & Fowler, N. (1989). A statistical analysis of the classification of depression in a mixed community and clinical sample.Journal of Affective Disorders, 16, 11–20.Google Scholar
  7. Chavez, J.M., & Buriel, R. (1988). Mother-child interactions involving a child with epilepsy: A comparison of immigrant and native-born Mexican Americans.Journal of Pediatric Psychology, 13, 349–361.Google Scholar
  8. Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37–46.Google Scholar
  9. Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit.Psychological Bulletin, 70, 213–220.Google Scholar
  10. Cooil, B., & Rust, R.T. (1995). General estimators for the reliability of qualitative data.Psychometrika, 60, 199–220.Google Scholar
  11. Dillon, W.R., & Mulani, N. (1984). A probabilistic latent class model for assessing inter-judge reliability.Multivariate Behavioral Research, 19, 438–458.Google Scholar
  12. Holland, J.L. (1985).Making vocational choices: A theory of vocational personalities and work environments. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
  13. Kiefer, J., & Wolfowitz, J. (1956). Consistency of the Maximum likelihood estimator in the presence of infinitely many incidental parameters.Annals of Mathematical Statistics, 27, 887–906.Google Scholar
  14. Klauer, C.K., & Batchelder, W.H. (1996). Structural analysis of subjective categorical data.Psychometrika, 61, 199–240.Google Scholar
  15. Manton, K.G., Woodbury, M.A., & Tolley, H.D. (1994).Statistical applications using fuzzy sets. New York, NY: John Wiley & Sons.Google Scholar
  16. Perreault, W.D., Jr., & Leigh, L.E. (1989). Reliability of nominal data based on qualitative judgments.Journal of Marketing Research, 26, 135–148.Google Scholar
  17. Richards, F.S.G. (1961). A method of maximum-likelihood estimation.Journal of the Royal Statistical Society, Series B, 23, 469–476.Google Scholar
  18. Tolley, H.D., & Manton, K.G. (1992). Large sample properties of estimates of a discrete grade of membership model.Annals of the Institute of Statistical Mathematics, 44, 85–95.Google Scholar
  19. Tsai, C.Y., & Denton, J.J. (1993). Reliability assessment of a classroom observation system.Journal of Classroom Interaction, 28, 23–32.Google Scholar
  20. Varki, S. (1996).New strategies and methodologies in customer satisfaction. Unpublished doctoral dissertation, Vanderbilt University, Nashville, TN.Google Scholar
  21. Varki, S., Cooil, B., & Rust, R.T. (2000). Modeling fuzzy data in qualitative marketing research.Journal of Marketing Research, 37, 480–489.Google Scholar
  22. Vertrees, J., & Manton, K.G. (1986). A multivariate approach for classifying hospitals and computing blended payment rates.Medical Care, 24, 283–300.Google Scholar
  23. Woodbury, M.A., & Clive, J. (1974). Clinical pure types as a fuzzy partition.Journal of Cybernetics, 4, 111–121.Google Scholar
  24. Woodbury, M.A., & Manton, K.G. (1982). A new procedure for analysis of medical classification.Methods of Information in Medicine, 21, 210–220.Google Scholar
  25. Woodbury, M.A., Manton, K.G., & Tolley, H.D. (1994). A general model for statistical analysis using fuzzy sets: Sufficient conditions for identifiability and statistical properties.Information Sciences, 1, 149–180.Google Scholar
  26. Yale, L., & Gilly, M.C. (1988). Trends in advertising research: A look at the content of marketing-oriented journals from 1967 to 1985.Journal of Advertising, 17, 12–22.Google Scholar

Copyright information

© The Psychometric Society 2003

Authors and Affiliations

  1. 1.OGSM, Vanderbilt UniversityNashville
  2. 2.College of Business AdministrationUniversity of Rhode IslandUSA

Personalised recommendations