Collaborative Quality Filtering: Establishing Consensus or Recovering Ground Truth?

  • Jonathan Traupman
  • Robert Wilensky
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3932)


We present a algorithm based on factor analysis for performing collaborative quality filtering (CQF). Unlike previous approaches to CQF, which estimate the consensus opinion of a group of reviewers, our algorithm uses a generative model of the review process to estimate the latent intrinsic quality of the items under reviews. We run several tests that demonstrate that consensus and intrinsic quality are, in fact different and unrelated aspects of quality. These results suggest that asymptotic consensus, which purports to model peer review, is, in fact, not recovering the ground truth quality of reviewed items.


Ground Truth Recommender System Synthetic Dataset Collaborative Filter Consensus Opinion 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Internet Movie Database: (Top 250 films),
  2. 2.
    Riggs, T.: Collaborative quality filtering in open review systems. Master’s thesis, University of California, Berkeley, Computer Science Division (2001)Google Scholar
  3. 3.
    Wilensky, R., Riggs, T.: An algorithm for automatically rating reviewers. In: Proc. of the First Joint Conf. on Digital Libraries, Roanoke, Virginia (2001)Google Scholar
  4. 4.
    Katerattanakui, P., et al.: Objective quality ranking of computing journals. Communications of the ACM 46, 111–114 (2003)CrossRefGoogle Scholar
  5. 5.
    Brown, L., Gardner, J.: Using citation analysis to assess the impact of journals and articles on contemporary accounting research. Journal of Accounting Research 23, 84–109 (1995)CrossRefGoogle Scholar
  6. 6.
    Goldberg, D., et al.: Using collaborative filtering to weave and information tapestry. Communications of the ACM 35, 61–70 (1992)CrossRefGoogle Scholar
  7. 7.
    Resnick, P., et al.: GroupLens: An Open Architecture for Collaborative Filtering of Netnews. In: Proc. of ACM 1994 Conf. on Computer Supported Cooperative Work, Chapel Hill, North Carolina, pp. 175–186. Chapel Hill, North Carolina, ACM (1994)Google Scholar
  8. 8.
    Hill, W., et al.: Recommending and evaluating choices in a virtual community of use. In: Proc. of the SIGCHI Conf. on Humand Factors in Computing Systems, pp. 194–201 (1995)Google Scholar
  9. 9.
    Shardanand, U., Maes, P.: Social information filtering: Algorithms for automating “word of mouth”. In: Proc. of ACM CHI 1995 Conf. on Human Factors in Computing Systems, vol. 1, pp. 210–217 (1995)Google Scholar
  10. 10.
    Herlocker, J.L., et al.: An algorithmic framework for performing collaborative filtering. In: SIGIR 1999: Proc. of the 22nd Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, Berkeley, CA, USA, August 15-19, 1999, pp. 230–237. ACM, New York (1999)CrossRefGoogle Scholar
  11. 11.
    Hofmann, T.: Latent semantic models for collaborative filtering. ACM Transactions on Information Systems 22, 89–115 (2004)CrossRefGoogle Scholar
  12. 12.
    Canny, J.: Collaborative filtering with privacy via factor analysis. In: Proc. of the 25th Annual International ACM SIGIR Conf. on Research and Development in Information Retrieval, pp. 238–245 (2002)Google Scholar
  13. 13.
    Resnick, P., Varian, H.R.: Recommender systems. Communications of the ACM (1997)Google Scholar
  14. 14.
    Konstan, J.A.: Introduction to recommender systems: Algorithms and evaluation. ACM Transactions on Information Systems 22, 1–4 (2004)CrossRefGoogle Scholar
  15. 15.
    Middleton, S.E., et al.: Ontological user profiling in recommender systems. ACM Transactions on Information Systems 22, 54–88 (2004)CrossRefGoogle Scholar
  16. 16.
    Zacharia, G., et al.: Collaborative reputation mechanisms in electronic marketplaces. In: HICSS (1999)Google Scholar
  17. 17.
    Rubin, D.B., Thayer, D.T.: EM algorithms for ML factor analysis. Psychometrika 47, 69–76 (1982)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Jordan, M.I.: An introduction to probabilistic graphic models (unpublished textbook manuscript)Google Scholar
  19. 19.
    Ghahramani, Z., Jordan, M.I.: Learning from incomplete data. Technical Report AIM-1509, Massachusettes Institute of Technology, Artificial Intelligence Laboratory (1994)Google Scholar
  20. 20.
    Jelinek, F., Mercer, R.L.: Interpolated estimation of markov source parameters from sparse data. In: Gelsema, E.S., Kanal, L.N. (eds.) Pattern Recognition in Practice, pp. 381–397. North-Holland Publishing Company, Amsterdam (1980)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Jonathan Traupman
    • 1
  • Robert Wilensky
    • 1
  1. 1.Computer Science DivisionUniversity of CaliforniaBerkeleyUSA

Personalised recommendations