Analyzing User Demographics and User Behavior for Trust Assessment

  • Davide CeolinEmail author
  • Paul Groth
  • Archana Nottamkandath
  • Wan Fokkink
  • Willem Robert van Hage
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8816)


In many systems, the determination of trust is reduced to reputation estimation. However, reputation is just one way of determining trust. The estimation of trust can be tackled from a variety of other perspectives. In this chapter, we model trust relying on user reputation, user demographics and from provenance. We then explore the effects of combining trust computed through these different methods. Concretely, the first contribution of this chapter is a study of the correlations of demographics with trust. This study helps us to understand which categories of users are better candidates for annotation tasks in the cultural heritage domain. Secondly, we detail a procedure for computing reputation-based trust assessments. The user reputation is modeled in subjective logic based on the user’s performance in the system evaluated (Waisda? in the case of the work presented here). The third contribution is a procedure for computing trust values based on provenance information, represented using the W3C PROV model. We show how merging the results of these procedures can be beneficial for the reliability of the estimated trust value. We evaluate the proposed procedures and their merger by estimating and verifying the trustworthiness of the tags created within the Waisda? video tagging game from the Netherlands Institute for Sound and Vision. Through a quantitative analysis of the results, we demonstrate that using provenance and demographic information is beneficial for the accuracy of trust assessments.


Trust Provenance Subjective logic Machine learning Uncertainty reasoning Tags 



We thank the Netherlands Institute for Sound and Vision for launching and guiding the Waisda? project, and our colleagues Michiel, Riste and Valentina for their support. This research was partially supported by the PrestoPRIME project, in the EC ICT FP7 program, and by the Data2Semantics and SEALINC Media projects in the Dutch national program COMMIT.


  1. 1.
    Artz, D., Gil, Y.: A survey of trust in computer science and the semantic web. J. Semant. Web 5(2), 131–197 (2007)CrossRefGoogle Scholar
  2. 2.
    Bizer, C., Cyganiak, R.: Quality-driven information filtering using the WIQA policy framework. J. Web Semant. 7(1), 1–10 (2009)CrossRefGoogle Scholar
  3. 3.
    Card, S., Moran, T.P., Newell, A.: The Psychology of Human Computer Interaction. Lawrence Erlbaum Associates, Hillsdale (1983)Google Scholar
  4. 4.
    Carroll, J., Bizer, C., Hayes, P., Stickler, P.: Named graphs, provenance and trust. In: WWW ’05, pp. 613–622. ACM (2005)Google Scholar
  5. 5.
    Burnett, C., Norman, T.J., Sycara, K.: Bootstrapping trust evaluations through stereotypes. In: AAMAS, IFAAMAS, pp. 241–248 (2010)Google Scholar
  6. 6.
    Ceolin, D., Groth, P., Hage, W.R.V.: Calculating the trust of event descriptions using provenance. In: SWPM 2010, pp. 7–12. CEUR-WS (2010)Google Scholar
  7. 7.
    Ceolin, D., Groth, P., van Hage, W.R., Nottamkandath, A., Fokkink, W.: Trust Evaluation through user reputation and provenance analysis. In: URSW, pp. 15–26. (2012)Google Scholar
  8. 8.
    Ceolin, D., Nottamkandath, A., Fokkink, W.: Automated evaluation of annotators for museum collections using subjective logic. In: Dimitrakos, T., Moona, R., Patel, D., McKnight, D.H. (eds.) IFIPTM 2012. IFIP AICT, vol. 374, pp. 232–239. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Ceolin, D., Nottamkandath, A., Fokkink, W.: Semi-automated assessment of annotation trustworthiness. In: PST 2013, pp. 325–332. IEEE Computer Society, July 2013Google Scholar
  10. 10.
    Ceolin, D., Nottamkandath, A., Fokkink, W.: Efficient semi-automated assessment of annotation trustworthiness. J. Trust Manag. 1, 1–31 (2014)CrossRefGoogle Scholar
  11. 11.
    Ceolin, D., Nottamkandath, A., Fokkink, W.J.: Subjective logic extensions for the semantic web. In: URSW, pp. 27–38. (2012)Google Scholar
  12. 12.
    Cortes, C., Vapnik, V.: Support-vector networks. M. Learn. 20, 273–297 (1995)zbMATHGoogle Scholar
  13. 13.
    Dan Brickley, L.M.: FOAF, Jan 2014.
  14. 14.
    Gambetta, D.: Can We Trust Trust?. Basil Blackwell, New York (1988)Google Scholar
  15. 15.
    Glass, G.V., Hopkins, K.D.: Statistical Methods in Education and Psychology. Allyn & Bacon, Boston (1995)Google Scholar
  16. 16.
    Golbeck, J.: Trust on the world wide web: a survey. Found. Trends Web Sci. 1(2), 131–197 (2006)CrossRefGoogle Scholar
  17. 17.
    Hartig, O., Zhao, J.: Using web data provenance for quality assessment. In: SWPM 2009, pp. 26–31. CEUR-WS (2009)Google Scholar
  18. 18.
    Javanmardi, S., Lopes, C., Baldi, P.: Modeling user reputation in wikis. Stat. Anal. Data Min. 3(2), 126–139 (2010)MathSciNetGoogle Scholar
  19. 19.
    Jøsang, A.: A logic for uncertain probabilities. Int. J. Uncertainty Fuzziness Knowl. Based Syst. 9(3), 279–311 (2001)CrossRefGoogle Scholar
  20. 20.
    Kazai, G., Kamps, J., Milic-Frayling, N.: The face of quality in crowdsourcing relevance labels: demographics, personality and labeling accuracy. In: CIKM, pp. 2583–2586. ACM (2012)Google Scholar
  21. 21.
    Kononenko, I.: Naive bayesian classifier and continuous attributes. Informatica 16(1), 1–8 (1992)MathSciNetGoogle Scholar
  22. 22.
    Liu, X., Datta, A., Rzadca, K., Lim, E.-P.: StereoTrust: a group based personalized trust model. In: CIKM, pp. 7–16. ACM (2009)Google Scholar
  23. 23.
    Masum, H., Tovey, M. (eds.): The Reputation Society. MIT Press, Cambridge (2012)Google Scholar
  24. 24.
    Netherlands Inst. for Sound and Vision. Waisda? June 2012.
  25. 25.
    O’Hara, K.: A General Definition of Trust. Technical report, University of Southampton (2012)Google Scholar
  26. 26.
    Pantola, A.V., Pancho-Festin, S., Salvador, F.: Rating the raters: a reputation system for wiki-like domains. In: SIN ’10, pp. 71–80. ACM (2010)Google Scholar
  27. 27.
    Pearson, K.: Mathematical contributions to the theory of evolution. In: Proceedings of the Royal Society of London, pp. 489–498 (1896)Google Scholar
  28. 28.
    Pearson, K.: On the criterion that a given system of deviations from the probable in the case of correlated system of variables is such that it can be reasonably supposed to have arisen from random sampling. Phil. Mag. 50, 157–175 (1900)CrossRefzbMATHGoogle Scholar
  29. 29.
    Rajbhandari, S., Rana, O.F., Wootten, I.: A fuzzy model for calculating workflow trust using provenance data. In: MG’08, pp. 1–8. ACM (2008)Google Scholar
  30. 30.
    Rajbhandari, S., Wootten, I., Ali, A.S., Rana, O.F.: Evaluating provenance-based trust for scientific workflows. In: CCGRID 06, pp. 365–372. IEEE (2006)Google Scholar
  31. 31.
    Sabater, J., Sierra, C.: Review on computational trust and reputation models. Artif. Intell. Rev. 24, 33–60 (2005)CrossRefzbMATHGoogle Scholar
  32. 32.
    U.S. Institute of Museum and Library Service. Steve Social Tagging Project, June 2012.
  33. 33.
    Venanzi, M., Guiver, J., Kazai, G., Kohli, P., Shokouhi, M.: Community-based bayesian aggregation models for crowdsourcing. In: WWW, pp. 155–164 (2014)Google Scholar
  34. 34.
    W3C. PROV-O, June 2012.
  35. 35.
    W3C. Resource description framework (rdf): concepts and abstract data model, June 2012.
  36. 36.
    Wien, T.: e1071: Misc functions of the department of statistics (e1071), June 2012.
  37. 37.
    Wilcoxon, F.: Individual comparisons by ranking methods. Biom. Bull. 1, 80–83 (1945)CrossRefGoogle Scholar
  38. 38.
    Zaihrayeu, I., da Silva, P.P., McGuinness, D.L.: IWTrust: improving user trust in answers from the web. In: Herrmann, P., Issarny, V., Shiu, S.C.K. (eds.) iTrust 2005. LNCS, vol. 3477, pp. 384–392. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Davide Ceolin
    • 1
    Email author
  • Paul Groth
    • 1
  • Archana Nottamkandath
    • 1
  • Wan Fokkink
    • 1
  • Willem Robert van Hage
    • 2
  1. 1.VU UniversityAmsterdamThe Netherlands
  2. 2.Synerscope B.V.EindhovenThe Netherlands

Personalised recommendations