Skip to main content

Advertisement

SpringerLink
Log in

Automatic evaluation of information provider reliability and expertise

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Q&A social media have gained a lot of attention during the recent years. People rely on these sites to obtain information due to a number of advantages they offer as compared to conventional sources of knowledge (e.g., asynchronous and convenient access). However, for the same question one may find highly contradicting answers, causing an ambiguity with respect to the correct information. This can be attributed to the presence of unreliable and/or non-expert users. These two attributes (reliability and expertise) significantly affect the quality of the answer/information provided. We present a novel approach for estimating these user’s characteristics relying on human cognitive traits. In brief, we propose each user to monitor the activity of his peers (on the basis of responses to questions asked by him) and observe their compliance with predefined cognitive models. These observations lead to local assessments that can be further fused to obtain a reliability and expertise consensus for every other user in the social network (SN). For the aggregation part we use subjective logic. To the best of our knowledge this is the first study of this kind in the context of Q&A SNs. Our proposed approach is highly distributed; each user can individually estimate the expertise and the reliability of his peers using his direct interactions with them and our framework. The online SN (OSN), which can be considered as a distributed database, performs continuous data aggregation for users expertise and reliability assesment in order to reach a consensus. In our evaluations, we first emulate a Q&A SN to examine various performance aspects of our algorithm (e.g., convergence time, responsiveness etc.). Our evaluations indicate that it can accurately assess the reliability and the expertise of a user with a small number of samples and can successfully react to the latter’s behavior change, provided that the cognitive traits hold in practice. Furthermore, the use of the consensus operator for the aggregation of multiple opinions on a specific user, reduces the uncertainty with regards to the final assessment. However, as real data obtained from Yahoo! Answers imply, the pairwise interactions between specific users are limited. Hence, we consider the aggregate set of questions as posted from the system itself and we assess the expertise and realibility of users based on their response behavior. We observe, that users have different behaviors depending on the level at which we are observing them. In particular, while their activity is focused on a few general categories, yielding them reliable, their microscopic (within general category) activity is highly scattered.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Aberer, K., Despotovic, Z.: Managing trust in a peer-2-peer information system. In: ACM CIKM (2001)

  2. Ackerman, M.S., McDonald, D.W.: Answer garden 2: merging organizational memory with collaborative help. In: CSCW (1996)

  3. Adamic, L.A., Zhang, J., Bakshy, E., Ackerman, M.S.: Knowledge sharing and yahoo answers: everyone knows something. In: Proceeding of the 17th International Conference on World Wide Web (WWW ’08), pp. 665–674. ACM, New York (2008)

    Chapter  Google Scholar 

  4. Agichtein, E., Castillo, C., Donato, D., Gionis, A., Mishne, G.: Finding high-quality content in social media. In: Proceedings of the International Conference on Web Search and Web Data Mining (WSDM ’08), pp. 183–194. ACM, New York (2008)

    Chapter  Google Scholar 

  5. Bian, J., Liu, Y., Agichtein, E., Zha, H.: Finding the right facts in the crowd: factoid question answering over social media. In: Proceeding of the 17th International Conference on World Wide Web (WWW ’08), pp. 467–476. ACM, New York (2008)

    Chapter  Google Scholar 

  6. Bian, J., Liu, Y., Zhou, D., Agichtein, E., Zha, H.: Learning to recognize reliable users and content in social media with coupled mutual reinforcement. In: Proceedings of the 18th International Conference on World Wide Web (WWW ’09), pp. 51–60. ACM, New York (2009)

    Chapter  Google Scholar 

  7. Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: CUAI (1998)

  8. Bouguessa, M., Dumoulin, B., Wang, S.: Identifying authoritative actors in question-answering forums: the case of Yahoo! Answers. In: Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD ’08), pp. 866–874. ACM, New York (2008)

    Chapter  Google Scholar 

  9. Buchegger, S., Le Boudec, J.-Y.: A robust reputation system for p2p and mobile ad-hoc networks. In: P2PEcon (2004)

  10. Cornelli, F., Damiani, E., Vimercati, S.D.C.D., Paraboschi, S., Samarati, S.: Choosing reputable servents in a p2p network. In: WWW (2002)

  11. Dom, B., Eiron, I., Cozzi, A., Zhang, Y.: Graph-based ranking algorithms for e-mail expertise analysis. In: DMKD (2003)

  12. Eysenbach, G., Kohler, C.: How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br. Med. J. 324, 573–577 (2002)

    Article  Google Scholar 

  13. Foner, L.N.: Yenta: a multi-agent, referral-based matchmaking system. In: Agents (1997)

  14. Ganeriwal, S., Srivastava, M.: Reputation-based framework for high integrity sensor networks. In: SASN (2004)

  15. Golbeck, J., Fleischmann, K.R.: Trust in social Q&A: the impact of text and photo cues of expertise. In: Proceedings of the American Society for Information Science and Technology (ASIS&T), pp. 1–10. Wiley Subscription Services (2010)

  16. Guo, J., Xu, S., Bao, S., Yu, Y.: Tapping on the potential of Q&A community by recommending answer providers. In: Proceeding of the 17th ACM Conference on Information and Knowledge Management (CIKM ’08), pp. 921–930. ACM, New York (2008)

    Chapter  Google Scholar 

  17. Hang, C.W., Wang, Y., Singh, M.P.: Operators for propagating trust and their evaluation in social networks. In: AAMAS (2009)

  18. Huynh, T.D., Jennings, N.R., Shadbolt, N.R.: An integrated trust and reputation model for open multi-agent systems. Journal of Autonomous Agents and MultiAgent Systems 13, 119–154 (2006)

    Google Scholar 

  19. John, A., Seligmann, D.: Collaborative tagging and expertise in the enterprise. In: WWW (2006)

  20. John, B.M., Chua, A.Y.K., Goh, D.H.L.: What makes a high-quality user-generated answer? IEEE Internet Computing 15(1), 66–71 (2011)

    Article  Google Scholar 

  21. Josang, A.: Artificial reasoning with subjective logic. In: Second Australian Workshop on Commonsense Reasoning (1997)

  22. Josang, A.: A logic for uncertain probabilities. Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 9(3), 279–311 (2001)

    Article  MathSciNet  Google Scholar 

  23. Jurczyk, P., Agichtein, E.: Discovering authorities in question answer communities by using link analysis. In: Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM ’07), pp. 919–922. ACM, New York (2007)

    Chapter  Google Scholar 

  24. Kamvar, S.D., Schlosser, M.T., Garcia-Molina, H.: The eigentrust algorithm for reputation management in p2p networks. In: WWW (2003)

  25. Kasneci, G., Van Gael, J., Stern, D., Graepel, T.: Cobayes: Baysian knowledge corroboration with assessors of unknown areas of expertise. In: WSDM (2011)

  26. Kautz, H., Milewski, A., Selman, B.: Agent amplified communication. In: National Conference of Artificial Intelligence (1996)

  27. Kautz, H., Selman, B., Shah, M.: Referralweb: combining social networks and collaborative filtering. ACM Commun. 40(3) (1997)

  28. Kotter, J.P.: Power and Influence: Beyond Formal Authority. Free Press, ISBN 0-02-918330-8 (1985)

  29. Krulwich, B., Burkey, C.: The contactfinder agent: answering bulleting board questions with referrals. In: National Conference of Artificial Intelligence (1996)

  30. Means, B., Toyama, Y., Murphy, R., Bakia, M., Jones, K.: Evaluation of evidence-based practices in online learning: a meta-analysis and review of online learning studies. In: U.S. Department of Education Office of Planning, Evaluation, and Policy Development Policy and Program Studies Service (2009)

  31. Mundinger, J., Le Boudec, J.-Y.: Reputation in self-organized communication systems and beyond. In: Inter-Perf (2006)

  32. Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: bringing order to the web. In: Stanford Digital Libraries Technologies Project (1998)

  33. Pal, A., Konstan, J.A.: Expert identification in community question answering: exploring question selection bias. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM ’10), pp. 1505–1508. ACM, New York (2010)

    Google Scholar 

  34. Panovich, K., Miller, R., Karger, D.: Tie strength in question & answer on social network sites. In: CSCW (2012)

  35. Resnick, P., Iacovou, N., Suchak, M., Bergstorm, P., Riedl, J.: Grouplens: an open architecture for collaborative filtering of netnews. In: ACM CCSCW (1994)

  36. Resnick, P., Zeckhauser, R., Friedman, E., Kuwabara, K.: Choosing reputable servents in a p2p network. In: ACM Communications (2000)

  37. Richardson, M., Agrawal, R., Dominigos, P.: Trust management for the semantic web. In: ISWC (2003)

  38. Sabater, J., Sierra, C.: Reputation and social network analysis in multi-agent systems. In: AAMAS (2002)

  39. Shachaf, P.: Answer quality on Q&A sites. In: Proceedings of the Sixteenth Americas Conference on Information Systems (AMCIS ’10). AIS Electronic Library (2010)

  40. Shah, C., Pomerantz, J.: Evaluating and predicting answer quality in community qa. In: Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval - SIGIR ’10, pp. 411–418, ACM (2010)

  41. Steiner, I.: Google Seller-Rating System a Threat to eBay?. http://www.ecommercebytes.com/cab/abn/y06/m08/i01/s04 (2006)

  42. Streeter, L., Lochbaum, K.: Who knows: a system based on automatic representation of semantic structure. In: RIAO (1988)

  43. Sun, Y., Yu, W., Han, Z., Ray Liu, K.J.: Information theoretic framework of trust modeling and evaluation for ad hoc networks. In: IEEE JSAC, pp. 305–317 (2006)

  44. Suryanto, M.A., Lim, E.P., Sun, A., Chiang, R.H.L.: Quality-aware collaborative question answering: methods and evaluation. In: Proceedings of the Second ACM International Conference on Web Search and Data Mining (WSDM ’09), pp. 142–151. ACM, New York (2009)

    Chapter  Google Scholar 

  45. Theodorakopoulos, G., Baras, J.: On trust models and trust evaluation metrics for ad hoc networks. In: IEEE JSAC (2006)

  46. Tu, X., Wang, X.-J., Feng, D., Zhang, L.: Analogical reasoning for answer ranking in social question answering. IEEE Intell. Syst. 27(5), 28–35 (2010)

    Article  Google Scholar 

  47. Wang, Y., Singh, M.P.: Trust via evidence combination: a mathematical approach based on uncertainty. In: TR 2006 North Carolina State University (2006)

  48. Wang, Y., Singh, M.P.: Trust representation and aggregation in a distributed agent system. In: AAAI (2006)

  49. Wang, Y., Singh, M.P.: Formal trust model for multiagent systems. In: IJCAI (2007)

  50. Wang, G., Gill, K., Mohanlal, M., Zheng, H., Zhao, B.: Wisdom in the social crowd: an analysis of quora. In: WWWW (2013)

  51. Zhang, J., Ackerman, M.A., Adamic, L.: Expertise networks in online communities: structure and algorithms. In: WWW (2007)

Download references

Author information

Authors and Affiliations

  1. School of Information Sciences, University of Pittsburgh, Pittsburgh, PA, USA

    Konstantinos Pelechrinis, Vladimir Zadorozhny, Velin Kounev & Yiling Lin

  2. Department of Information & Communication Technology, University of Agder, Grimstad, Norway

    Vladimir Oleshchuk

  3. Department of Computer Science, NC A&T State University, Greensboro, NC, USA

    Mohd Anwar

Authors
  1. Konstantinos Pelechrinis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vladimir Zadorozhny
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Velin Kounev
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vladimir Oleshchuk
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohd Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yiling Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Konstantinos Pelechrinis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pelechrinis, K., Zadorozhny, V., Kounev, V. et al. Automatic evaluation of information provider reliability and expertise. World Wide Web 18, 33–72 (2015). https://doi.org/10.1007/s11280-013-0249-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-013-0249-x

Keywords

Search

Navigation