The presence of numerous and disparate information sources available to support decision-making calls for efficient methods of harnessing their potential. Information sources may be unreliable, and misleading reports can affect decisions. Existing trust and reputation mechanisms typically rely on reports from as many sources as possible to mitigate the influence of misleading reports on decisions. In the real world, however, it is often the case that querying information sources can be costly in terms of energy, bandwidth, delay overheads, and other constraints. We present a model of source selection and fusion in resource-constrained environments, where there is uncertainty regarding the trustworthiness of sources. We exploit diversity among sources to stratify them into homogeneous subgroups to both minimise redundant sampling and mitigate the effect of certain biases. Through controlled experiments, we demonstrate that a diversity-based approach is robust to biases introduced due to dependencies among source reports, performs significantly better than existing approaches when sampling budget is limited and equally as good with an unlimited budget.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Price excludes VAT (USA)
Tax calculation will be finalised during checkout.
See: http://www.bbc.co.uk/news/uk-14490693 for how Twitter was used to spread false rumours during the England riots of 2011.
This is consistent with most real-world economic settings .
We use the M5 implementation of Weka , a popular open-source machine learning toolkit written in Java
The M5 algorithm can accommodate other feature types including qualitative. As well as this, different metrics exist for computing the distance between features of other kinds .
Antos, A., Grover, V., & Szepesvári, C. (2008). Active learning in multi-armed bandits. In Proceedings of the 19th International Conference on Algorithmic Learning Theory (pp. 287–302).
Antos, A., Grover, V., & Szepesvári, C. (2010). Active learning in heteroscedastic noise. Theoretical Computer Science, 411(29), 2712–2728.
Breiman, L., Friedman, J., Stone, C., & Olshen, R. (1984). Classification and Regression Trees. Belmont, CA: Wadsworth.
Burke, J., Estrin, D., Hansen, M., Parker, A., Ramanathan, N., Reddy, S., & Srivastava, M. B. (2006). Participatory sensing. In ACM Sensys World Sensor Web Workshop.
Burnett, C., Norman, T. J., & Sycara, K. (2010). Bootstrapping trust evaluations through stereotypes. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems (pp. 241–248).
Campbell, A. T., Lane, N. D., Miluzzo, E., Peterson, R. A., Lu, H., Zheng, X., et al. (2008). The rise of people-centric sensing. Internet Computing, 12(4), 12–21.
Cochran, W. G. (1977). Sampling Techniques. New York: Wiley.
Dong, X., Berti-Equille, L., & Srivastava D. (2009) Integrating conflicting data: the role of source dependence. In Proceedings of the VLDB Endowment (PVLDB) (vol. 2).
Etoré, P., & Jourdain, B. (2010). Adaptive optimal allocation in stratified sampling methods. Methodology and Computing in Applied Probability, 12, 335–360.
Fang, H., Zhang, J., & Magnenat Thalmann, N. (2014) Subjectivity grouping: Learning from users’ rating behavior. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (pp. 1241–1248).
Feldstein, M. (1997). The costs and benefits of going from low inflation to price stability. In Reducing Inflation: Motivation and Strategy (pp. 123–166). Chicago: University of Chicago Press.
Frank, E., Wang, Y., Inglis, S., Holmes, G., & Witten, I. H. (1998). Using model trees for classification. Machine Learning, 32(1), 63–76.
Ganeriwal, S., Balzano, L., & Srivastava, M. (2008). Reputation-based framework for high integrity sensor networks. ACM Transactions on Sensor Networks, 4(3), 15.
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009). The weka data mining software: An update. ACM SIGKDD Explorations Newsletter, 11, 10–18.
Hang, C.-W., Singh, M.P. (2009). Selecting trustworthy service in service-oriented environments. In The 12th AAMAS Workshop on Trust in Agent Societies (pp. 1–12).
Hang, C.-W., & Singh, M. P. (2011). Trustworthy service selection and composition. ACM Transactions on Autonomous and Adaptive Systems (TAAS), 6(1), 5.
Ichino, M., & Yaguchi, H. (1994). Generalized minkowski metrics for mixed feature-type data analysis. IEEE Transactions Systems on Man and Cybernetics, 24, 698–708.
Irissappane, A., Oliehoek, F., Zhang, J. (2014). A pomdp based approach to optimally select sellers in electronic marketplaces. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems (pp. 1329–1336). International Foundation for Autonomous Agents and Multiagent Systems.
Jain, A., Murty, M., & Flynn, P. (1999). Data clustering: A review. ACM Computing Surveys, 31, 264–323.
Jøsang, A. (2013). Subjective Logic. Book Draft
Jøsang, A., & Ismail, R. (2002). The beta reputation system. In Proceedings of the 15th Bled Electronic Commerce Conference (pp. 41–55), New York, USA
Jøsang, A., Ismail, R., & Boyd, C. (2007). A survey of trust and reputation systems for online service provision. Decision Support Systems, 43(2), 618–644.
Jøsang, A., & Presti, S. (2004). Analysing the relationship between risk and trust. In Trust Management (pp. 135–145).
Kamar, E., Hacker, S., & Horvitz, E. (2012) Combining human and machine intelligence in large-scale crowdsourcing. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems (pp. 467–474).
Kittur, A., Chi, E., & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 453–456). ACM.
Kwak, H., Lee, C., Park, H., Moon, S. (2010). What is twitter, a social network or a news media? In Proceedings of the 19th International Conference on World Wide Web (pp. 591–600). ACM
Liu, X., Datta, A., Rzadca, K., & Lim, E.-P. (2009). StereoTrust: A group based personalized trust model. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (pp. 7–16).
O’Connor, B., Balasubramanyan, R., Routledge, B., & Smith, N. (2010). From tweets to polls: Linking text sentiment to public opinion time series. In Proceedings of the 4th International AAAI Conference on Weblogs and Social Media (pp. 122–129).
Osborne, M. A., Roberts, S. J., Rogers, A., Ramchurn, S. D., & Jennings, N. R. (2008). Towards real-time information processing of sensor network data using computationally efficient multi-output gaussian processes. In Proceedings of the 7th International Conference on Information Processing in Sensor Networks (pp. 109–120).
Ouyang, R., Srivastava, M., Toniolo, A., & Norman, T. J. (2014). Truth discovery in crowdsourced detection of spatial events. Technical report, International Technology Alliance (ITA), https://www.usukitacs.com/node/2637.
Quinlan, J. (1992). Learning with continuous classes. In Proceedings of the 5th Australian Joint Conference on Artificial Intelligence (pp. 343–348).
Raafat, R., Chater, N., & Frith, C. (2009). Herding in humans. Trends in Cognitive Sciences, 13, 420–428.
Reece, S., Rogers, A., Roberts, S., Jennings, N.R. (2007). Rumours and reputation: Evaluating multi-dimensional trust within a decentralised reputation system. In Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems (pp. 1063–1070).
Regan, K., Poupart, P., & Cohen, R. (2006). Bayesian reputation modeling in e-marketplaces sensitive to subjectivity, deception and change. In Proceedings of the 21st National Conference on Artificial Intelligence (pp. 206–212).
Şensoy, M., Fokoue, A., Pan, J., Norman, T. J., Tang, Y., Oren, N., & Sycara, K. (2013). Reasoning about uncertain information and conflict resolution through trust revision. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems (pp. 837–844).
Şensoy, M., Yilmaz, B., & Norman, T. J. (2014). STAGE: Stereotypical trust assessment through graph extraction. Computational Intelligence. doi:10.1111/coin.12046.
Shiller, J. R. (1995). Conversation, information, and herd behaviour. American Economic Review, 85, 181–185.
Surowiecki, J. (2005). The wisdom of crowds. Anchor.
Teacy, W. L., Luck, M., Rogers, A., & Jennings, N. R. (2012). An efficient and versatile approach to trust and reputation using hierarchical Bayesian modelling. Artificial Intelligence, 193, 149–185.
Teacy, W.T.L., Chalkiadakis, G., Rogers, A., & Jennings, N. (2008). Sequential decision making with untrustworthy service providers. In Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems (pp. 755–762).
Teacy, W. T. L., Patel, J., Jennings, N. R., & Luck, M. (2006). Travos: Trust and reputation in the context of inaccurate information sources. Autonomous Agents and Multi-Agent Systems, 12, 183–198.
Tran-Thanh, L., Huynh, T.D., Rosenfeld, A., Ramchurn, S. D., & Jennings, N.R. (2014). Budgetfix: Budget limited crowdsourcing for interdependent task allocation with quality guarantees. In Proceedings of the 13th International Conference on Autonomous Agents and Multiagent Systems (pp. 477–484).
Uddin, M., Amin, M., Le, H., Abdelzaher, T., Szymanski, B., & Nguyen, T. (2012). On diversifying source selection in social sensing. In Proceedings of the 9th International Conference on Networked Sensing Systems (INSS) (pp. 1–8).
Waksberg, J. (1978). Sampling methods for random digit dialing. Journal of the American Statistical Association, 73(361), 40–46.
Wang, D., Kaplan, L., & Abdelzaher, T. F. (2014). Maximum likelihood analysis of conflicting observations in social sensing. ACM Transactions on Sensor Networks, 10(2), 30:1–30:27.
Whitby, A., Jøsang, A., Indulska, J. (2004). Filtering out unfair ratings in bayesian reputation systems. In Proceedings of the 7th International Workshop on Trust in Agent Societies.
Witten, I. H., & Frank, E. (2005). Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann.
Wolfson, O., Xu, B., & Tanner, R. (2007). Mobile peer-to-peer data dissemination with resource constraints. In Proceedings of the 8th International Conference on Mobile Data Management (pp. 16–23).
Yin, X., Han, J., & Yu, P. (2008). Truth discovery with multiple conflicting information providers on the web. IEEE Transactions on Knowledge and Data Engineering, 20(6), 796–808.
Zhang, W., Das, S., & Liu, Y. (2006). A trust based framework for secure data aggregation in wireless sensor networks. Proceedings of the 3rd Annual IEEE Communications Society Conference on Sensor (Vol. 1, pp. 60–69)., Mesh and Ad Hoc Communications and Networks IEEE: IEEE.
Zheng, Z., & Padmanabhan, B. (2002). On active learning for data acquisition. In Proceedings of IEEE Conference on Data Mining (pp. 562–569).
Dr. Şensoy thanks to the U.S. Army Research Laboratory for its support under grant W911NF-14-1-0199 and The Scientific and Technological Research Council of Turkey (TUBITAK) for its support under grant 113E238.