A framework to restrict viewing of peer commentary on Web objects based on trust modeling

Abstract

In this paper, we present a framework aimed at assisting users in coping with the deluge of information within social networks. We focus on the scenario where a user is trying to digest feedback provided on a Web document (or a video) by peers. In this context, it is ideal for the user to be presented with a restricted view of all the commentary, namely those messages that are most beneficial in increasing the user’s understanding of the document. Operating within the computer science subfield of artificial intelligence, the centerpiece of our approach is a modeling of the trustworthiness of the person leaving commentary (the annotator), determined on the basis of ratings provided by peers, adjusted by a modeling of the similarity of those peers to the current user. We compare three competing formulae for restricting what is shown to users which vary in the extent to which they integrate trust modeling, to emphasize the value of this component. By simulating the knowledge gains achieved by users (inspired by methods used in peer-based intelligent tutoring), we are able to validate the effectiveness of our algorithms. Overall, we offer a framework to make the Social Web a viable source of information, through effective modeling of the credibility of peers. When peers are misguided or deceptive, our approach is able to remove these messages from consideration, for the user.

Appendix: Algorithmic complexity

Although our results demonstrate a clear ability to avoid displaying low-quality annotations, the system does require some computational effort to decide which comments will be displayed, the rate at which this computational effort scales will be quite important when considering the deployment of the system, especially in very large environments like social networks. Ideally, a deployed scheme should scale linearly in the resource usage otherwise required by the site (i.e., at worst a constant multiple of the product of the total number of users with the average number of web objects loaded by each user). This would be consistent with the usage rates of other resources required in a social network setting (storage, bandwidth, etc.). We will show that, although a naive implementation of our system would not scale linearly under all trust measures (the Cauchy measure in particular), a trivial change to the implementation would allow for linear scaling and thus widespread deployment, by using approximation algorithms when necessary.

We consider three possible trust metrics in this paper: the Tally, Cauchy, and trust-based measures. Computing the Tally measure for an individual user viewing a particular Web object means computing the sum of all votes cast for the object. However, this only needs to be done once, since the Tally measure of an object’s value is the same for every user (i.e., it does not take into account the similarity between users). Further, the Tally measure for an object can be updated in constant time: A new vote simply changes the total score of the object. Consequently, the Tally measure requires a constant amount of storage space per object, no additional storage space per user, and both computation and bandwidth scale linearly in the total number of votes cast (and thus, in the worst case, the total number of users times the average number of Web objects loaded per user). Further, it is trivial to store annotations sorted by their Tally scores, and thus any of the techniques used to select an annotation to display (apart from the Greedy God) could choose an annotation in constant time as well. Thus, when using the Tally measure, our system is guaranteed to be efficient and could be readily deployed to a large scale environment.

The Cauchy measure initially appears to be rather more computationally expensive, but we will show that with a simple sampling algorithm, we can achieve a linear runtime similar to the Tally measure (though with a larger constant factor). When computing the Cauchy measure to determine whether a particular user viewing a particular Web object should be shown a particular annotation, we first must compute the $T_q$ term in Eq. 5. Although this requires $O(|A_q| \times \hbox{mean}(|R^{ai}|))$ computational effort, where $\hbox{mean}(|R^{ai}|)$ is the mean number of votes over the annotator $q$’s annotations, and $|A_q|$ is the number of such annotations, this term can be computed just once, and updated incrementally, much like the Tally measure, for a given annotation. It can also be retrieved in constant time when required. Consequently, the computation of this term scales linearly with the usage of other required resources. The other term in Equation 5, however, is a weighted average of votes cast, based on the similarity of the viewer to each of the voters in question. A naive implementation would then require us to store a record of every vote cast by every user (which is still linear in terms of other resource usage, though potentially large), and also to perform $O(|U|*|C|)$ work to compute the final value, where $|U|$ is the total number of users in the system, and $|C|$ is the total number of annotations across all such objects. Since this amount of computational effort is required every time any user accesses any annotation, it scales roughly quadratically with the usage of other required resources and would clearly be infeasible for use in a large scale setting like a commercial social network. However, in the event that the number of users who have interacted with a given object is large (>100), statistical sampling techniques can be used to obtain an estimate of $(vF^A - vA^a)$ in constant time. To accomplish this, we could draw a small constant number (n) of users from those who cast votes on a particular annotation. If the users are drawn without replacement, than a value of $(vF^A - vA^a)$ computed using only the drawn votes is an unbiased estimator for the true value, with error proportionate to $\frac{1}{\sqrt{(}n)}$. Similarly, when computing the similarity between any two users, we could draw samples from the set of objects both users have cast ballots over and thus provide a constant bound on both the computation time and error in our estimations. The resulting Cauchy approximation would scale linearly with the usage of other required resources and so could also be deployed in a social network, or other large-scale application. However, the constant coefficients would be somewhat larger than for the other two methods.

The trust-based measure, a combination of the Tally measure with the $T_q$ term used in the Cauchy measure, scales linearly with other required resource usage by virtue of both its components scaling linearly (as shown above). Like the Tally measure, annotations can be sorted according to their trust-based scores and then selected for display efficiently. Consequently, the trust-based measure can also be applied directly to large scale applications.

Given the performance gains provided by the Cauchy measure in situations where authors generate undesirable annotations most of the time, its slightly larger (though still just a linear multiple of other resource usage in the worst case) computational and storage costs could be justified. However, in applications where most content is desirable to show, the Tally and trust-based measures may be preferred for their lower resource usage, and strong absolute performance.