A Gamified Approach to Relevance Judgement

Abstract

In contrast to standard procedures for pooled relevance judgments which require considerable manual effort, we take a radically different approach to obtaining relevance assessments as the by-product of a game designed with a purpose (GWAP). The objective is to harness human processing skills via fun and entertainment. DocMiner is a document guessing game where a human player (H) needs to correctly guess a document chosen by the computer (C). To start the game, C chooses an information need and a document that is relevant to it. C then shares the information need, expressed as a query string, with H. H wins the round if he correctly guesses the document that C has chosen, and is either rewarded with bonus points or penalized by deducting points on submission of relevant and non-relevant documents respectively. The human player, as a part of his game playing strategy, thus needs to find relevant documents with the help of a search engine. Experiments on the TREC-8 ad-hoc task with the objective of reproducing the existing relevance assessments demonstrate that gamified assessments, when used to evaluate the official submissions to TREC-8, show fair correlation (Pearson’s correlation coefficient of up to 0.84) with the official assessments (depth-100 pooling).