Label Aggregation for Crowdsourced Triplet Similarity Comparisons

Abstract

Organizing objects such as human ideas and designs based on their similarity relationships is important in data exploration and decision making. Because humans are better at relative judgments than absolute judgments, those similarity comparisons are often cast as triplet comparisons asking which of two given objects is more similar to another given object. Crowdsourcing is an effective way to collect such human judgments easily; however, how to aggregate the labels of crowdsourced triplet similarity comparisons for estimating similarity relations of all objects when there are only a smaller number of labels remains a challenge. In this work, we construct two novel real datasets for investigating this research topic. For the label aggregation approach, we propose a family of models to learn the object embeddings from crowdsourced triplet similarity comparisons by incorporating worker abilities and object difficulties. Because of the diverse properties of real datasets, we automatically search for the optimal model from all variants of the proposed model. The experimental results verified the effectiveness of our approach.