Online Crowdsource System Supporting Ground Truth Datasets Creation

  • Paweł Drozda
  • Krzysztof Sopyła
  • Przemysław Górecki
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7894)


This paper proposes a design of a system for creating image similarity datasets which are necessary for testing the quality of supervised ranking algorithms. In particular, the main goal is to facilitate the creation of similar images rankings for given a imaginary dataset. The system was designed in a manner that involves user feedback in the process of creating the rankings. In each iteration of ranking construction, the query image and twelve candidates are presented to the user, who is intended to select the most similar one. Moreover, in order to accelerate the method convergence the approach based on simulated annealing is adapted. It initially chooses the images randomly from a dataset and in the later stages the images with rank rate above zero are chosen with certain probability.


System Design CBIR Simulated Annealing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Collins, M., Duffy, N.: New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In: ACL, pp. 263–270 (2002)Google Scholar
  2. 2.
    Cooper, S., Khatib, F., Treuille, A., Barbero, J., Lee, J., Beenen, M., Leaver-Fay, A., Baker, D., Popović, Z., Foldit Players: Predicting protein structures with a multiplayer online game. Nature 466(7307), 756–760 (2010)CrossRefGoogle Scholar
  3. 3.
    Elo, A.E.: The Rating of Chess Players, Past and Present. Ishi Press (2008)Google Scholar
  4. 4.
    Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions and the Bayesian restoration of images. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(6), 721–741 (1984)MATHCrossRefGoogle Scholar
  5. 5.
    Górecki, P., Sopyła, K., Drozda, P.: Ranking by K-means voting algorithm for similar image retrieval. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 509–517. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Herbrich, R., Minka, T., Graepel, T.: TrueSkill(TM): A Bayesian Skill Rating System, vol. 20 (2007)Google Scholar
  7. 7.
    Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 2002, pp. 133–142. ACM, New York (2002)CrossRefGoogle Scholar
  8. 8.
    Kawrykow, A., Roumanis, G., Kam, A., Kwak, D., Leung, C., Wu, C., Zarour, E., Sarmenta, L., Blanchette, M., Waldispühl, J., Phylo players: Phylo: A citizen science approach for improving multiple sequence alignment. PLoS ONE 7(3), e31362 (2012)Google Scholar
  9. 9.
    Khatib, F., Cooper, S., Tyka, M.D., Xu, K., Makedon, I., Popović, Z., Baker, D., Foldit Players: Algorithm discovery by protein folding game players. Proceedings of the National Academy of Sciences 108(47), 18949–18953 (2011)CrossRefGoogle Scholar
  10. 10.
    Korytkowski, M., Nowicki, R., Rutkowski, L., Scherer, R.: Adaboost ensemble of DCOG rough–neuro–fuzzy systems. In: Jędrzejowicz, P., Nguyen, N.T., Hoang, K. (eds.) ICCCI 2011, Part I. LNCS, vol. 6922, pp. 62–71. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  11. 11.
    Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab. Previous number = SIDL-WP-1999-0120 (November 1999)Google Scholar
  12. 12.
    Paolacci, G., Chandler, J., Ipeirotis, P.G.: Running experiments on amazon mechanical turk. Judgment and Decision Making 5(5), 411–419 (2010)Google Scholar
  13. 13.
    Ross, J., Irani, L., Silberman, M.S., Zaldivar, A., Tomlinson, B.: Who are the crowdworkers?: shifting demographics in mechanical turk. In: Proceedings of the 28th of the International Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA 2010, pp. 2863–2872. ACM, New York (2010)CrossRefGoogle Scholar
  14. 14.
    Shen, L., Joshi, A.K.: Ranking and reranking with perceptron. Machine Learning, 73–96 (2005)Google Scholar
  15. 15.
    Sopyła, K., Drozda, P., Górecki, P.: Picrank - online crowdsource system for image ranking creation (2012)Google Scholar
  16. 16.
    Sopyła, K., Drozda, P., Górecki, P.: SVM with CUDA accelerated kernels for big sparse problems. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012, Part I. LNCS, vol. 7267, pp. 439–447. Springer, Heidelberg (2012)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Paweł Drozda
    • 1
  • Krzysztof Sopyła
    • 1
  • Przemysław Górecki
    • 1
  1. 1.Department of Mathematics and Computer SciencesUniversity of Warmia and MazuryOlsztynPoland

Personalised recommendations