Human Versus Machine: Establishing a Human Baseline for Multimodal Location Estimation

  • Jaeyoung ChoiEmail author
  • Howard Lei
  • Venkatesan Ekambaram
  • Pascal Kelm
  • Luke Gottlieb
  • Thomas Sikora
  • Kannan Ramchandran
  • Gerald Friedland


In recent years, the problem of video location estimation (i.e., estimating the longitude/latitude coordinates of a video without GPS information) has been approached with diverse methods and ideas in the research community and significant improvements have been made. So far, however, systems have only been compared against each other and no systematic study on human performance has been conducted. Based on a human-subject study with 11,900 experiments, this article presents a human baseline for location estimation for different combinations of modalities (audio, audio/video, audio/video/text). Furthermore, this article compares state-of-the-art location estimation systems with the human baseline. Although the overall performance of humans’ multimodal video location estimation is better than current machine learning approaches, the difference is quite small: For 41 % of the test set, the machine’s accuracy was superior to the humans. We present case studies and discuss why machines did better for some videos and not for others. Our analysis suggests new directions and priorities for future work on the improvement of location inference algorithms.


Training Dataset Gaussian Mixture Model Location Estimation Human Worker Automatic Speech Recognition System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    S. Chatzichristofis, Y. Boutalis, CEDD: color and edge directivity descriptor: a compact descriptor for image indexing and retrieval, Computer Vision Systems (Springer, Berlin, 2008), pp. 312–322Google Scholar
  2. 2.
    S. Chatzichristofis, Y. Boutalis, Fcth: Fuzzy color and texture histogram-a low level feature for accurate image retrieval, in Ninth International Workshop on Image Analysis for Multimedia Interactive Services, WIAMIS’08, pp. 191–196. IEEE (2008)Google Scholar
  3. 3.
    J. Choi, G. Friedland, V. Ekambaram, K. Ramchandran, Multimodal location estimation of consumer media: dealing with sparse training data, in 2012 IEEE International Conference on Multimedia and Expo (ICME). pp. 43–48, IEEE (2012)Google Scholar
  4. 4.
    J. Choi, H. Lei, V. Ekambaram, P. Kelm, L. Gottlieb, T. Sikora, K. Ramchandran, G. Friedland, Human vs machine: Establishing a human baseline for multimodal location estimation, in Proceedings of the 21st ACM International Conference on Multimedia, MM ’13, pp. 867–876. ACM, New York, USA (2013)Google Scholar
  5. 5.
    L. Gottlieb, J. Choi, G. Friedland, P. Kelm, T. Sikora. Pushing the limits of Mechanical Turk: qualifying the crowd for video geo-location, in Proceedings of the 2012 ACM Workshop on Crowdsourcing for Multimedia (CrowdMM) (2012)Google Scholar
  6. 6.
    A. Hatch, S. Kajarekar, A. Stolcke, Within-class covariance normalization for SVM-based speaker recognition, in Proceedings of ISCA Interspeech, vol. 4 (2006)Google Scholar
  7. 7.
    J. Hays, A. Efros, IM2GPS: estimating geographic information from a single image, in IEEE CVPR 2008, pp. 1–8 (2008)Google Scholar
  8. 8.
    S. Ioffe, Probabilistic linear discriminant analysis, Computer Vision-ECCV (Springer, Berlin, 2006), pp. 531–542Google Scholar
  9. 9.
    P.G. Ipeirotis, Analyzing the Amazon Mechanical Turk marketplace. XRDS 17(2), 16–21 (2010)Google Scholar
  10. 10.
    D. Karger, S. Oh, D. Shah, Budget-optimal crowdsourcing using low-rank matrix approximations, in 49th Annual Allerton Conference Communication, Control, and Computing (Allerton) 2011, pp. 284–291, September 2011Google Scholar
  11. 11.
    P. Kelm, S. Schmiedeke, T. Sikora, A hierarchical, multi-modal approach for placing videos on the map using millions of Flickr photographs, in Proceedings of SBNMA ’11, pp. 15–20. ACM, New York, USA (2011)Google Scholar
  12. 12.
    A. Kittur, E. H. Chi, B. Suh, Crowdsourcing user studies with Mechanical Turk, in Proceedings of the Twenty-Sixth Annual SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, pp. 453–456. ACM, New York, USA (2008)Google Scholar
  13. 13.
    M. Larson, M. Soleymani, P. Serdyukov, S. Rudinac, C. Wartena, V. Murdock, G. Friedland, R. Ordelman, G. J. Jones, Automatic tagging and geo-tagging in video collections and communities, in ACM International Conference on Multimedia Retrieval (ICMR 2011), pp. 51:1-51:8, April 2011Google Scholar
  14. 14.
    H. Lei, J. Choi, G. Friedland, City-Identification on Flickr Videos Using Acoustic Features. Technical report, ICSI Technical Report TR-11-001, 2011Google Scholar
  15. 15.
    D.M. Mount, S. Arya, ANN: A library for approximate nearest neighbor searching, in CGC 2nd Annual Fall Workshop on Computational Geometry, pp. 153 (1997)Google Scholar
  16. 16.
    A. Oliva, A. Torralba, Building the gist of a scene: the role of global image features in recognition. Prog. Brain Res. 155, 23–36 (2006)CrossRefGoogle Scholar
  17. 17.
    M.C. Palmer, Calculation of distance traveled by fishing vessels using GPS positional data: a theoretical evaluation of the sources of error. Fish. Res. 89(1), 57–64 (2008)Google Scholar
  18. 18.
    B. Russell, A. Torralba, K. Murphy, W. Freeman, LabelMe: a database and web-based tool for image annotation. Int. J. Comput. Vision 77, 157–173 (2008). doi: 10.1007/s11263-007-0090-8
  19. 19.
    M. Soufifar, M. Kockmann, L. Burget, O. Plchot, O. Glembek, T. Svendsen, iVector approach to phonotactic language recognition, in Proceedings of Interspeech, pp. 2913–2916 (2011)Google Scholar
  20. 20.
    H. Tamura, S. Mori, T. Yamawaki, Textural features corresponding to visual perception. IEEE Trans. Syst. Man Cybern. 8(6), 460–473 (1978)CrossRefGoogle Scholar
  21. 21.
    M. Wainwright, M. Jordan, Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1, 1–305 (2008)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Jaeyoung Choi
    • 1
    Email author
  • Howard Lei
    • 1
    • 2
  • Venkatesan Ekambaram
    • 3
  • Pascal Kelm
    • 4
  • Luke Gottlieb
    • 1
  • Thomas Sikora
    • 4
  • Kannan Ramchandran
    • 3
  • Gerald Friedland
    • 1
  1. 1.International Computer Science InstituteBerkeleyUSA
  2. 2.California State UniversityEast BayUSA
  3. 3.University of California at BerkeleyBerkeleyUSA
  4. 4.Technische UniversitätBerlinGermany

Personalised recommendations