VIOLA: Video Labeling Application for Security Domains

  • Elizabeth BondiEmail author
  • Fei Fang
  • Debarun Kar
  • Venil Noronha
  • Donnabell Dmello
  • Milind Tambe
  • Arvind Iyer
  • Robert Hannaford
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10575)


Advances in computational game theory have led to several successfully deployed applications in security domains. These game-theoretic approaches and security applications learn game payoff values or adversary behaviors from annotated input data provided by domain experts and practitioners in the field, or collected through experiments with human subjects. Beyond these traditional methods, unmanned aerial vehicles (UAVs) have become an important surveillance tool used in security domains to collect the required annotated data. However, collecting annotated data from videos taken by UAVs efficiently, and using these data to build datasets that can be used for learning payoffs or adversary behaviors in game-theoretic approaches and security applications, is an under-explored research question. This paper presents VIOLA, a novel labeling application that includes (i) a workload distribution framework to efficiently gather human labels from videos in a secured manner; (ii) a software interface with features designed for labeling videos taken by UAVs in the domain of wildlife security. We also present the evolution of VIOLA and analyze how the changes made in the development process relate to the efficiency of labeling, including when seemingly obvious improvements surprisingly did not lead to increased efficiency. VIOLA enables collecting massive amounts of data with detailed information from challenging security videos such as those collected aboard UAVs for wildlife security. VIOLA will lead to the development of a new generation of game-theoretic approaches for security domains, including approaches that integrate deep learning and game theory for real-time detection and response.


UAV Security Video surveillance Labeling application 



This research was supported by UCAR N00173-16-2-C903, with the primary sponsor being the Naval Research Laboratory (Z17-19598). It was also partially supported by the Harvard Center for Research on Computation and Society Fellowship and the Viterbi School of Engineering Ph.D. Merit Top-Off Fellowship.


  1. 1.
    Alpcan, T., Basar, T.: A game theoretic approach to decision and analysis in network intrusion detection. In: Proceedings of 42nd IEEE Conference on Decision and Control, vol. 3, pp. 2595–2600. IEEE (2003)Google Scholar
  2. 2.
    Badrinarayanan, V., Galasso, F., Cipolla, R.: Label propagation in video sequences. In: CVPR, pp. 3265–3272. IEEE (2010)Google Scholar
  3. 3.
    Bae, S.H., Yoon, K.J.: Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In: CVPR (2014)Google Scholar
  4. 4.
    Basilico, N., Nittis, G.D., Gatti, N.: A security game combining patrolling and alarm-triggered responses under spatial and detection uncertainties. In: AAAI, pp. 397–403 (2016)Google Scholar
  5. 5.
    Catania, C.A., Bromberg, F., Garino, C.G.: An autonomous labeling approach to support vector machines algorithms for network traffic anomaly detection. Expert Syst. Appl. 39(2), 1822–1829 (2012)CrossRefGoogle Scholar
  6. 6.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: CVPR, pp. 248–255. IEEE (2009)Google Scholar
  7. 7.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)CrossRefGoogle Scholar
  8. 8.
    Fang, F., Nguyen, T.H., Pickles, R., Lam, W.Y., Clements, G.R., An, B., Singh, A., Tambe, M., Lemieux, A.: Deploying PAWS: field optimization of the protection assistant for wildlife security. In: AAAI, pp. 3966–3973 (2016)Google Scholar
  9. 9.
    Haskell, W., Kar, D., Fang, F., Tambe, M., Cheung, S., Denicola, E.: Robust protection of fisheries with COmPASS. In: IAAI, pp. 2978–2983 (2014)Google Scholar
  10. 10.
    Hodgson, J.C., et al.: Precision wildlife monitoring using unmanned aerial vehicles. Sci. Rep. 6, 22574 (2016). doi: 10.1038/srep22574 CrossRefGoogle Scholar
  11. 11.
    Johnson, M.P., Fang, F., Tambe, M.: Patrol strategies to maximize pristine forest area. In: AAAI (2012)Google Scholar
  12. 12.
    Kar, D., Fang, F., Fave, F.D., Sintov, N., Tambe, M.: “A Game of Thrones”: when human behavior models compete in repeated Stackelberg security games. In: AAMAS (2015)Google Scholar
  13. 13.
    Kar, D., Ford, B., Gholami, S., Fang, F., Plumptre, A., Tambe, M., Driciru, M., Wanyama, F., Rwetsiba, A., Nsubaga, M., Mabonga, J.: Cloudy with a chance of poaching: adversary behavior modeling and forecasting with real-world poaching data. In: AAMAS, pp. 159–167 (2017)Google Scholar
  14. 14.
    Kristan, M., Matas, J., Leonardis, A., Felsberg, M., Cehovin, L., Fernández, G., Vojir, T., Hager, G., Nebehay, G., Pflugfelder, R.: The visual object tracking vot2015 challenge results. In: ICCV Workshops, pp. 1–23 (2015)Google Scholar
  15. 15.
    Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision (1981)Google Scholar
  16. 16.
    Ma, C., Yang, X., Zhang, C., Yang, M.H.: Long-term correlation tracking. In: CVPR, pp. 5388–5396 (2015)Google Scholar
  17. 17.
    Milan, A., Leal-Taixé, L., Reid, I., Roth, S., Schindler, K.: Mot16: a benchmark for multi-object tracking. arXiv preprint arXiv:1603.00831 (2016)
  18. 18.
    Nguyen, P., Kim, J., Miller, R.C.: Generating annotations for how-to videos using crowdsourcing. In: CHI 2013 Extended Abstracts on Human Factors in Computing Systems, pp. 835–840 (2013)Google Scholar
  19. 19.
    Nguyen, T.H., Sinha, A., Gholami, S., Plumptre, A., Joppa, L., Tambe, M., Driciru, M., Wanyama, F., Rwetsiba, A., Critchlow, R., et al.: Capture: a new predictive anti-poaching tool for wildlife protection. In: AAMAS, pp. 767–775 (2016)Google Scholar
  20. 20.
    Nguyen, T.H., Yang, R., Azaria, A., Kraus, S., Tambe, M.: Analyzing the effectiveness of adversary modeling in security games. In: AAAI, pp. 718–724 (2013)Google Scholar
  21. 21.
    Nguyen-Dinh, L.V., Waldburger, C., Roggen, D., Tröster, G.: Tagging human activities in video by crowdsourcing. In: ICMR, pp. 263–270 (2013)Google Scholar
  22. 22.
    Pai, C.H., Lin, Y.P., Medioni, G.G., Hamza, R.R.: Moving object detection on a runway prior to landing using an onboard infrared camera. In: CVPR, pp. 1–8. IEEE (2007)Google Scholar
  23. 23.
    Park, S., Mohammadi, G., Artstein, R., Morency, L.P.: Crowdsourcing micro-level multimedia annotations: the challenges of evaluation and interface. In: CrowdMM, pp. 29–34 (2012)Google Scholar
  24. 24.
    Pita, J., Jain, M., Western, C., Portway, C., Tambe, M., Ordonez, F., Kraus, S., Paruchuri, P.: Deployed ARMOR protection: the application of a game theroetic model for security at the Los Angeles International Airport. In: AAMAS (2008)Google Scholar
  25. 25.
    Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR, pp. 779–788 (2016)Google Scholar
  26. 26.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NIPS, pp. 91–99 (2015)Google Scholar
  27. 27.
    Sheng, V.S., Provost, F., Ipeirotis, P.G.: Get another label? Improving data quality and data mining using multiple, noisy labelers. In: KDD, pp. 614–622 (2008)Google Scholar
  28. 28.
    Tambe, M.: Security and Game Theory: Algorithms, Deployed Systems, Lessons Learned. Cambridge University Press, Cambridge (2011)CrossRefzbMATHGoogle Scholar
  29. 29.
    Vondrick, C., Patterson, D., Ramanan, D.: Efficiently scaling up crowdsourced video annotation. Int. J. Comput. Vis. 101, 184–204 (2013)CrossRefGoogle Scholar
  30. 30.
    Yan, R., Yang, J., Hauptmann, A.: Automatically labeling video data using multi-class active learning. In: ICCV (2003)Google Scholar
  31. 31.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. (CSUR) 38(4), 13 (2006)CrossRefGoogle Scholar
  32. 32.
    Zhang, L., Li, Y., Nevatia, R.: Global data association for multi-object tracking using network flows. In: CVPR, pp. 1–8. IEEE (2008)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Elizabeth Bondi
    • 1
    Email author
  • Fei Fang
    • 2
  • Debarun Kar
    • 1
  • Venil Noronha
    • 1
  • Donnabell Dmello
    • 1
  • Milind Tambe
    • 1
  • Arvind Iyer
    • 3
  • Robert Hannaford
    • 3
  1. 1.University of Southern CaliforniaLos AngelesUSA
  2. 2.Carnegie Mellon UniversityPittsburghUSA
  3. 3.Air ShepherdBerkeley SpringsUSA

Personalised recommendations