Advertisement

Detecting Social Actions of Fruit Flies

  • Eyrun Eyjolfsdottir
  • Steve Branson
  • Xavier P. Burgos-Artizzu
  • Eric D. Hoopfer
  • Jonathan Schor
  • David J. Anderson
  • Pietro Perona
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8690)

Abstract

We describe a system that tracks pairs of fruit flies and automatically detects and classifies their actions. We compare experimentally the value of a frame-level feature representation with the more elaborate notion of ‘bout features’ that capture the structure within actions. Similarly, we compare a simple sliding window classifier architecture with a more sophisticated structured output architecture, and find that window based detectors outperform the much slower structured counterparts, and approach human performance. In addition we test our top performing detector on the CRIM13 mouse dataset, finding that it matches the performance of the best published method. Our Fly-vs-Fly dataset contains 22 hours of video showing pairs of fruit flies engaging in 10 social interactions in three different contexts; it is fully annotated by experts, and published with articulated pose trajectory features.

Keywords

Confusion Matrix Confusion Matrice Bout Duration Wing Angle Wing Extension 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
Download to read the full conference paper text

Supplementary material

978-3-319-10605-2_50_MOESM1_ESM.pdf (2.7 mb)
Electronic Supplementary Material (PDF 2,777 KB)

References

  1. 1.
    Burgos-Artizzu, X.P., Dollár, P., Lin, D., Anderson, D.J., Perona, P.: Social behavior recognition in continuous video. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1322–1329. IEEE (2012)Google Scholar
  2. 2.
    Schuldt, C., Laptev, I., Caputo, B.: Recognizing human actions: a local svm approach. In: Proceedings of the 17th International Conference on Pattern Recognition, ICPR 2004, vol. 3, pp. 32–36. IEEE (2004)Google Scholar
  3. 3.
    Gorelick, L., Blank, M., Shechtman, E., Irani, M., Basri, R.: Actions as space-time shapes. Transactions on Pattern Analysis and Machine Intelligence 29(12), 2247–2253 (2007)CrossRefGoogle Scholar
  4. 4.
    Marszałek, M., Laptev, I., Schmid, C.: Actions in context. In: IEEE Conference on Computer Vision & Pattern Recognition (2009)Google Scholar
  5. 5.
    Niebles, J.C., Chen, C.W., Fei-Fei, L.: Modeling temporal structure of decomposable motion segments for activity classification. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part II. LNCS, vol. 6312, pp. 392–405. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  6. 6.
    Kuehne, H., Jhuang, H., Garrote, E., Poggio, T., Serre, T.: HMDB: a large video database for human motion recognition. In: Proceedings of the International Conference on Computer Vision (ICCV) (2011)Google Scholar
  7. 7.
    Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)Google Scholar
  8. 8.
    Ryoo, M., Aggarwal, J.: Ut-interaction dataset, icpr contest on semantic description of human activities (sdha) (2010)Google Scholar
  9. 9.
    Oh, S., Hoogs, A., Perera, A., Cuntoor, N., Chen, C.C., Lee, J.T., Mukherjee, S., Aggarwal, J., Lee, H., Davis, L., et al.: A large-scale benchmark dataset for event recognition in surveillance video. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3153–3160. IEEE (2011)Google Scholar
  10. 10.
    Sigal, L., Black, M.J.: Humaneva: Synchronized video and motion capture dataset for evaluation of articulated human motion. Brown Univertsity TR 120 (2006)Google Scholar
  11. 11.
    Müller, M., Röder, T., Clausen, M., Eberhardt, B., Krüger, B., Weber, A.: Documentation mocap database hdm05 (2007)Google Scholar
  12. 12.
    Tenorth, M., Bandouch, J., Beetz, M.: The tum kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1089–1096. IEEE (2009)Google Scholar
  13. 13.
    De la Torre, F., Hodgins, J., Montano, J., Valcarcel, S., Forcada, R., Macey, J.: Guide to the carnegie mellon university multimodal activity (cmu-mmac) database. Tech. rep., Citeseer (2009)Google Scholar
  14. 14.
    Sung, J., Ponce, C., Selman, B., Saxena, A.: Unstructured human activity detection from rgbd images. In: 2012 IEEE International Conference on Robotics and Automation (ICRA), pp. 842–849. IEEE (2012)Google Scholar
  15. 15.
    Koppula, H.S., Gupta, R., Saxena, A.: Learning human activities and object affordances from rgb-d videos. arXiv preprint arXiv:1210.1207 (2012)Google Scholar
  16. 16.
    Oh, S.M., Rehg, J.M., Balch, T., Dellaert, F.: Learning and inferring motion patterns using parametric segmental switching linear dynamic systems. International Journal of Computer Vision 77(1-3), 103–124 (2008)CrossRefGoogle Scholar
  17. 17.
    Dollár, P., Rabaud, V., Cottrell, G., Belongie, S.: Behavior recognition via sparse spatio-temporal features. In: VS-PETS (October 2005)Google Scholar
  18. 18.
    Jhuang, H., Garrote, E., Yu, X., Khilnani, V., Poggio, T., Steele, A.D., Serre, T.: Automated home-cage behavioural phenotyping of mice. Nature Communications 1, 68 (2010)CrossRefGoogle Scholar
  19. 19.
    Dankert, H., Wang, L., Hoopfer, E.D., Anderson, D.J., Perona, P.: Automated monitoring and analysis of social behavior in drosophila. Nature Methods 6(4), 297–303 (2009)CrossRefGoogle Scholar
  20. 20.
    Kabra, M., Robie, A.A., Rivera-Alba, M., Branson, S., Branson, K.: Jaaba: interactive machine learning for automatic annotation of animal behavior. Nature Methods (2012)Google Scholar
  21. 21.
    Altun, Y., Tsochantaridis, I., Hofmann, T., et al.: Hidden markov support vector machines. In: ICML, vol. 3, pp. 3–10 (2003)Google Scholar
  22. 22.
    Hoai, M., Lan, Z.Z., De la Torre, F.: Joint segmentation and classification of human actions in video. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3265–3272. IEEE (2011)Google Scholar
  23. 23.
    Shi, Q., Cheng, L., Wang, L., Smola, A.: Human action segmentation and recognition using discriminative semi-markov models. International Journal of Computer Vision 93(1), 22–32 (2011)CrossRefzbMATHGoogle Scholar
  24. 24.
    Hoyer, S.C., Eckart, A., Herrel, A., Zars, T., Fischer, S.A., Hardie, S.L., Heisenberg, M.: Octopamine in male aggression of drosophila. Current Biology 18(3), 159–167 (2008)CrossRefGoogle Scholar
  25. 25.
    Hoopfer, E.D., Anderson, D.J.: Unpublished workGoogle Scholar
  26. 26.
    Asahina, K., Watanabe, K., Duistermars, B.J., Hoopfer, E., González, C.R., Eyjólfsdóttir, E.A., Perona, P., Anderson, D.J.: Tachykinin-expressing neurons control male-specific aggressive arousal in drosophila. Cell 156(1), 221–235 (2014)CrossRefGoogle Scholar
  27. 27.
    Chen, S., Lee, A.Y., Bowens, N.M., Huber, R., Kravitz, E.A.: Fighting fruit flies: a model system for the study of aggression. Proceedings of the National Academy of Sciences 99(8), 5664–5668 (2002)CrossRefGoogle Scholar
  28. 28.
    Hall, J.C.: The mating of a fly. Science 264(5166), 1702–1714 (1994)CrossRefGoogle Scholar
  29. 29.
    Branson, K., Robie, A.A., Bender, J., Perona, P., Dickinson, M.H.: High-throughput ethomics in large groups of drosophila. Nature Methods 6(6), 451–457 (2009)CrossRefGoogle Scholar
  30. 30.
    Viterbi, A.: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory 13(2), 260–269 (1967)CrossRefzbMATHGoogle Scholar
  31. 31.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: A library for large linear classification. The Journal of Machine Learning Research 9, 1871–1874 (2008)zbMATHGoogle Scholar
  32. 32.
    Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. Journal of Machine Learning Research, 1453–1484 (2005)Google Scholar
  33. 33.
    Bellman, R.: Dynamic programming and lagrange multipliers. Proceedings of the National Academy of Sciences of the United States of America 42(10), 767 (1956)zbMATHMathSciNetGoogle Scholar
  34. 34.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Eyrun Eyjolfsdottir
    • 1
  • Steve Branson
    • 1
  • Xavier P. Burgos-Artizzu
    • 1
  • Eric D. Hoopfer
    • 1
  • Jonathan Schor
    • 1
  • David J. Anderson
    • 1
    • 2
  • Pietro Perona
    • 1
  1. 1.California Institute of TechnologyPasadenaUSA
  2. 2.Howard Hughes Medical Institute (HHMI)USA

Personalised recommendations

Cite paper