International Journal of Computer Vision

, Volume 119, Issue 3, pp 346–373

Recognizing Fine-Grained and Composite Activities Using Hand-Centric Features and Script Data

  • Marcus Rohrbach
  • Anna Rohrbach
  • Michaela Regneri
  • Sikandar Amin
  • Mykhaylo Andriluka
  • Manfred Pinkal
  • Bernt Schiele
Article

DOI: 10.1007/s11263-015-0851-8

Cite this article as:
Rohrbach, M., Rohrbach, A., Regneri, M. et al. Int J Comput Vis (2016) 119: 346. doi:10.1007/s11263-015-0851-8

Abstract

Activity recognition has shown impressive progress in recent years. However, the challenges of detecting fine-grained activities and understanding how they are combined into composite activities have been largely overlooked. In this work we approach both tasks and present a dataset which provides detailed annotations to address them. The first challenge is to detect fine-grained activities, which are defined by low inter-class variability and are typically characterized by fine-grained body motions. We explore how human pose and hands can help to approach this challenge by comparing two pose-based and two hand-centric features with state-of-the-art holistic features. To attack the second challenge, recognizing composite activities, we leverage the fact that these activities are compositional and that the essential components of the activities can be obtained from textual descriptions or scripts. We show the benefits of our hand-centric approach for fine-grained activity classification and detection. For composite activity recognition we find that decomposition into attributes allows sharing information across composites and is essential to attack this hard task. Using script data we can recognize novel composites without having training data for them.

Keywords

Activity recognition Fine-grained recognition Script data Hand detection 

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Marcus Rohrbach
    • 1
    • 2
  • Anna Rohrbach
    • 1
  • Michaela Regneri
    • 3
    • 6
  • Sikandar Amin
    • 1
    • 4
  • Mykhaylo Andriluka
    • 1
    • 5
  • Manfred Pinkal
    • 3
  • Bernt Schiele
    • 1
  1. 1.Max Planck Institute for InformaticsSaarbrückenGermany
  2. 2.UC Berkeley EECS and ICSIBerkeleyUSA
  3. 3.Department of Computational Linguistics and PhoneticsSaarland UniversitySaarbrückenGermany
  4. 4.Department of InformaticsTechnische Universität MünchenMünchenGermany
  5. 5.Stanford UniversityStanfordUSA
  6. 6.SPIEGEL-Verlag, IT DepartmentSaarland UniversityHamburgGermany

Personalised recommendations