Signal, Image and Video Processing

, Volume 8, Issue 7, pp 1211–1231 | Cite as

PROMETHEUS: heterogeneous sensor database in support of research on human behavioral patterns in unrestricted environments

  • Stavros Ntalampiras
  • Dejan Arsić
  • Martin Hofmann
  • Maria Andersson
  • Todor Ganchev
Original Paper


The multi-modal multi-sensor PROMETHEUS database was created in support of research and development activities [PROMETHEUS (FP7-ICT-214901):] aiming at the creation of a framework for monitoring and interpretation of human behaviors in unrestricted indoor and outdoor environments. The distinctiveness of the PROMETHEUS database comes from the unique sensor sets, used in the various recording scenarios, but also from the database design, which covers a range of real-world applications, correlated to smart-home automation and indoors/outdoors surveillance of public areas. Numerous single-person and multi-person scenarios, but also scenarios with interactions between groups of people, motivated by these applications were implemented with the help of skilled actors and supernumerary personnel. In these scenarios, the actors and personnel were instructed to implement a range of typical and atypical behaviors, and simulations of emergency and crisis situations. In summary, the database contains more than 4 h of synchronized recordings from heterogeneous sensors (an infrared motion detection sensor, thermal imaging cameras, overview/surveillance video cameras, close-view video cameras, a 3D camera, a stereoscopic camera, a general-purpose camcoder, microphone arrays, and motion capture equipment) collected in common setups, simulating smart-home environment, airport, and ATM security environment. Selected scenes of the database were annotated for the needs of human detection and tracking. The entire audio part of the database was annotated for the needs of sound event detection, sound source enumeration, emotion recognition, etc.


Multimodal database Heterogeneous sensors Signal-based surveillance Civil safety 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    PROMETHEUS (FP7-ICT-214901).
  2. 2.
    CARETAKER project (IST FP6-027231).
  3. 3.
    Smith K., Ba S., Odobez J.-M., Gatica-Perez D.: Tracking the visual focus of attention for a varying number of wandering people. In: IEEE Trans. Pattern Anal. Mach. Intell. 30(7), 1212–1229 (2008)Google Scholar
  4. 4.
    Carincotte, C., Desurmont, X., Ravera, B., Bremond, F., Orwell, J., Velastin, S., Odobez, J., Corbucci, J., Palo, B.J., Cernocky J.: Toward generic intelligent knowledge extraction from video and audio: the EU-funded CARETAKER project. In: IEEE Conference on Imaging Detection and Prevention (ICDP), London, UK, pp. 470–475, 13–14 June 2006Google Scholar
  5. 5.
  6. 6.
    Desurmont, X., Sebbe, R., Martin, F., Machy, C., Delaigle J.-F.: Performance evaluation of frequent event detection system. In: International IEEE Workshop on Performance Evaluation of Tracking and Surveillance (PETS), New York, USA (2006)Google Scholar
  7. 7.
    PETS. Performance evaluation of tracking and surveillance.
  8. 8.
    Benabbas, Y., Ihaddadene, N., Djerba C.:Global analysis of motion vectors for event detection in crowded scenes. In: Proceedings 11th IEEE International Workshop on PETS, Miami, pp. 109–116, 25 June 2009Google Scholar
  9. 9.
    Sharma, P.K., Huang, C., Nevatia, R.: Evaluation of people tracking, counting and density estimation in crowded environments. In: Proceedings 11th IEEE International Workshop on PETS, Miami, pp. 39–46, 25 June 2009Google Scholar
  10. 10.
    Dalley, G., Wang, X., Grimsin W.E.L.: Event detection using an attention-based tracker. In: Proceedings 10th IEEE International Workshop on PETS, Rio de Janeiro, pp. 71–79, 14 Oct 2009Google Scholar
  11. 11.
    Arsic, D., Hofmann, M., Schulller, B., Rigoll, G.: Multi-camera person tracking and left luggage detection applying homographic transformation. In: Proceedings 10th IEEE International Workshop on PETS, Rio de Janeiro, pp. 55–62, 14 Oct 2007Google Scholar
  12. 12.
    CAVIAR. Context aware vision using image-based active recognition. EU IST programme project IST 2001 37540.
  13. 13.
    Nascimento, J.C., Figueiredo, M.A.T., Marques, J.S.: Recognizing human activities using space dependent switched dynamical models. In: IEEE International Conference on Image Processning, ICIP’2005, Genoa, Italy, 11–14 Sept 2005Google Scholar
  14. 14.
    HERMES project (IST-2005-027110).
  15. 15.
    Mozerov M., Amato A., Roca X., Gonzales J.: Trajectory occlusion handling with multiple-view distance-minimization clustering. Opt. Eng. 47(4), 2021–2029 (2008)CrossRefGoogle Scholar
  16. 16.
    CogVis project (IST-2000-29375).
  17. 17.
    Needham C.J., Santos P.E., Magee D.R., Devin V., Hogg D.C., Cohn A.G.: Protocols from perceptual observations. Artif. Intell. 167(1–2), 103–136 (2005)CrossRefGoogle Scholar
  18. 18.
  19. 19.
    Abad, A., Canton-Ferrer, C., Segura, C., Landabaso, J.L., Macho, D., Casas, J.R., Hernando, J., Pardas, M., Nadeu, C.: UPC audio, video and multimodal person tracking systems in the CLEAR evaluation campaign. In: Lecture Notes in Computer Science, vol. 4122, pp. 93–104 (2006)Google Scholar
  20. 20.
    Black, J., Ellis, T., Makris D.: A distributed database for effective management and evaluation of CCTV systems. In: Velastin, S.A., Remagnino, P. (eds.) Intelligent Distributed Video Surveillance Systems. Institution of Electrical Engineers, London, UK pp. 55–89 (2006)Google Scholar
  21. 21.
    O’Toole A.J., Harms J., Sow S.L., Hurst D.R., Pappas M.R., Ayyad J.H., Abdi H.: A video database of moving faces and people. In: IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 812–816 (2005)Google Scholar
  22. 22.
    Ortega-Garcia J. et al.: The multiscenario multienvironment biosecure multimodal database (BMDB). In: IEEE Trans. Pattern Anal. Mach. Intell. 32(6), 1097–1111 (2010)Google Scholar
  23. 23.
    Ntalampiras, S., Potamitis, I., Ganchev, T., Fakotakis, N.: Audio database in support of potential threat and crisis situation management. In: Proceedings of the 6th Conference on Language Resources and Evaluation, Morocco, pp. 1288–1291 (2008)Google Scholar
  24. 24.
    Clavel, C., Vasilescu, I., Devillers, L., Ehrette, T.: Fiction database for emotion detection in abnormal situations. In: Proceedings of International Conference on Spoken Language Processing, pp. 2277–2280, Korea, Oct 2004Google Scholar
  25. 25.
    Ntalampiras, S., Arsic, D., Stormer, A., Ganchev, T. Potamitis, I., Fakotakis N.: PROMETHEUS database: a multimodal corpus for research on modeling and interpreting human behavior. In: IEEE 17th International Conference on Digital Signal Processing 2009, Special Session: Fusion of Heterogeneous Data for Robust Estimation and Classification, Santorini, Greece, pp. 1–8 (2009)Google Scholar
  26. 26.
    D2.2: Usage scenarios, functional specifications and hardware components. Deliverable D2.2. PROMETHEUS project, Feb 2009Google Scholar
  27. 27.
    FP7 cooperation work programme 2009–2010: Information and Communication Technologies. European Commission. July 2009Google Scholar
  28. 28.
    FP7-ICT-SEC-2007-1: Joint call between ICT and security with themes on critical infrastructure protection. In: FP7 Cooperation Work Programme 2007, European Commission. Feb 2007Google Scholar
  29. 29.
    Ferryman, J., Tweed D.: An overview of the PETS 2007 dataset. In: Proceedings of the Tenth IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Rio de Janeiro, Oct 2007Google Scholar
  30. 30.
    Thirde, D., Li, L., Ferryman, J.: An overview of the PETS 2006 dataset. In: Proceedings of the 2nd Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, pp. 317–324, Oct 2005Google Scholar
  31. 31.
    Directive 95/46/EC of the European Parliament and of the Council of 24 October 1995 on the protection of individuals with regard to the processing of personal data and on the free movement of such data OJ. No L. 281, pp. 0031–0050, 23 Nov 1995Google Scholar
  32. 32.
    Wallhoff, F., Ruß, M., Rigoll, G., Göbel, J., Diehl, H.: Improved image segmentation using photonic mixer devices. In: Proceedings of International Conference on Image Processing, Texas, vol.~VI, pp. 53–56, Sept 2007Google Scholar
  33. 33.
    PMD Technologies: Data sheet PMD (vision) 3k-s. Online document: Accessed 7 June 2012
  34. 34.
    Lichtenauer, J., Valstar, M., Shen, J., Pantic, M.: Cost-effective solution to synchronized audio-visual capture using multiple sensors. In: Proceedings of the Advanced Video and Signal Based Surveillance, pp. 324–329, 2–4 Sept 2009Google Scholar
  35. 35.
    Kipp M.: Anvil—a generic annotation tool for multimodal dialogue. In: Proceedings of the 7th European Conference on Speech Communication and Technology, Aalborg, pp. 1367–1370, 3–7 Sept 2001Google Scholar
  36. 36.
    UK Home Office: Multiple-camera tracking scenario. Online document. Available on-line at: Oct 2008
  37. 37.
    Mariano, V.Y., Min, J., Park, J.-H., Kasturi, R., Mihalcik, D., Doermann, D., Drayer, T.: Performance evaluation of object detection algorithms. In: Proceedings of International Conference on Pattern Recognition, Quebec, pp. 965–969, 11–15 Aug 2002Google Scholar
  38. 38.
  39. 39.
    Hiroaki, N., Takanobu, N., Hiroshi, K.: Acoustic-based security system: towards a robust understanding of emergency shout. In: 5th International Conference on Information Assurance and Security, Xian, China, pp. 725–728, 18–20 Aug 2009Google Scholar
  40. 40.
    Ntalampiras, S., Potamitis, I., Fakotakis, N.: An adaptive framework for acoustic monitoring of potential hazards. EURASIP Journal on Audio, Speech, and Music Processing. 2009. Article ID 594103 (2009). doi: 10.1155/2009/594103
  41. 41.
    Ntalampiras, S., Potamitis, I., Fakotakis, N.: On acoustic surveillance of hazardous situations. In: International Conference on Acoustics, Speech and Signal Processing, Taiwan, Taipei, 19–24 April 2009, pp. 165–168Google Scholar
  42. 42.
    Ntalampiras, S., Potamitis, I., Fakotakis, N.: A portable system for robust acoustic detection of atypical situations. In: 17th European Signal Processing Conference, Glasgow, Scotland, 24–28 Aug 2009, pp. 1121–1125Google Scholar
  43. 43.
    Ganchev, T., Mporas, I., Fakotakis, N.: Automatic height estimation from speech in real-world setup. In: 18th European Signal Processing Conference, Aalborg, Danmark, pp. 800–804, 23–27 Aug 2010Google Scholar
  44. 44.
    Andersson, M., Ntalampiras, S., Ganchev, T., Rydell, J., Ahlberg, J., Fakotakis, N.: Fusion of acoustic and optical sensor data for automatic fight detection in urban environment. In: International Conference on Information Fusion, Edinburgh, UK, pp. 1–8, 26–29 July 2010Google Scholar

Copyright information

© Springer-Verlag London Limited 2012

Authors and Affiliations

  • Stavros Ntalampiras
    • 1
  • Dejan Arsić
    • 2
  • Martin Hofmann
    • 2
  • Maria Andersson
    • 3
  • Todor Ganchev
    • 4
  1. 1.System Architectures Group, Dipartimento di Elettronica e InformazionePolitecnico di MilanoMilanItaly
  2. 2.Institute for Human Machine CommunicationTechnische Universität MünchenMunichGermany
  3. 3.Division of Information SystemsFOI, Swedish Defense Research AgencyLinköpingSweden
  4. 4.Wire Communications Laboratory, Electrical and Computer Engineering DepartmentUniversity of PatrasRion, PatrasGreece

Personalised recommendations