Intelligent Multi Sensor Fusion System for Advanced Situation Awareness in Urban Environments

  • Georg Hummel
  • Martin Russ
  • Peter Stütz
  • John Soldatos
  • Lorenzo Rossi
  • Thomas Knape
  • Ákos Utasi
  • Levente Kovács
  • Tamás Szirányi
  • Charalampos Doulaverakis
  • Ioannis Kompatsiaris
Part of the Communications in Computer and Information Science book series (CCIS, volume 318)


This paper presents a distributed multi sensor data processing and fusion system providing sophisticated surveillance capabilities in the urban environment. The system enables visual/non-visual event detection, situation assessment, and semantic event-based reasoning for force protection and civil surveillance applications. The novelties lie in the high level system view approach, not only concentrating on data fusion methodologies per se, but rather on a holistic view of sensor data fusion that provides both lower (sensor) level and higher level (semantic) fusion. At the same time, we concentrate on easy and quick extensibility with new sensors and processing capabilities. The system also makes provisions for visualizing and processing space-time alerts from sensor detections up to high level alerts based on rule-based semantic reasoning over sensor data and fusion events. The proposed architecture has been validated in a number of different synthetic and live urban scenarios.


Sensor Fusion Common Operational Picture UAV JDL 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Weiser, M.: The Computer for the 21st Century. Scientific American 265(3), 66–75 (1991)CrossRefGoogle Scholar
  2. 2.
    Yau, S.S., Karim, F., Wang, Y., Wang, B., Gupta, S.K.S.: Reconfigurable Context-Sensitive Middleware for Pervasive Computing. IEEE Pervasive Computing, Joint Special Issue with IEEE Personal Communications on Context-Aware Pervasive Computing 1(3), 33–40 (2002)Google Scholar
  3. 3.
    Soldatos, J., Pandis, I., Stamatis, K., Polymenakos, L., Crowley, J.L.: Agent based middleware infrastructure for autonomous context-aware ubiquitous computing services. Computer Communications (COMCOM) 30(3), 577–591 (2007)CrossRefGoogle Scholar
  4. 4.
    Dimakis, N., Soldatos, J., Polymenakos, L., Fleury, P., Curín, J., Kleindienst, J.: Integrated Development of Context-Aware Applications in Smart Spaces. IEEE Pervasive Computing 7(4), 71–79 (2008)CrossRefGoogle Scholar
  5. 5.
    Aberer, K., Hauswirth, M., Salehi, A.: Infrastructure for data processing in large-scale interconnected sensor networks. In: MDM 2007, pp. 198–205 (2007)Google Scholar
  6. 6.
    Floerkemeier, C., Roduner, C., Lampe, M.: RFID Application Development with the Accada Middleware Platform. IEEE Systems Journal 1(2), 82–94 (2007)CrossRefGoogle Scholar
  7. 7.
    Chatzigiannakis, I., Mylonas, G., Nikoletseas, S.: 50 ways to build your application: A survey of middleware and systems for Wireless Sensor Networks. In: Proc. IEEE Conference on Emerging Technologies and Factory Automation, ETFA (2007)Google Scholar
  8. 8.
    Chen, H., et al.: Semantic Web in the Context Broker Architecture. In: Proc. Second Annual IEEE International Conference on Pervasive Computer and Communications (2004)Google Scholar
  9. 9.
    Kokar, M.M., Matheus, C.J., Baclawski, K.: Ontology-based situation awareness. Information Fusion, Special Issue on High-level Information Fusion and Situation Awareness 10(1), 83–98 (2009)Google Scholar
  10. 10.
    Pfisterer, D., Römer, K., Bimschas, D., Kleine, O., Mietz, R., Truong, C., Hasemann, H., Kröller, A., Pagel, M., Hauswirth, M., Karnstedt, M., Leggieri, M., Passant, A., Richardson, R.: SPITFIRE: toward a semantic web of things. IEEE Communications Magazine 49(11), 40–48 (2011)CrossRefGoogle Scholar
  11. 11.
    Kovács, L., Benedek, Cs.: Visual real-time detection, recognition and tracking of ground and airborne targets. In: Proc. of Computational Imaging IX, SPIE-IS&T Electronic Imaging, vol. 7873, pp. 787311–1–12. SPIE (2011)Google Scholar
  12. 12.
    Szlávik, Z., Kovács, L., Havasi, L., Benedek, Cs., Petrás, I., Utasi, Á., Licsár, A., Czúni, L., Szirányi, T.: Behavior and event detection for annotation and surveillance. In: Intl. Workshop on Content-Based Multimedia Indexing (CBMI 2008), pp. 117–124 (2008)Google Scholar
  13. 13.
    Utasi, Á.: Novel Probabilistic Methods for Visual Surveillance Applications. Phd Thesis, University of Pannonia, Veszprém, Hungary (2012)Google Scholar
  14. 14.
    Utasi, Á., Czúni, L.: Anomaly detection with low-level processes in videos. In: Proc. 3rd International Conference on Computer Vision Theory and Applications, pp. 678–681 (2008)Google Scholar
  15. 15.
    Tham, C., Buyya, R.: SensorGrid: Integrating sensor networks and grid computing. CSI Communications 29, 24–29 (2005)Google Scholar
  16. 16.
    Doulaverakis, C., Konstantinou, N., Knape, T., Kompatsiaris, I., Soldatos, J.: An approach to intelligent information fusion in sensor saturated urban environments. In: Proc. IEEE European Intelligence and Security Informatics Conference (EISIC), pp. 108–115 (2011)Google Scholar
  17. 17.
    White, F.E.: Data fusion lexicon. Data Fusion Subpanel of the Joint Directors of Laboratories, Technical Panel for C3 (1991)Google Scholar
  18. 18.
    Steinberg, A.N., Bowman, C.L., White, F.E.: Revisions to the JDL Data Fusion Model. Jrnl. of Sensor Fusion: Architectures, Algorithms, and Applications III (1999)Google Scholar
  19. 19.
    Virtuoso Universal Server,
  20. 20.
    Barwise, J., Perry, J.: Situations and Attitudes. MIT Press (1983)Google Scholar
  21. 21.
    Devlin, K.: Logic and Information. Cambridge U. Press (1991)Google Scholar
  22. 22.
    Mallot, H.A., Bülthoff, H.H., Little, J.J., Bohrer, S.: Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological Cybernetics 64, 177–185 (1991)zbMATHCrossRefGoogle Scholar
  23. 23.
    Talantzis, F., Pnevmatikakis, A., Constantinides, A.G.: Audio-Visual Person Tracking: A Practical Approach. World Scientific Publication Co., Imperial College Press, London (2011)CrossRefGoogle Scholar
  24. 24.
    Salehi, A., Riahi, M., Michel, S., Aberer, K.: GSN, middleware for stream world. In: Proc. 10th International Conference on Mobile Data Management (2009)Google Scholar
  25. 25.
    Konstantinou, N., Spanos, D.E., Stavrou, P., Mitrou, N.: Technically Approaching the Se-mantic Web Bottleneck. International Journal of Web Engineering and Technology (IJWET) 6(1), 83–111 (2010)CrossRefGoogle Scholar
  26. 26.
    Guangyu, L., Kefa, Z., Li, S., Jinlin, W., Qianfeng, W.: Regional Eco-Environmental Information Service System Based on Open Source Projects. Energy Procedia 11, 3892–3898 (2011)CrossRefGoogle Scholar
  27. 27.
    Hummel, G., Stütz, P.: Conceptual design of a simulation test bed for ad-hoc sensor networks based on a serious gaming environment. In: Proc. Intl. Training and Education Conference, ITEC (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Georg Hummel
    • 1
  • Martin Russ
    • 1
  • Peter Stütz
    • 1
  • John Soldatos
    • 2
  • Lorenzo Rossi
    • 3
  • Thomas Knape
    • 4
  • Ákos Utasi
    • 5
  • Levente Kovács
    • 5
  • Tamás Szirányi
    • 5
  • Charalampos Doulaverakis
    • 6
  • Ioannis Kompatsiaris
    • 6
  1. 1.Institute of Flight SystemsBundeswehr University MunichGermany
  2. 2.Athens Information TechnologyAthensGreece
  3. 3.Vitrociset SpaRomaItaly
  4. 4.Data Fusion InternationalDublinIreland
  5. 5.Computer and Automation Research InstituteHungarian Academy of SciencesHungary
  6. 6.Centre for Research and Technology HellasInformation Technologies InstituteThessalonikiGreece

Personalised recommendations