Multimedia Tools and Applications

, Volume 46, Issue 2–3, pp 425–461 | Cite as

A context-aware middleware for real-time semantic enrichment of distributed multimedia metadata

  • Nikolaos Konstantinou
  • Emmanuel Solidakis
  • Anastasios Zafeiropoulos
  • Panagiotis Stathopoulos
  • Nikolas Mitrou


This paper investigates the problem of the real-time integration and processing of multimedia metadata collected by a distributed sensor network. The discussed practical problem is the efficiency of the technologies used in creating a Knowledge Base in real-time. Specifically, an approach is proposed for the real-time, rule-based semantic enrichment of lower level context features with higher-level semantics. The distinguishing characteristic is the provision of an intelligent middleware-based architecture on which low level components such as sensors, feature extraction algorithms, data sources, and high level components such as application-specific ontologies can be plugged. Throughout the paper, Priamos, a middleware architecture based on Semantic Web technologies is presented, together with a stress-test of the system’s operation under two test case scenarios: A smart security surveillance application and a smart meeting room application. Performance measurements are conducted and corresponding results are exposed.


Semantic Middleware Rule-based Real-time Context-aware Annotation 



The work presented in this paper is carried out within the Priamos project, sponsored by the Greek General Secretarial of Research and Technology “Image, Sound, Language” research and development action. The authors of this paper would like to thank their colleagues in Athens Information Technology ( for their collaboration.


  1. 1.
    Allen BD, Bishop G, Welch G (2001) Tracking: beyond 15 minutes of thought. SIGGRAPH Course PackGoogle Scholar
  2. 2.
    Baader F, Nutt W (2002) The description logic handbook. In: Baader F, Calvanese D, McGuinness DL, Nardi D, Patel-Schneider PF (eds) Basic description logics. Cambridge University Press, Cambridge, pp 47–100Google Scholar
  3. 3.
    Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component analysis. IEEE Trans Neural Netw 13(6):1450–1464CrossRefGoogle Scholar
  4. 4.
    Bechhofer S (2006) DIG 2.0: the DIG description logic interface.
  5. 5.
    Bechhofer S, van Harmelen F, Hendler J, Horrocks I, McGuinness D, Patel-Schneider P, Stein LA (2004) OWL web ontology language reference. World wide web consortium, recommendation REC-owl-ref-20040210Google Scholar
  6. 6.
    Beckett D (2004) RDF/XML syntax specification(Revised). W3C Recommendation
  7. 7.
    Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: recognition using class specific linear projection. IEEE Trans Pattern Anal Mach Intell 19(7):711–720CrossRefGoogle Scholar
  8. 8.
    Black W, McNaught J, Vasilakopoulos A, Zervanou K, Rinaldi F (2003) CAFETIERE: conceptual annotations for facts, events, terms, individual entities and rElations. Tech rep, UMIST, Parmenides TR-U4, 3.1. Manchester, UKGoogle Scholar
  9. 9.
    Bradski GR (1998) Computer vision face tracking for use in a perceptual user interface. Intel Technol J (1998)Google Scholar
  10. 10.
    Buitelaar P, Olejnik D, Sintek M (2004) A protégé plug-in for ontology extraction from text based on linguistic analysis. In: First European semantic web symposium (ESWS). Heraklion, GreeceGoogle Scholar
  11. 11.
    Cardoso J (2007) The semantic web vision: where are we? IEEE Intell Syst 22(5):84–88CrossRefGoogle Scholar
  12. 12.
    Carroll J, Dickinson I, Dollin C, Reynolds D, Seaborne A, Wilkinson K (2003) Jena: implementing the semantic web recommendations. Tech Rep HPL-2003-146, Hewlett-PackardGoogle Scholar
  13. 13.
    Chakravarthy A, Ciravegna F, Lanfranchi V (2006) Cross-media document annotation and enrichment. In: Proc 1st semantic web authoring and annotation workshop (SAAW2006)Google Scholar
  14. 14.
    Chen H, Finin T, Joshi A (2004) Semantic web in the context broker architecture. In: PERCOM ’04: proc of the second IEEE int’l conf on pervasive computing and communications (PerCom’04). IEEE Computer Society, Washington, DC, p 277CrossRefGoogle Scholar
  15. 15.
    Comaniciu D, Meer P (1999) Mean shift analysis and applications. In: Proc of the seventh IEEE int’l conf on computer vision (ICCV’99), vol 2. Kerkyra, Greece, pp 1197–1203Google Scholar
  16. 16.
    Dey A (2001) Understanding and using context. Personal and Ubiquitous Computing 5(1):4–7. doi:10.1007/s007790170019 CrossRefGoogle Scholar
  17. 17.
    Dey A, Abowd G, Salber D (1999) A context-based infrastructure for smart environments. In: First int’l workshop on managing interactions in smart environments (MANSE 99). Dublin, Ireland, pp 114–128Google Scholar
  18. 18.
    Dou D, Pan J, Qin H, LePendu P (2006) Towards populating and querying the semantic web. In: Proc of 2nd international workshop on scalable semantic web knowledge base systems (SSWS’06). Co-located with ISWC, pp 129–142Google Scholar
  19. 19.
    Dougherty E, Laplante P (1995) Introduction to real-time imaging, chap. What is real-time processing? Wiley-IEEE, New York, pp 1–9Google Scholar
  20. 20.
    Etzioni O, Cafarella M, Downey D, Popescu AM, Shaked T, Soderland S, Weld D, Yates A (2005) Unsupervised named-entity extraction from the web: an experimental study. Artif Intell 165(1):91–134. doi:10.1016/j.artint.2005.03.001 CrossRefGoogle Scholar
  21. 21.
    Francois ARJ, Nevatia R, Hobbs J, , Bolles RC (2005) VERL: an ontology framework for representing and annotating video events. IEEE MultiMed 12(4):76–86CrossRefGoogle Scholar
  22. 22.
    Goasdoué F, Reynaud C (1999) Modeling information sources for information integration. In: 11th European workshop on knowledge acquisition, modeling and management (EKAW 99), Lecture notes in artificial intelligence, vol 1621. Springer, Dagstuhl Castle, pp 121–138Google Scholar
  23. 23.
    Hirtle D, Boley H, Grosof B, Kifer M, Sintek M, Tabet S, Wagner G (2006) Schema specification of ruleML 0.91.
  24. 24.
    Horrocks I (1998) Using an expressive description logic: faCT or fiction? In: Cohn AG, Schubert L, Shapiro S (eds) Principles of knowledge representation and reasoning: proceedings of the 6th int’l conference (KR-98). Morgan Kaufman, San Francisco, pp 636–647Google Scholar
  25. 25.
    Horrocks I, Patel-Schneider P, van Harmelen F (2003) From \(\mathcal{SHIQ}\) and RDF to OWL: the making of a web ontology language. Journal of Web Semantics: Science, Services and Agents on the World Wide Web 1(1):7–26CrossRefGoogle Scholar
  26. 26.
    Iria J, Ciravegna F, Cimiano P, Lavelli A, Motta E, Gilardoni L, Mönch E (2004) Integrating information extraction, ontology learning and semantic browsing into organizational knowledge processes. In: Proc of the EKAW workshop on the application of language and semantic technologies to support knowledge management processes, at the 14th int’l conf on knowledge engineering and knowledge managementGoogle Scholar
  27. 27.
    Kagal L, Finin T, Johshi A (2003) A policy language for pervasive computing environment. In: Proc of IEEE fourth international workshop on policies for distributed systems and networks (POLICY’03)Google Scholar
  28. 28.
    Karame G, Stergiou A, Katsarakis N, Papageorgiou P, Pnevmatikakis A (2007) 2D and 3D face localization for complex scenes. In: Advanced video and signal based surveillance, AVSS 2007. IEEE Conference on, pp 371–376Google Scholar
  29. 29.
    Kaykova O, Khriyenko O, Kovtun D, Naumenko A, Terziyan, VAAZ (2005) General adaption framework: enabling interoperability for industrial web resources. International Journal on Semantic Web and Information Systems 1(3):31–63Google Scholar
  30. 30.
    Konstantinou N, Solidakis E, Zoi S, Zafeiropoulos A, Stathopoulos P, Mitrou N (2007) Priamos: a middleware architecture for real-time semantic annotation of context features. In: IET int’l conf on intelligent environments (IE’07). Ulm, Germany, pp 96–103CrossRefGoogle Scholar
  31. 31.
    Lassila O, Khushraj D (2005) Contextualizing applications via semantic middleware. In: Proc of the second annual international conference on mobile and ubiquitous systems: networking and services (MOBIQUITOUS’05). IEEE Computer Society, Washington, DC, pp 183–191CrossRefGoogle Scholar
  32. 32.
    Lien CC, Chiang CL, Lee CH (2007) Scene-based event detection for baseball videos. J Vis Commun Image Represent 18(1):1–14CrossRefGoogle Scholar
  33. 33.
    Liu S, Xu M, Yi H, Chia LT, Rajan D (2006) Multimodal semantic analysis and annotation for basketball video. EURASIP J Appl Signal Process 2006:1–13Google Scholar
  34. 34.
    Masuoka R, Parsia B, Labrou Y (2003) Task computing—the semantic web meets pervasive computing. In: 2nd int’l semantic web conference (ISWC’03). Sanibel Island, Florida, USAGoogle Scholar
  35. 35.
    May W, Alferes J, Amador R (2005) Rules and rule markup languages for the semantic Web, Lecture notes in computer science, vol 3791, chap active rules in the semantic web: dealing with language heterogeneity. Springer, Berlin, pp 30–44Google Scholar
  36. 36.
    Motik B, Sattler U (2006) A comparison of reasoning techniques for querying large description logic aBoxes. In: Hermann M, Voronkov A (eds) Proc of the 13th int’l conf on logic for programming artificial intelligence and reasoning (LPAR’06), LNCS, vol 4246. Springer, Phnom Penh, pp 227–241CrossRefGoogle Scholar
  37. 37.
    Pandis I, Soldatos J, Paar A, Reuter J, Carras M, Polymenakos L (2005) An ontology-based framework for dynamic resource management in ubiquitous computing environments. In: Proc of the 2nd int’l conf on embedded software and systems (ICESS’05)Google Scholar
  38. 38.
    Papamarkos G, Poulovassilis A, Wood PT (2003) Event-condition-action rule languages for the semantic web. In: Workshop on semantic web and databases (SWDB 03), pp 309–327Google Scholar
  39. 39.
    Patel-Schneider P, Horrocks I (2004) OWL web ontology language: semantics and abstract syntax Section 3. Direct model-theoretic semantics.
  40. 40.
    Petridis K, Anastasopoulos D, Saathoff C, Timmermann N, Kompatsiaris I, Staab S (2006) M-ontoMat-annotizer: image annotation. Linking ontologies and multimedia low-level features. In: Engineered applications of semantic web session (SWEA) at the 10th int’l conf on knowledge-based intelligent information and engineering systems (KES’06). Bournemouth, UKGoogle Scholar
  41. 41.
    Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF.
  42. 42.
    Román M, Hess C, Cerqueira R, Ranganathan A, Campbell R, Nahrstedt K (2002) Gaia: a middleware platform for active spaces. ACM SIGMOBILE Mobile Computing and Communications Review 6(4):65–67CrossRefGoogle Scholar
  43. 43.
    Schroeter R, Hunter J, Guerin J, Khan I, Henderson M (2006) A synchronous multimedia annotation system for secure collaboratories. In: Proceedings of the second IEEE int’l conf on e-science and grid computing (E-SCIENCE’06). IEEE Computer Society, Washington, DC, p 41CrossRefGoogle Scholar
  44. 44.
    Seaborne A, Manjunath G (2008) SPARQL/Update: a language for updating RDF graphs
  45. 45.
    Sirin E, Parsia B, Grau B, Kalyanpur A, Katz Y (2007) Pellet: a practical OWL-DL reasoner. Journal of Web Semantics: Science, Services and Agents on the World Wide Web 5(2):51–53CrossRefGoogle Scholar
  46. 46.
    Sohn J, Kim NS, Sung W (1999) A statistical model-Based voice activity detection. IEEE Signal Process Lett 6(1):1–3CrossRefGoogle Scholar
  47. 47.
    Stamou G, van Ossenbruggen J, Pan J, Schreiber G, Smith J (2006) Multimedia annotations on the semantic web. IEEE Multimed 13(1):86–90CrossRefGoogle Scholar
  48. 48.
    Stergiou A, Pnevmatikakis A, Polymenakos L (2007) The AIT multimodal person identification system for CLEAR 2007. In: Multimodal technologies for perception of humans, pp 221–232Google Scholar
  49. 49.
    Toninelli A, Montanari R, Kagal L, Lassila O (2006) The semantic web - ISWC 2006, Lecture notes in computer science, vol 4273, chap A semantic context-aware access control framework for secure collaborations in pervasive computing environments. Springer, Berlin, pp 473–486Google Scholar
  50. 50.
    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86CrossRefGoogle Scholar
  51. 51.
    Uren V, Cimiano P, Iria J, Handschuh S, Vargas-Vera M, Motta E, Ciravegna F (2006) Semantic annotation for knowledge management: requirements and a survey of the state of the art. Journal of Web Semantics, Elsevier 4(1):14–28CrossRefGoogle Scholar
  52. 52.
    Vazquez JI, de Ipiña DL, nigo Sedano I (2006) Computational science and its applications—ICCSA 2006, workshop on ubiquitous web systems and intelligence (UWSI 2006), Lecture notes in computer science, vol 3983/2006, chap SOAM: an environment adaptation Model for the pervasive semantic web. Springer, Berlin, pp 108–117Google Scholar
  53. 53.
    Vembu S, Kiesel M, Sintek M, Baumann S (2006) Towards bridging the semantic gap in multimedia annotation and retrieval. In: Proc of the 1st international workshop on semantic web annotations for multimedia, SWAMM 2006 at the 15th int’l world wide web conferenceGoogle Scholar
  54. 54.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: IEEE conf on computer vision and pattern recognition. Hawaii, pp 511–518Google Scholar
  55. 55.
    Vu VT, Bremond F, Thonnat M (2003) Automatic video interpretation: a novel algorithm for temporal scenario recognition. In: Proc of the 18th int’l joint conf on artificial intelligence (IJCAI 2003), pp 1295–1302Google Scholar
  56. 56.
    Zafeiropoulos A, Konstantinou N, Arkoulis S, Spanos DE, Mitrou N (2008) A semantic-based architecture for sensor data fusion. In: 2nd int’l conf on mobile ubiquitous computing, systems, services and technologies (UBICOMM’08). Valencia, Spain, pp 116–121CrossRefGoogle Scholar
  57. 57.
    Zhang D, Chang SF (2002) Event detection in baseball video using superimposed caption recognition. In: Proc of the tenth ACM int’l conf on multimedia. ACM, New York, pp 315–318CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Nikolaos Konstantinou
    • 1
  • Emmanuel Solidakis
    • 1
  • Anastasios Zafeiropoulos
    • 1
  • Panagiotis Stathopoulos
    • 1
  • Nikolas Mitrou
    • 1
  1. 1.National Technical University of AthensAthensGreece

Personalised recommendations