Multimedia Tools and Applications

, Volume 48, Issue 1, pp 23–49 | Cite as

RUSHES—an annotation and retrieval engine for multimedia semantic units

  • Oliver Schreer
  • Ingo Feldmann
  • Isabel Alonso Mediavilla
  • Pedro Concejero
  • Abdul H. Sadka
  • Mohammad Rafiq Swash
  • Sergio BeniniEmail author
  • Riccardo Leonardi
  • Tijana Janjusevic
  • Ebroul Izquierdo


Multimedia analysis and reuse of raw un-edited audio visual content known as rushes is gaining acceptance by a large number of research labs and companies. A set of research projects are considering multimedia indexing, annotation, search and retrieval in the context of European funded research, but only the FP6 project RUSHES is focusing on automatic semantic annotation, indexing and retrieval of raw and un-edited audio-visual content. Even professional content creators and providers as well as home-users are dealing with this type of content and therefore novel technologies for semantic search and retrieval are required. In this paper, we present a summary of the most relevant achievements of the RUSHES project, focusing on specific approaches for automatic annotation as well as the main features of the final RUSHES search engine.


Rushes Video retrieval Annotation Visualisation 



This work is a result of the FP6 project “RUSHES” Proposal no.: FP6-045189, which is funded by the European Commission. We would also like to thank Leticia Fuentes Ardeo, Mikel Frutos Hernandez and journalists at EiTB for priceless help during the experimental evaluation.


  1. 1.
    Adcock J, Cooper M, Pickens J (2008) Experiments in interactive video search by addition and subtraction. In: CIVR’08: proceedings of the 2008 international conference on content-based image and video retrieval. ACM, New York, NY, USA, pp 465–474CrossRefGoogle Scholar
  2. 2.
    Allen JF (1984) Towards a general theory of action and time. Artif Intell 23(2):123–154zbMATHCrossRefGoogle Scholar
  3. 3.
    Beardsley PA, Torr PHS, Zisserman A (1996) 3D model acquisition from extended image sequences. In: ECCV’96: proceedings of the 4th European conference on computer vision-volume II. Springer, London, UK, pp 683–695Google Scholar
  4. 4.
    Bederson B (2001) Photomesa: a zoomable image browser using quantum treemaps and bubble maps. In: Proceedings of the 14th annual ACM symposium on user interface software and technology, pp 71–80Google Scholar
  5. 5.
    Benini S, Bianchetti A, Leonardi R, Migliorati P (2006) Extraction of significant video summaries by dendrogram analysis. In: Proceedings of the international conference on image processing, ICIP’06. Atlanta, GA, USA, 8–11 OctoberGoogle Scholar
  6. 6.
    Benini S et al (2009) D21 report on final development of low level AV media processing and knowledge discovery, 2009. RUSHES Project, FP6-045189, Deliverable D21, WP2Google Scholar
  7. 7.
    Benmokhtar R, Dumont E, Merialdo B, Huet B (2006) Eurecom in trecvid 2006: high level features extractions and rushes study. In: TrecVid 2006, 10th international workshop on video retrieval evaluation, November 2006, Gaithersburg, USAGoogle Scholar
  8. 8.
    Borth D, Schulze C, Ulges A, Breuel TM (2008) Navidgator—similarity based browsing for image and video databases. In: KI’08: proceedings of the 31st annual german conference on advances in artificial intelligence. Springer, Berlin, pp 22–29Google Scholar
  9. 9.
    Cho J, Jeong S, Choi BU (2004) Automatic classification and skimming of articles in a news video using Korean closed-caption. In: Gelbukh AF (ed) Computational linguistics and intelligent text processing. Lecture notes in computer science, vol 2945. Springer, Berlin, pp 498–501Google Scholar
  10. 10.
    Duda RO, Hart PE, Stork DG (2001) Pattern classification, 2nd edn. Wiley, New YorkzbMATHGoogle Scholar
  11. 11.
    EiTB. Euskal irrati telebista.
  12. 12.
    Feldmann I, Waizenegger W, Schreer O (2008) Extraction of 3D scene structure for semantic annotation and retrieval of unedited video. In: IEEE 10th workshop on multimedia signal processing, pp 82–87Google Scholar
  13. 13.
    Fuentes Ardeo L et al (2008) Requirement analysis and use-cases definition for professional content creators or providers and home-users. RUSHES Project, FP6-045189, Deliverable D5, WP1Google Scholar
  14. 14.
    Hearst MA, Karadi C (1997) Cat-a-cone: an interactive interface for specifying searches and viewing retrieval results using a large category hierarchy. In: Proceedings of the 20th annual international ACM SIGIR conference on research and development in information retrieval, pp 246–255Google Scholar
  15. 15.
    Heer J, Card SK, Landay JA (2005) Prefuse: a toolkit for interactive information visualization. In: CHI’05: Proceeding of the SIGCHI conference on human factors in computing systems. ACM, New York, NY, USA, pp 421–430Google Scholar
  16. 16.
    Janjusevic T, Benini S, Izquierdo E, Leonardi R (2009) Random assisted browsing of Rushes archives. J Multimedia (in press)Google Scholar
  17. 17.
    Jeannin S, Divakaran A (2001) Mpeg-7 visual motion descriptors. IEEE Trans Circuits Syst Video Technol 11(6):720–724CrossRefGoogle Scholar
  18. 18.
    Johnson B, Shneiderman B (1991) Tree-maps: a space-filling approach to the visualization of hierarchical information structures. In: Proceedings of the IEEE conference on visualization. IEEE Computer Society Press, pp 284–291Google Scholar
  19. 19.
    Lamping J, Rao R (1994) Laying out and visualizing large trees using a hyperbolic space. In: Proceedings of the 7th ACM symposium on user interface software and technology. ACM, pp 13–14Google Scholar
  20. 20.
    Lozano A, Villegas P (2007) Recursive partitional hierarchical clustering for navigation in large media databases. In: Eighth international workshop on image analysis for multimedia interactive services, WIAMIS 2007. Santorini, Greece, 6–8 JuneGoogle Scholar
  21. 21.
    Munzner T (1998) Exploring large graphs in 3D hyperbolic space. IEEE Comput Graph Appl 18(4):18–23CrossRefGoogle Scholar
  22. 22.
    Over P, Smeaton AF, Awad G (2008) The trecvid 2008 bbc rushes summarization evaluation. In: TVS’08: Proceedings of the 2nd ACM TRECVid video summarization workshop. ACM, New York, NY, USA, pp 1–20CrossRefGoogle Scholar
  23. 23.
    PHAROS IST-45035. Platform for searching of audiovisual resources across online spaces.
  24. 24.
    Robertson GG, Mackinlay JD, Card SK (1991) Cone trees: animated 3D visualizations of hierarchical information. In: Proceedings of the SIGCHI conference on human factors in computing systems: reaching through technology, pp 189–194Google Scholar
  25. 25.
    Robertson GG, Czerwinski M, Larson K, Robbins DC, Thiel D, Dantzich MV (1998) Data mountain: using spatial memory for document management. In: Proceedings of the 11th annual ACM symposium on user interface software and technology, pp 153–162Google Scholar
  26. 26.
    RUSHES FP6-045189. Retrieval of multimedia semantic units for enhanced reusability.
  27. 27.
    Rutledge L, Hardman L, van Ossenbruggen J (1999) The use of SMIL: multimedia research currently applied on a global scale. In: Modeling multimedia information and systems conference, pp 1–17Google Scholar
  28. 28.
    Shade J, Gortler S, He L-W, Szeliski R (1998) Layered depth images. In: SIGGRAPH’98: proceedings of the 25th annual conference on computer graphics and interactive techniques. ACM, New York, NY, USA, pp 231–242CrossRefGoogle Scholar
  29. 29.
    Shi J, Tomasi C (1994) Good features to track. In: 1994 IEEE conference on computer vision and pattern recognition (CVPR’94), pp 593–600Google Scholar
  30. 30.
    Snoek CGM, Worring M (2005) Multimodal video indexing: a review of the state-of-the-art. Multimedia Tools and Applications 25(1):5–35CrossRefGoogle Scholar
  31. 31.
    Villa R, Gildea N, Jose JM (2008) Facetbrowser: a user interface for complex search tasks. In: El-Saddik A, Vuong S, Griwodz C, Del Bimbo A, Candan KS, Jaimes A (eds) Proceedings of the international conference on multimedia. ACM, pp 489–498Google Scholar
  32. 32.
    W3C. Synchronized multimedia integration language. World wide web consortium—web standards.

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Oliver Schreer
    • 1
  • Ingo Feldmann
    • 1
  • Isabel Alonso Mediavilla
    • 2
  • Pedro Concejero
    • 2
  • Abdul H. Sadka
    • 3
  • Mohammad Rafiq Swash
    • 3
  • Sergio Benini
    • 4
    Email author
  • Riccardo Leonardi
    • 4
  • Tijana Janjusevic
    • 5
  • Ebroul Izquierdo
    • 5
  1. 1.Fraunhofer Institute for Telecommunications/Heinrich-Hertz-InstitutBerlinGermany
  2. 2.Telefónica I+DMadridSpain
  3. 3.Brunel UniversityLondonUK
  4. 4.University of BresciaBresciaItaly
  5. 5.Queen MaryUniversity of LondonLondonUK

Personalised recommendations