Information extraction from multimedia web documents: an open-source platform and testbed

  • David Paul Dupplaw
  • Michael Matthews
  • Richard Johansson
  • Giulia Boato
  • Andrea Costanzo
  • Marco Fontani
  • Enrico Minack
  • Elena Demidova
  • Roi Blanco
  • Thomas Griffiths
  • Paul Lewis
  • Jonathon Hare
  • Alessandro Moschitti
Regular Paper


The LivingKnowledge project aimed to enhance the current state of the art in search, retrieval and knowledge management on the web by advancing the use of sentiment and opinion analysis within multimedia applications. To achieve this aim, a diverse set of novel and complementary analysis techniques have been integrated into a single, but extensible software platform on which such applications can be built. The platform combines state-of-the-art techniques for extracting facts, opinions and sentiment from multimedia documents, and unlike earlier platforms, it exploits both visual and textual techniques to support multimedia information retrieval. Foreseeing the usefulness of this software in the wider community, the platform has been made generally available as an open-source project. This paper describes the platform design, gives an overview of the analysis algorithms integrated into the system and describes two applications that utilise the system for multimedia information retrieval.


Multimedia retrieval Web analysis  Text analysis  Opinion analysis Image analysis  Open-source software 



This work was supported by the European Union under the Seventh Framework project LivingKnowledge (IST-FP7-231126). We would also like to thank all our partners who contributed to the LivingKnowledge project on which this work has been built.


  1. 1.
    Agrawal R, Gollapudi S, Halverson A, Ieong S (2009) Diversifying search results. In: WSDM ’09: Proceedings of the second ACM international conference on web search and data mining. ACM, New York, pp 5–14. doi: 10.1145/1498759.1498766
  2. 2.
    Apache Software Foundation: Solr. (2010)
  3. 3.
    Bianchi T, De Rosa A, Piva A (2011) Improved dct coefficient analysis for forgery localization in jpeg images. In: IEEE international conference on acoustics, speech, and signal processing, IEEE. pp 2444–2447Google Scholar
  4. 4.
    Bianchi T, Piva A (2011) Detection of non-aligned double jpeg compression with estimation of primary compression parameters. In: IEEE international conference on image processingGoogle Scholar
  5. 5.
    Blanco R, Mika P, Atserias Batalla J, Matthews M, Tolchinsky P, Zaragoza H (2010) Searching through time in the New York Times. In: HCIR 2010.
  6. 6.
    Boato G, De Natale F, Zontone P (2010) How digital forensics may help assessing the perceptual impact of image formation and manipulation. In: Video processing and quality metrics for consumer electronicsGoogle Scholar
  7. 7.
    Boser B, Guyon I, Vapnik V (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on computational learning theory. Pittsburgh, United States, pp 144–152Google Scholar
  8. 8.
    Breck E, Choi Y, Cardie C (2007) Identifying expressions of opinion in context. In: IJCAI 2007, Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 2683–2688Google Scholar
  9. 9.
    Carbonell J, Goldstein J (1998) The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: SIGIR ’98: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 335–336. doi: 10.1145/290941.291025
  10. 10.
    Chapelle O, Metlzer D, Zhang Y, Grinspan P (2009) Expected reciprocal rank for graded relevance. In: Proceedings of the 18th ACM conference on information and knowledge management, CIKM ’09. ACM, New York, pp 621–630. doi: 10.1145/1645953.1646033
  11. 11.
    Choi Y, Breck E, Cardie C (2006) Joint extraction of entities and relations for opinion recognition. In: Proceedings of the 2006 conference on empirical methods in natural language processing. Sydney, Australia, pp 431–439Google Scholar
  12. 12.
    Christoudias C, Georgescu B, Meer P (2002) Synergism in low level vision. In: Proceedings of 16th international conference on Pattern recognition, 2002, vol. 4, pp 150–155. doi: 10.1109/ICPR.2002.1047421
  13. 13.
    Ciaramita M, Altun Y (2006) Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In: Processings of the 2006 conference on empirical methods in natural language processing. Sydney, Australia, pp 594–602Google Scholar
  14. 14.
    Clarke CL, Kolla M, Cormack GV, Vechtomova O, Ashkan A, Büttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 659–666. doi: 10.1145/1390334.1390446
  15. 15.
    Clarke CL, Kolla M, Vechtomova O (2009) An effectiveness measure for ambiguous and underspecified queries. In: Proceedings of the 2nd international conference on theory of information retrieval: advances in information retrieval theory, ICTIR ’09. Springer, Berlin, pp 188–199. doi: 10.1007/978-3-642-04417-5_17
  16. 16.
    Cunningham H, Maynard D, Bontcheva K, Tablan V, Aswani N, Roberts I, Gorrell G, Funk A, Roberts A, Damljanovic D, Heitz T, Greenwood MA, Saggion H, Petrak J, Li Y, Peters W (2011) Text processing with GATE (Version 6).
  17. 17.
    Drelie Gelasca E, Tomasic D, Ebrahimi T (2005) Which colors best catch your eyes: a subjective study of color saliency. In: Fisrt international workshop on video processing and quality metrics for consumer electronics, Scottsdale, Arizona, USA, ISCAS. SPIEGoogle Scholar
  18. 18.
    Farid H (2009) Exposing digital forgeries from JPEG ghosts. IEEE Trans Inf Forensics Secur 4:154–160CrossRefGoogle Scholar
  19. 19.
    Fontani M, Bianchi T, De Rosa A, Piva A, Barni M (2011) A Desmpter-Shafer framework for decision fusion in image forensics. In: IEEE international workshop on information forensics and securityGoogle Scholar
  20. 20.
    Gollapudi S, Sharma A (2009) An axiomatic approach for result diversification. In: WWW ’09: Proceedings of the 18th international conference on world wide web. ACM, New York, pp 381–390. doi: 10.1145/1526709.1526761
  21. 21.
    Hare J, Samangooei S, Dupplaw D, Lewis P (2012) Imageterrier: an extensible platform for scalable high-performance image retrieval. In: The ACM international conference on multimedia retrieval (ICMR 2012) (to appear)Google Scholar
  22. 22.
    Hare JS, Samangooei S, Dupplaw DP (2011) Openimaj and imageterrier: Java libraries and tools for scalable multimedia analysis and indexing of images. In: Proceedings of the 19th ACM international conference on Multimedia, MM ’11. ACM, New York, pp 691–694. doi: 10.1145/2072298.2072421
  23. 23.
    Hare JS, Samangooei S, Lewis PH (2011) Efficient clustering and quantisation of sift features: exploiting characteristics of the sift descriptor and interest region detectors under image inversion. In: Proceedings of the 1st ACM international conference on multimedia retrieval, ICMR ’11. ACM, New York , pp 2:1–2:8. doi: 10.1145/1991996.1991998
  24. 24.
    Johansson R (2009) Statistical bistratal dependency parsing. In: Proceedings of the 2009 conference on empirical methods in natural language processing. Singapore, pp 561–569Google Scholar
  25. 25.
    Johansson R, Moschitti A (2010) A flexible representation of heterogeneous annotation data. In: Proceedings of the seventh conference on international language resources and evaluation (LREC’10). Valetta, Malta, pp 3712–3715Google Scholar
  26. 26.
    Johansson R, Moschitti A (2010) Reranking models in fine-grained opinion analysis. In: Proceedings of the 23rd international conference of computational linguistics (Coling 2010). Beijing, China, pp 519–527Google Scholar
  27. 27.
    Johansson R, Moschitti A (2011) Extracting opinion expressions and their polarities—exploration of pipelines and joint models. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. Portland, United States, pp 101–106Google Scholar
  28. 28.
    Johansson R, Nugues P (2008) Dependency-based syntactic-semantic analysis with PropBank and NomBank. In: CoNLL 2008: Proceedings of the twelfth conference on natural language learning. Manchester, United Kingdom, pp 183–187Google Scholar
  29. 29.
    Karger D, Smith M (2009) Simile timeline.
  30. 30.
    Lin Z, He J, Tang X, Tang C (2009) Fast, automatic and fine-grained tampered JPEG image detection via DCT coefficient analysis. Pattern Recognit 42(11):2492–2501CrossRefzbMATHGoogle Scholar
  31. 31.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110. doi: 10.1023/B:VISI.0000029664.99615.94 CrossRefGoogle Scholar
  32. 32.
    Luo W, Qu Z, Huang J, Qiu G (2007) A novel method for detecting cropped and recompressed image blocks. In: IEEE conference on acoustics, speech, and signal processing (ICASSP)Google Scholar
  33. 33.
    Meyers A, Reeves R, Macleod C, Szekely R, Zielinska V, Young B, Grishman R (2004) The NomBank project: an interim report. In: HLT-NAACL 2004 workshop: frontiers in corpus annotation. Boston, United States, pp 24–31Google Scholar
  34. 34.
    Minack E, Siberski W, Nejdl W (2011) Incremental diversification for very large sets: a streaming-based approach. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, SIGIR ’11. ACM, New York, pp 585–594. doi: 10.1145/2009916.2009996
  35. 35.
    Moreau L, Clifford B, Freire J, Futrelle J, Gil Y, Groth P, Kwasnikowska N, Miles S, Missier P, Myers J, Plale B, Simmhan Y, Stephan E, den Bussche JV (2010) The open provenance model core specification (v1.1). Future generation computer systems.
  36. 36.
    Muratov O, Zontone P, Boato G, De Natale F (2011) A segment-based image saliency detection. In: IEEE conference on acoustics, speech, and signal processing (ICASSP)Google Scholar
  37. 37.
    Palmer M, Gildea D, Kingsbury P (2005) The proposition bank: an annotated corpus of semantic roles. Comput Linguist 31(1):71–106CrossRefGoogle Scholar
  38. 38.
    Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends Inf Retr 2(1–2):1–135CrossRefGoogle Scholar
  39. 39.
    Pavan M, Pelillo M (2003) A new graph-theoretic approach to clustering and segmentation. In: Proceedings of CVPR 2003, pp 145–152Google Scholar
  40. 40.
    Rocha A, Scheirer W, Boult T, Goldenstein S (2011) Vision of the unseen: current trends and challenges in digital image and video forensics. ACM Computing Surveys (CSUR) 43(4): Article id 26Google Scholar
  41. 41.
    Rosa AD, Uccheddu F, Costanzo A, Piva A, Barni M (2010) Exploring image dependencies: a new challenge in image forensics. In: Media forensics and security XII conference, SPIE electronic imaginingGoogle Scholar
  42. 42.
    Ruppenhofer J, Somasundaran S, Wiebe J (2008) Finding the sources and targets of subjective expressions. In: Proceedings of the sixth international language resources and evaluation (LREC’08). Marrakech, Morocco, pp 2781–2788Google Scholar
  43. 43.
    Sandeep K, Rajagopalan AN (2002) Human face detection in cluttered color images using skin color, edge information. In: Chaudhuri S, Zisserman A, Jain AK, Majumder KL (eds) ICVGIP. Allied Publishers Private Limited, Ahmadabad Google Scholar
  44. 44.
    Sandhaus E (2008) The New York times annotated corpus. LDC2008T19. Linguistic Data Consortium, University of PennsylvaniaGoogle Scholar
  45. 45.
    Shafer G (1976) A mathematical theory of evidence. Princeton University Press, PrincetonzbMATHGoogle Scholar
  46. 46.
    Sikora T (2001) The MPEG-7 visual standard for content description-an overview. IEEE Trans Circuits Syst Video Technol 11(6):696–702. doi: 10.1109/76.927422 CrossRefMathSciNetGoogle Scholar
  47. 47.
    Surdeanu M, Johansson R, Meyers A, Màrquez L, Nivre J (2008) The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In: Proceedings of CoNLL 2008. Manchester, United Kingdom, pp 159–177Google Scholar
  48. 48.
    Vailaya A, Jain A, Zhang H (1998) On image classification: city images vs. landscapes. Pattern Recognit 31(12):1921–1935. doi: 10.1016/S0031-3203(98)00079-X Google Scholar
  49. 49.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features.
  50. 50.
    Wang J, Zhu J (2009) Portfolio theory of information retrieval. In: SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, New York, pp 115–122. doi:  10.1145/1571941.1571963
  51. 51.
    Wiebe J, Wilson T, Cardie C (2005) Annotating expressions of opinions and emotions in language. Lang Res Eval 39(2–3):165–210CrossRefGoogle Scholar
  52. 52.
    Wilson T, Wiebe J, Hoffmann P (2009) Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Comput Linguist 35(3):399–433CrossRefGoogle Scholar
  53. 53.
    Yosef MA, Hoffart J, Bordino I, Spaniol M, Weikum G (2011) AIDA: accurate online disambiguation of named entities in text and tables. In: Proceedings of the 37th international conference on very large databases, VLDB 2011, pp 1450–1453Google Scholar
  54. 54.
    Zhai CX, Cohen WW, Lafferty J (2003) Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, SIGIR ’03. ACM, New York, pp 10–17. doi: 10.1145/860435.860440
  55. 55.
    Zontone P, Carli M, Boato G, De Natale F (2010) Impact of contrast modification on human feeling: an objective and subjective assessment. In: IEEE conference on image processing (ICIP)Google Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • David Paul Dupplaw
    • 1
  • Michael Matthews
    • 2
  • Richard Johansson
    • 3
  • Giulia Boato
    • 3
  • Andrea Costanzo
    • 4
  • Marco Fontani
    • 4
  • Enrico Minack
    • 5
  • Elena Demidova
    • 5
  • Roi Blanco
    • 2
  • Thomas Griffiths
    • 6
  • Paul Lewis
    • 1
  • Jonathon Hare
    • 1
  • Alessandro Moschitti
    • 3
  1. 1.University of SouthamptonSouthamptonUK
  2. 2.Barcelona MediaBarcelonaSpain
  3. 3.University of TrentoTrentoItaly
  4. 4.CNITFlorenceItaly
  5. 5.Leibniz Universität HannoverHannoverGermany
  6. 6.SORAViennaAustria

Personalised recommendations