Skip to main content
SpringerLink
Log in

Information extraction from multimedia web documents: an open-source platform and testbed

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

The LivingKnowledge project aimed to enhance the current state of the art in search, retrieval and knowledge management on the web by advancing the use of sentiment and opinion analysis within multimedia applications. To achieve this aim, a diverse set of novel and complementary analysis techniques have been integrated into a single, but extensible software platform on which such applications can be built. The platform combines state-of-the-art techniques for extracting facts, opinions and sentiment from multimedia documents, and unlike earlier platforms, it exploits both visual and textual techniques to support multimedia information retrieval. Foreseeing the usefulness of this software in the wider community, the platform has been made generally available as an open-source project. This paper describes the platform design, gives an overview of the analysis algorithms integrated into the system and describes two applications that utilise the system for multimedia information retrieval.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. http://diversityengine.org.

  2. http://solr.apache.org/.

  3. The Diversty Engine documentation contains speed estimates for each analysis tool.

  4. http://hadoop.apache.org.

  5. https://sites.google.com/site/hcirworkshop/hcir-2010/challenge.

  6. http://opennlp.sourceforge.net.

  7. http://www.timeml.org/site/tarsqi/.

  8. http://www.timeml.org/site/index.html.

  9. Now available as OpenSource through the OpenIMAJ project at http://openimaj.org/.

  10. http://trec.nist.gov/data/web10.html.

  11. http://fbmya01.barcelonamedia.org:8080/future/.

References

  1. Agrawal R, Gollapudi S, Halverson A, Ieong S (2009) Diversifying search results. In: WSDM ’09: Proceedings of the second ACM international conference on web search and data mining. ACM, New York, pp 5–14. doi:10.1145/1498759.1498766

  2. Apache Software Foundation: Solr. (2010) http://lucene.apache.org/solr/

  3. Bianchi T, De Rosa A, Piva A (2011) Improved dct coefficient analysis for forgery localization in jpeg images. In: IEEE international conference on acoustics, speech, and signal processing, IEEE. pp 2444–2447

  4. Bianchi T, Piva A (2011) Detection of non-aligned double jpeg compression with estimation of primary compression parameters. In: IEEE international conference on image processing

  5. Blanco R, Mika P, Atserias Batalla J, Matthews M, Tolchinsky P, Zaragoza H (2010) Searching through time in the New York Times. In: HCIR 2010. http://www.docstoc.com/docs/96068142/Searching-through-time-in-the-New-York-Times

  6. Boato G, De Natale F, Zontone P (2010) How digital forensics may help assessing the perceptual impact of image formation and manipulation. In: Video processing and quality metrics for consumer electronics

  7. Boser B, Guyon I, Vapnik V (1992) A training algorithm for optimal margin classifiers. In: Proceedings of the fifth annual workshop on computational learning theory. Pittsburgh, United States, pp 144–152

  8. Breck E, Choi Y, Cardie C (2007) Identifying expressions of opinion in context. In: IJCAI 2007, Proceedings of the 20th international joint conference on artificial intelligence. Hyderabad, India, pp 2683–2688

  9. Carbonell J, Goldstein J (1998) The use of mmr, diversity-based reranking for reordering documents and producing summaries. In: SIGIR ’98: Proceedings of the 21st annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 335–336. doi:10.1145/290941.291025

  10. Chapelle O, Metlzer D, Zhang Y, Grinspan P (2009) Expected reciprocal rank for graded relevance. In: Proceedings of the 18th ACM conference on information and knowledge management, CIKM ’09. ACM, New York, pp 621–630. doi:10.1145/1645953.1646033

  11. Choi Y, Breck E, Cardie C (2006) Joint extraction of entities and relations for opinion recognition. In: Proceedings of the 2006 conference on empirical methods in natural language processing. Sydney, Australia, pp 431–439

  12. Christoudias C, Georgescu B, Meer P (2002) Synergism in low level vision. In: Proceedings of 16th international conference on Pattern recognition, 2002, vol. 4, pp 150–155. doi:10.1109/ICPR.2002.1047421

  13. Ciaramita M, Altun Y (2006) Broad-coverage sense disambiguation and information extraction with a supersense sequence tagger. In: Processings of the 2006 conference on empirical methods in natural language processing. Sydney, Australia, pp 594–602

  14. Clarke CL, Kolla M, Cormack GV, Vechtomova O, Ashkan A, Büttcher S, MacKinnon I (2008) Novelty and diversity in information retrieval evaluation. In: SIGIR ’08: Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval. ACM, New York, pp 659–666. doi:10.1145/1390334.1390446

  15. Clarke CL, Kolla M, Vechtomova O (2009) An effectiveness measure for ambiguous and underspecified queries. In: Proceedings of the 2nd international conference on theory of information retrieval: advances in information retrieval theory, ICTIR ’09. Springer, Berlin, pp 188–199. doi:10.1007/978-3-642-04417-5_17

  16. Cunningham H, Maynard D, Bontcheva K, Tablan V, Aswani N, Roberts I, Gorrell G, Funk A, Roberts A, Damljanovic D, Heitz T, Greenwood MA, Saggion H, Petrak J, Li Y, Peters W (2011) Text processing with GATE (Version 6). http://tinyurl.com/gatebook

  17. Drelie Gelasca E, Tomasic D, Ebrahimi T (2005) Which colors best catch your eyes: a subjective study of color saliency. In: Fisrt international workshop on video processing and quality metrics for consumer electronics, Scottsdale, Arizona, USA, ISCAS. SPIE

  18. Farid H (2009) Exposing digital forgeries from JPEG ghosts. IEEE Trans Inf Forensics Secur 4:154–160

    Article  Google Scholar 

  19. Fontani M, Bianchi T, De Rosa A, Piva A, Barni M (2011) A Desmpter-Shafer framework for decision fusion in image forensics. In: IEEE international workshop on information forensics and security

  20. Gollapudi S, Sharma A (2009) An axiomatic approach for result diversification. In: WWW ’09: Proceedings of the 18th international conference on world wide web. ACM, New York, pp 381–390. doi:10.1145/1526709.1526761

  21. Hare J, Samangooei S, Dupplaw D, Lewis P (2012) Imageterrier: an extensible platform for scalable high-performance image retrieval. In: The ACM international conference on multimedia retrieval (ICMR 2012) (to appear)

  22. Hare JS, Samangooei S, Dupplaw DP (2011) Openimaj and imageterrier: Java libraries and tools for scalable multimedia analysis and indexing of images. In: Proceedings of the 19th ACM international conference on Multimedia, MM ’11. ACM, New York, pp 691–694. doi:10.1145/2072298.2072421

  23. Hare JS, Samangooei S, Lewis PH (2011) Efficient clustering and quantisation of sift features: exploiting characteristics of the sift descriptor and interest region detectors under image inversion. In: Proceedings of the 1st ACM international conference on multimedia retrieval, ICMR ’11. ACM, New York , pp 2:1–2:8. doi:10.1145/1991996.1991998

  24. Johansson R (2009) Statistical bistratal dependency parsing. In: Proceedings of the 2009 conference on empirical methods in natural language processing. Singapore, pp 561–569

  25. Johansson R, Moschitti A (2010) A flexible representation of heterogeneous annotation data. In: Proceedings of the seventh conference on international language resources and evaluation (LREC’10). Valetta, Malta, pp 3712–3715

  26. Johansson R, Moschitti A (2010) Reranking models in fine-grained opinion analysis. In: Proceedings of the 23rd international conference of computational linguistics (Coling 2010). Beijing, China, pp 519–527

  27. Johansson R, Moschitti A (2011) Extracting opinion expressions and their polarities—exploration of pipelines and joint models. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. Portland, United States, pp 101–106

  28. Johansson R, Nugues P (2008) Dependency-based syntactic-semantic analysis with PropBank and NomBank. In: CoNLL 2008: Proceedings of the twelfth conference on natural language learning. Manchester, United Kingdom, pp 183–187

  29. Karger D, Smith M (2009) Simile timeline. http://www.simile-widgets.org/timeline/

  30. Lin Z, He J, Tang X, Tang C (2009) Fast, automatic and fine-grained tampered JPEG image detection via DCT coefficient analysis. Pattern Recognit 42(11):2492–2501

    Article  MATH  Google Scholar 

  31. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60(2):91–110. doi:10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  32. Luo W, Qu Z, Huang J, Qiu G (2007) A novel method for detecting cropped and recompressed image blocks. In: IEEE conference on acoustics, speech, and signal processing (ICASSP)

  33. Meyers A, Reeves R, Macleod C, Szekely R, Zielinska V, Young B, Grishman R (2004) The NomBank project: an interim report. In: HLT-NAACL 2004 workshop: frontiers in corpus annotation. Boston, United States, pp 24–31

  34. Minack E, Siberski W, Nejdl W (2011) Incremental diversification for very large sets: a streaming-based approach. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, SIGIR ’11. ACM, New York, pp 585–594. doi:10.1145/2009916.2009996

  35. Moreau L, Clifford B, Freire J, Futrelle J, Gil Y, Groth P, Kwasnikowska N, Miles S, Missier P, Myers J, Plale B, Simmhan Y, Stephan E, den Bussche JV (2010) The open provenance model core specification (v1.1). Future generation computer systems. http://eprints.ecs.soton.ac.uk/21449/

  36. Muratov O, Zontone P, Boato G, De Natale F (2011) A segment-based image saliency detection. In: IEEE conference on acoustics, speech, and signal processing (ICASSP)

  37. Palmer M, Gildea D, Kingsbury P (2005) The proposition bank: an annotated corpus of semantic roles. Comput Linguist 31(1):71–106

    Article  Google Scholar 

  38. Pang B, Lee L (2008) Opinion mining and sentiment analysis. Found Trends Inf Retr 2(1–2):1–135

    Article  Google Scholar 

  39. Pavan M, Pelillo M (2003) A new graph-theoretic approach to clustering and segmentation. In: Proceedings of CVPR 2003, pp 145–152

  40. Rocha A, Scheirer W, Boult T, Goldenstein S (2011) Vision of the unseen: current trends and challenges in digital image and video forensics. ACM Computing Surveys (CSUR) 43(4): Article id 26

  41. Rosa AD, Uccheddu F, Costanzo A, Piva A, Barni M (2010) Exploring image dependencies: a new challenge in image forensics. In: Media forensics and security XII conference, SPIE electronic imagining

  42. Ruppenhofer J, Somasundaran S, Wiebe J (2008) Finding the sources and targets of subjective expressions. In: Proceedings of the sixth international language resources and evaluation (LREC’08). Marrakech, Morocco, pp 2781–2788

  43. Sandeep K, Rajagopalan AN (2002) Human face detection in cluttered color images using skin color, edge information. In: Chaudhuri S, Zisserman A, Jain AK, Majumder KL (eds) ICVGIP. Allied Publishers Private Limited, Ahmadabad

  44. Sandhaus E (2008) The New York times annotated corpus. LDC2008T19. Linguistic Data Consortium, University of Pennsylvania

  45. Shafer G (1976) A mathematical theory of evidence. Princeton University Press, Princeton

    MATH  Google Scholar 

  46. Sikora T (2001) The MPEG-7 visual standard for content description-an overview. IEEE Trans Circuits Syst Video Technol 11(6):696–702. doi:10.1109/76.927422

    Article  MathSciNet  Google Scholar 

  47. Surdeanu M, Johansson R, Meyers A, Màrquez L, Nivre J (2008) The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In: Proceedings of CoNLL 2008. Manchester, United Kingdom, pp 159–177

  48. Vailaya A, Jain A, Zhang H (1998) On image classification: city images vs. landscapes. Pattern Recognit 31(12):1921–1935. doi:10.1016/S0031-3203(98)00079-X

    Google Scholar 

  49. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.3.7597

  50. Wang J, Zhu J (2009) Portfolio theory of information retrieval. In: SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. ACM, New York, pp 115–122. doi: 10.1145/1571941.1571963

  51. Wiebe J, Wilson T, Cardie C (2005) Annotating expressions of opinions and emotions in language. Lang Res Eval 39(2–3):165–210

    Article  Google Scholar 

  52. Wilson T, Wiebe J, Hoffmann P (2009) Recognizing contextual polarity: An exploration of features for phrase-level sentiment analysis. Comput Linguist 35(3):399–433

    Article  Google Scholar 

  53. Yosef MA, Hoffart J, Bordino I, Spaniol M, Weikum G (2011) AIDA: accurate online disambiguation of named entities in text and tables. In: Proceedings of the 37th international conference on very large databases, VLDB 2011, pp 1450–1453

  54. Zhai CX, Cohen WW, Lafferty J (2003) Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, SIGIR ’03. ACM, New York, pp 10–17. doi:10.1145/860435.860440

  55. Zontone P, Carli M, Boato G, De Natale F (2010) Impact of contrast modification on human feeling: an objective and subjective assessment. In: IEEE conference on image processing (ICIP)

Download references

Acknowledgments

This work was supported by the European Union under the Seventh Framework project LivingKnowledge (IST-FP7-231126). We would also like to thank all our partners who contributed to the LivingKnowledge project on which this work has been built.

Author information

Authors and Affiliations

  1. University of Southampton, Southampton, UK

    David Paul Dupplaw, Paul Lewis & Jonathon Hare

  2. Barcelona Media, Barcelona, Spain

    Michael Matthews & Roi Blanco

  3. University of Trento, Trento, Italy

    Richard Johansson, Giulia Boato & Alessandro Moschitti

  4. CNIT, Florence, Italy

    Andrea Costanzo & Marco Fontani

  5. Leibniz Universität Hannover, Hannover, Germany

    Enrico Minack & Elena Demidova

  6. SORA, Vienna, Austria

    Thomas Griffiths

Authors
  1. David Paul Dupplaw
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michael Matthews
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Johansson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Giulia Boato
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrea Costanzo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Marco Fontani
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Enrico Minack
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Elena Demidova
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Roi Blanco
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Thomas Griffiths
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Paul Lewis
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Jonathon Hare
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Alessandro Moschitti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to David Paul Dupplaw.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Dupplaw, D.P., Matthews, M., Johansson, R. et al. Information extraction from multimedia web documents: an open-source platform and testbed. Int J Multimed Info Retr 3, 97–111 (2014). https://doi.org/10.1007/s13735-014-0051-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13735-014-0051-2

Keywords

Search

Navigation