Advertisement

A Proposed Methodology for Subjective Evaluation of Video and Text Summarization

  • Begona Garcia-Zapirain
  • Cristian Castillo
  • Aritz Badiola
  • Sofia Zahia
  • Amaia MendezEmail author
  • David Langlois
  • Denis Jouvet
  • Juan-Manuel Torres
  • Mikołaj Leszczuk
  • Kamel Smaili
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 833)

Abstract

To evaluate a system that automatically summarizes video files (image and audio), it should be taken into account how the system works and which are the part of the process that should be evaluated, as two main topics to be evaluated can be differentiated: the video summary and the text summary. So, in the present article it is presented a complete way in order to evaluate this type of systems efficiently. With this objective, the authors have performed two types of evaluation: objective and subjective (the main focus of this paper). The objective evaluation is mainly done automatically, using established and proven metrics or frameworks, but it may need in some way the participation of humans, while the subjective evaluation is based directly on the opinion of people, who evaluate the system by answering a set of questions, which are then processed in order to obtain the targeted conclusions. The obtained general results from both evaluation systems will provide valuable information about the completeness and coherence, as well as the correctness of the generated summarizations from different points of view, as the lexical, semantical, etc. perspective. Apart from providing information about the state of the art, it will be presented an experimental proposal too, including the parameters of the experiment and the evaluation methods to be applied.

Keywords

Video summarization Objective and subjective evaluation Text summary 

Notes

Acknowledgements

Research work funded by the Spanish Ministry of Economy, Competitiveness and Industry (Spain) conferred under the Chist-Era AMIS project.

References

  1. 1.
    Truong, B.T., Venkatesh, S.: Video Abstraction: A Systematic Review and Classification (2007)Google Scholar
  2. 2.
    Fontes de Avila, S.E., Brandão Lopes, A., da Luz Jr, A., de Alburquerque Araújo, A.: VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method (2010)Google Scholar
  3. 3.
    Cayllahua Cahuina, E.J., Camara Chavez, G.: A New Method for Static Video Summarization Using Local Descriptors and Video Temporal Segmentation (2013)Google Scholar
  4. 4.
    Dumont, E., Bernard, M.: Automatic Evaluation Method for Rushes Summary Content (2009)Google Scholar
  5. 5.
    Money, A.G., Agius, H.: Video Summarisation: A Conceptual Framework and Survey of the State of the Art (2008)Google Scholar
  6. 6.
    Kannappan, S., Liu, Y., Tiddeman, B.: A Pertinent Evaluation of Automatic Video Summary (2016)Google Scholar
  7. 7.
    Li, Y., Merialdo, B.: Multi-video Summarization Based on Video-MMR (2010)Google Scholar
  8. 8.
    Li, Y., Merialdo, B.: VERT: Automatic Evaluation of Video Summaries (2010)Google Scholar
  9. 9.
    Mohamed, K.M., Ismail, M.A., Ghanem, N.M.: VSCAN: An Enhanced Video Summarization using Density-based Spatial Clustering (2014)Google Scholar
  10. 10.
    Molina, A., Torres-Moreno, J.-M.: The Turing Test for Automatic Text Summarization Evaluation (2016)Google Scholar
  11. 11.
    Molina Villegas, A., Torres-Moreno, J.-M., Sanjuan, E.: A Turing Test to Evaluate a Complex Summarization Task (2013)Google Scholar
  12. 12.
    Lin, C.-Y., Hovy, E.: Manual and Automatic Evaluation of Summaries (2002)Google Scholar
  13. 13.
    Hassel, M.: Evaluation of Automatic Text Summarization: a practical implementation (2004)Google Scholar
  14. 14.
    Saziyabegum, S., Sajja, P.S.: Review on Text Summarization Evaluation Method (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Begona Garcia-Zapirain
    • 1
  • Cristian Castillo
    • 1
  • Aritz Badiola
    • 1
  • Sofia Zahia
    • 1
  • Amaia Mendez
    • 1
    Email author
  • David Langlois
    • 2
  • Denis Jouvet
    • 2
  • Juan-Manuel Torres
    • 3
  • Mikołaj Leszczuk
    • 4
  • Kamel Smaili
    • 2
  1. 1.eVida Research GroupUniversity of DeustoBilbaoSpain
  2. 2.Loria, University of LorraineLorraineFrance
  3. 3.LIA, Universite d’Avignon et des Pays de VaucluseAvignonFrance
  4. 4.AGH University of Science and Technology KrakówKrakówPoland

Personalised recommendations