Multimedia Tools and Applications

, Volume 75, Issue 20, pp 12597–12625 | Cite as

Scalable storyboards in handheld devices: applications and evaluation metrics

  • Luis HerranzEmail author
  • Shuqiang Jiang


Summaries are an essential component of video retrieval and browsing systems. Most research in video summarization has focused on content analysis to obtain compact yet comprehensive representations of video items. However, important aspects such as how they can be effectively integrated in mobile interfaces and how to predict the quality and usability of the summaries have not been investigated. Conventional summaries are limited to a single instance with certain length (i.e. a single scale). In contrast, scalable summaries target representations with multiple scales, that is, a set of summaries with increasing length in which longer summaries include more information about the video. Thus, scalability provides high flexibility that can be exploited in devices such as smartphones or tablets to provide versions of the summary adapted to the limited visualization area. In this paper, we explore the application of scalable storyboards to summary adaptation and zoomable video navigation in handheld devices. By introducing a new adaptation dimension related with the summarization scale, we can formulate navigation and adaptation in a two-dimensional adaptation space, where different navigation actions modify the trajectory in that space. We also describe the challenges to evaluate scalable summaries and some usability issues that arise from having multiple scales, proposing some objective metrics that can provide useful insight about their potential quality and usability without requiring very costly user studies. Experimental results show a reasonable agreement with the trends shown in subjective evaluations. Experiments also show that content-based scalable storyboards are less redundant and useful than the content-blind baselines.


Scalable storyboards Video browsing Usability Quality metrics 



This work was supported in part by the National Basic Research Program of China (973 Program): 2012CB316400, in part by the National Natural Science Foundation of China: 61322212 and 61350110237, in part by the National Hi-Tech Development Program (863 Program) of China: 2014AA015202, and in part by the Chinese Academy of Sciences Fellowships for Young International Scientists: 2011Y1GB05. This work was also funded by Lenovo Outstanding Young Scientists Program (LOYS).


  1. 1.
    Adami N, Signoroni A, Leonardi R (2007) State-of-the-art and trends in scalable video compression with wavelet-based approaches. IEEE Trans Circ Systx Video Technol 17(9):1238–1255CrossRefGoogle Scholar
  2. 2.
    Ahmad I, Wei X, Sun Y, Zhang YQ (2005) Video transcoding: an overview of various techniques and research issues. IEEE Trans Multimed 7(5):793–804CrossRefGoogle Scholar
  3. 3.
    Albanese M, Fayzullin M, Picariello A, Subrahmanian V (2006) The priority curve algorithm for video summarization. Inf Syst 31(7):679–695CrossRefGoogle Scholar
  4. 4.
    Benini S, Bianchetti A, Leonardi R, Migliorati P (2006) Extraction of significant video summaries by dendrogram analysis. In: Proceedings IEEE International Conference on Image Processing, pp 133–136Google Scholar
  5. 5.
    Benini S, Migliorati P, Leonardi R (2010) Statistical skimming of feature films. Int J Digit Multimed Broadcast 2010:11Google Scholar
  6. 6.
    Bescos J, Martinez JM, Herranz L, Tiburzi F (2007) Content-driven adaptation of on-line video. Signal Process: Image Commun 22:651–668Google Scholar
  7. 7.
    Chang SF, Vetro A (2005) Video adaptation: concepts, technologies, and open issues. IEEE Proc 93(1):148–158CrossRefGoogle Scholar
  8. 8.
    Cong Y., Yuan J., Luo J. (2012) Towards scalable summarization of consumer videos via sparse dictionary selection. IEEE Trans Multimed 14(1):66–75CrossRefGoogle Scholar
  9. 9.
    Dong P, Xia Y, Wang S, Zhuo L, Feng D (2014) An iteratively reweighting algorithm for dynamic video summarization Multimedia Tools and Applications, pp 1–25. doi: 10.1007/s11042-014-2126-8
  10. 10.
    Dumont E, Merialdo B (2010) Rushes video summarization and evaluation. Multimed Tools Appl 48:51–68CrossRefGoogle Scholar
  11. 11.
    Friedland G, Gottlieb L, Janin A (2013) Narrative theme navigation for sitcoms supported by fan-generated scripts. Multimed Tools Appl 63(2):387–406. doi: 10.1007/s11042-011-0877-z CrossRefGoogle Scholar
  12. 12.
    Gong Y, Liu X (2000) Video summarization using singular value decomposition. In: Proceedings IEEE Conference Computer Vision and Pattern Recognition, vol 2, pp 174–180Google Scholar
  13. 13.
    Haesen M, Meskens J, Luyten K, Coninx K, Becker J, Tuytelaars T, Poulisse GJ, Pham P, Moens MF (2013) Finding a needle in a haystack: an interactive video archive explorer for professional video searchers. Multimed Tools Appl 63(2):331–356CrossRefGoogle Scholar
  14. 14.
    Hanjalic A, Zhang H (1999) An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis. IEEE Trans Circ Syst Video Technol 9(8):1280–1289CrossRefGoogle Scholar
  15. 15.
    Herranz L, Calic J, Martínez JM, Mrak M (2012) Scalable comic-like video summaries and layout disturbance. IEEE Trans Multimed 14(4):1290–1297CrossRefGoogle Scholar
  16. 16.
    Herranz L, Martinez JM (2008) Generation of scalable summaries based on iterative GOP ranking. In: Proceedings IEEE International Conference on Image Processing, pp 2544–2547Google Scholar
  17. 17.
    Herranz L, Martínez JM (2009) An integrated approach to summarization and adaptation using H.264/MPEG-4 SVC. Signal Process: Image Commun 24(6):499–509Google Scholar
  18. 18.
    Herranz L, Martínez JM (2010) A framework for scalable summarization of video. IEEE Trans Circ Syst Video Technol 20(9):1265–1270CrossRefGoogle Scholar
  19. 19.
    Hürst W, Darzentas D (2012) History: a hierarchical storyboard interface design for video browsing on mobile devices. In: Proceedings International Conference on Mobile and Ubiquitous Multimedia, pp 17:1–17:4Google Scholar
  20. 20.
    Irie G, Satou T, Kojima A, Yamasaki T, Aizawa K (2010) Automatic trailer generation. In: Proceedings of the International Conference on Multimedia, MM ’10. ACM, NY, USA, pp 839–842CrossRefGoogle Scholar
  21. 21.
    Li Y, Merialdo B (2010) Vert: automatic evaluation of video summaries. In: Proceedings of the international conference on Multimedia, pp 851–854Google Scholar
  22. 22.
    Likert R (1932) A technique for the measurement of attitudes. Arch Psychol 22 (140):1–55Google Scholar
  23. 23.
    Liu H, Liu Y, Sun F (2014) Video key-frame extraction for smart phones. Multimed Tools Appl:1–19. doi: 10.1007/s11042-014-2390-7
  24. 24.
    Marchionini G, Wildemuth BM, Geisler G (2006) The Open Video digital library: A M?bius strip of research and practice. J Am Soc Inf Sci Technol 57(12):1629–1643CrossRefGoogle Scholar
  25. 25.
    Mohanta PP, Saha SK, Chanda B (2009) Generation of size constrained video storyboard using spanning tree. In: Proceedings of the First International Conference on Internet Multimedia Computing and Service, pp 179–182Google Scholar
  26. 26.
    Money AG, Agius H (2008) Video summarisation: A conceptual framework and survey of the state of the art. J Vis Commun Image Represent 19(2):121–143CrossRefGoogle Scholar
  27. 27.
    Mukherjee D, Said A, Liu S (2005) A framework for fully format-independent adaptation of scalable bit streams. IEEE Trans Circ Syst Video Technol 15(10):1280–1290CrossRefGoogle Scholar
  28. 28.
    Mundur P, Rao Y, Yesha Y (2006) Keyframe-based video summarization using delaunay clustering. Int J Digit Libr 6(2):219–232CrossRefGoogle Scholar
  29. 29.
    Ohm JR (2005) Advances in scalable video coding. IEEE Proc 93(1):42–56CrossRefGoogle Scholar
  30. 30.
    Over P, Smeaton AF, Awad G (2008) The TRECVid 2008 BBC rushes summarization evaluation. In: Proceedings 2nd ACM TRECVid Video Summarization Workshop. ACM, pp 1–20Google Scholar
  31. 31.
    Over P, Smeaton AF, Kelly P The TRECVid 2007 bbc rushes summarization evaluation pilot. In: TRECVidGoogle Scholar
  32. 32.
    Santini S (2007) Who needs video summarization anyway? In: Proceedings International Conference Semantic Computing ICSC 2007, pp 177–184Google Scholar
  33. 33.
    Scherp A, Mezaris V (2014) Survey on modeling and indexing events in multimedia. Multimed Tools Appl 70(1):7–23. doi: 10.1007/s11042-013-1427-7 CrossRefGoogle Scholar
  34. 34.
    Schoeffmann K (2014) A user-centric media retrieval competition: The video browser showdown 2012-2014. IEEE MultiMedia 21(4):8–13. doi: 10.1109/MMUL.2014.56 CrossRefGoogle Scholar
  35. 35.
    Schoeffmann K, Ahlstrom D, Hudelist M (2014) 3-d interfaces to improve the performance of visual known-item search. Multimed, IEEE Trans 16(7):1942–1951. doi: 10.1109/TMM.2014.2333666 CrossRefGoogle Scholar
  36. 36.
    Schwarz H, Marpe D, Wiegand T (2007) Overview of the scalable video coding extension of the H.264/AVC standard. IEEE Trans Circ Syst Video Technol 17(9):1103–1120CrossRefGoogle Scholar
  37. 37.
    Sibson R (1973) Slink: an optimally efficient algorithm for the single-link cluster method. Comput J 16(1):30–34MathSciNetCrossRefGoogle Scholar
  38. 38.
    Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: Proceedings of the ACM international workshop on Multimedia information retrieval. ACM, pp 321–330Google Scholar
  39. 39.
    Smeulders A, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Trans Pattern Anal Mach Intell 22(12):1349–1380CrossRefGoogle Scholar
  40. 40.
    Truong BT, Venkatesh S (2007) Video abstraction: A systematic review and classification. ACM Trans Multimed Comput Commun Appl 3(1):3CrossRefGoogle Scholar
  41. 41.
    Valdes V, Martinez JM (2012) Automatic evaluation of video summaries. ACM Trans Multimed Comput Commun Appl 8(3):25:1–25:21CrossRefGoogle Scholar
  42. 42.
    Vetro A (2004) MPEG-21 digital item adaptation: Enabling universal multimedia access. IEEE Multimed 11(1):84–87CrossRefGoogle Scholar
  43. 43.
    Xin J, Lin CW, Sun MT (2005) Digital video transcoding. IEEE Proc 93(1):84–97CrossRefGoogle Scholar
  44. 44.
    Yang Y, Ma Z, Xu Z, Yan S, Hauptmann AG (2013) How related exemplars help complex event detection in web videos? In: IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1–8, pp 2104–2111Google Scholar
  45. 45.
    Yuan Z, Lu T, Wu D, Huang Y, Yu H (2011) Video summarization with semantic concept preservation. In: Proceedings International Conference on Mobile and Ubiquitous Multimedia. ACM, pp 109–112Google Scholar
  46. 46.
    Zhang L, Gao Y, Hong C, Feng Y, Zhu J, Cai D (2014) Feature correlation hypergraph: Exploiting high-order potentials for multimodal recognition. Cybern, IEEE Trans 44(8):1408–1419. doi: 10.1109/TCYB.2013.2285219 CrossRefGoogle Scholar
  47. 47.
    Zhao S, Yao H, Sun X (2013) Video classification and recommendation based on affective analysis of viewers. Neurocomputing 119:101–110CrossRefGoogle Scholar
  48. 48.
    Zhao S, Yao H, Sun X, Jiang X, Xu P (2013) Flexible presentation of videos based on affective content analysis. In: Li S, El Saddik A, Wang M, Mei T, Sebe N, Yan S, Hong R, Gurrin C (eds) Proceedings International Conference on Multimedia Modeling, Lecture Notes in Computer Science, vol 7732, pp 368–379Google Scholar
  49. 49.
    Zhu X, Fan J, Elmagarmid AK, Wu X (2003) Hierarchical video content description and summarization using unified semantic and visual similarity. Multimed Syst 9(1):31–53CrossRefGoogle Scholar
  50. 50.
    Zhu XQ, Wu XD, Fan JP, Elmagarmid AK, Aref WF (2004) Exploring video content structure for hierarchical summarization. Multimed Syst 10(2):98–115CrossRefGoogle Scholar
  51. 51.
    Zhuang Y, Rui Y, Huang T, Mehrotra S (1998) Adaptive key frame extraction using unsupervised clustering. In: Proceedings of International Conference on Image Processing, pp 866–870Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of SciencesBeijingChina

Personalised recommendations