A User Experience Model for Home Video Summarization

  • Wei-Ting Peng
  • Wei-Jia Huang
  • Wei-Ta Chu
  • Chien-Nan Chou
  • Wen-Yan Chang
  • Chia-Han Chang
  • Yi-Ping Hung
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5371)


In this paper, we propose a novel system for automatically summarizing home videos based on a user experience model. The user experience model takes account of user’s spontaneous behaviors when viewing videos. Based on users’ reaction when viewing videos, we can construct a systematic framework to automate video summarization. In this work, we analyze the variations of viewer’s eye movement and facial expression when he or she watching the raw home video. We transform these behaviors into the clues of determining the important part of each video shot. With the aids of music analysis, the developed system automatically generates a music video (MV) style summarized home videos. Experiments show that this new type of editing mechanism can effectively generate home video summaries and can largely reduce the efforts of manual summarization.


User experience model video summarization facial expression eye movement 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Peng, W.T., Chiang, Y.H., Chu, W.T., Huang, W.J., Chang, W.L., Huang, P.C., Hung, Y.P.: Aesthetics-based Automatic Home Video Skimming System. In: Satoh, S., Nack, F., Etoh, M. (eds.) MMM 2008. LNCS, vol. 4903, pp. 186–197. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  2. 2.
    Ma, Y.F., Hua, X.S., Lu, L., Zhang, H.J.: A generic framework of user attention model and its application in video summarization. IEEE Trans. on Multimedia 7(5), 907–919 (2005)CrossRefGoogle Scholar
  3. 3.
    Hanjalic, A.: Multimodal approach to measuring excitement in video. In: Proceedings of IEEE International Conference Multimedia and Expo. (2003)Google Scholar
  4. 4.
    Kleban, J., Sarkar, A., Moxley, E., Mangiat, S., Joshi, S., Kuo, T., Manjunath, B.S.: Feature fusion and redundancy pruning for rush video summarization. In: Proceedings of the international workshop on TRECVID video summarization (2007)Google Scholar
  5. 5.
    Mei, T., Hua, X.S., Zhou, H.Q., Li, S.: Modeling and mining of users’ capture intention for home videos. IEEE TMM 9(1), 66–77 (2007)Google Scholar
  6. 6.
    Goodman, R.M., McGrath, P.: Editing Digital Video: The Complete Creative and Technical Guide. McGraw-Hill/TAB Electronics (2002)Google Scholar
  7. 7.
    Chandler, G.: CUT BY CUT: Editing Your Film or Video, Michael Wiese (2006)Google Scholar
  8. 8.
    Zettl, H.: Sight, sound, motion: Applied media aesthetics, Wadsworth (1998)Google Scholar
  9. 9.
    Dorai, C., Venkatesh, S.: Computational media aesthetics: Finding meaning beautiful. IEEE Multimedia 8, 10–12 (2001)CrossRefGoogle Scholar
  10. 10.
    Mulhem, P., Kankanhalli, M.S., Hassan, H., Yi, J.: Pivot vector space approach for audio-video mixing. IEEE Multimedia IO(2), 28–40 (2003)CrossRefGoogle Scholar
  11. 11.
    Stelmach, L., Tam, W.J.: Processing image sequences based on eye movements. In: Proceedings of SPIE Human Vision, Visual Processing and Digital Display V, vol. 2179 (1994)Google Scholar
  12. 12.
    Tosi, V., Mecacci, L., Pasquali, E.: Scanning eye movements made when viewing film: preliminary observations. International Journal Neuroscience (1997)Google Scholar
  13. 13.
    Goldstein, R.B., Woods, R.L., Peli, E.: Where people look when watching movies: Do all viewers look at the same place? Computers in Biology and Medicine (2006)Google Scholar
  14. 14.
    Essa, I.A., Pentland, A.P.: Coding, Analysis, Interpretation, and Recognition of Facial Expressions. IEEE Trans. on PAMI 19(7), 757–763 (1997)CrossRefGoogle Scholar
  15. 15.
    Bartlett, M.S., Littlewort, G., Frank, M., Lainscsek, C.: Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 568–573 (2005)Google Scholar
  16. 16.
    Chang, W.Y., Chen, C.S., Hung, Y.P.: Analyzing Facial Expression by Fusing Manifolds. In: Proceedings of Asian Conference on Computer Vision Conference (2007)Google Scholar
  17. 17.
    Viola, P., Jones, M.J.: Robust real-time face detection. International Journal of Computer Vision 57(2), 137–154 (2004)CrossRefGoogle Scholar
  18. 18.
    Al-Oayedi, A., Clark, A.F.: An algorithm for face and facial-feature location based on gray-scale information and facial geometry. In: Proceedings of International Conference on Image Processing and Its Applications, vol. 2, pp. 625–629 (1999)Google Scholar
  19. 19.
    Sirohey, S., Rosenfeld, A.: Eye detection in a face image using linear and nonlinear filters. Pattern Recognition 34, 1367–1391 (2001)CrossRefzbMATHGoogle Scholar
  20. 20.
    Ekman, P., Friesen, W.V.: Unmasking the face. Prentice-Hall, Englewood Cliffs (1975)Google Scholar
  21. 21.
    Schwaninger, A., Wallraven, C., Cunningham, D.W., Chiller-Glaus, S.D.: Processing of identity and emotion in faces: a psychophysical, physiological and computational perspective. Progress in Brain Research 156, 321–343 (2006)CrossRefGoogle Scholar
  22. 22.
    Goldstein, R.B., Peli, E., Lerner, S., Luo, G.: Eye Movements While Watching a Video: Comparisons Across Viewer Groups. Vision Science Society (2004)Google Scholar
  23. 23.
    Klein, R.M., Pontefract, A.: Does oculomotor readiness mediate cognitive control of visual attention: Revisited! Attention and performance 15, 333–350 (1994)Google Scholar
  24. 24.
    Germeys, F., d’Ydewalle, G.: The psychology of film: perceiving beyond the cut. Psychological Research 71, 458–466 (2007)CrossRefGoogle Scholar
  25. 25.
    Liversedge, S.P., Findlay, J.M.: Saccadic eye movements and cognition. Trends in Cognition Sciences 4, 6–14 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Wei-Ting Peng
    • 1
  • Wei-Jia Huang
    • 2
  • Wei-Ta Chu
    • 3
  • Chien-Nan Chou
    • 2
  • Wen-Yan Chang
    • 2
  • Chia-Han Chang
    • 2
  • Yi-Ping Hung
    • 1
    • 2
  1. 1.Graduate Institute of Networking and MultimediaNational Taiwan UniversityTaipeiTaiwan
  2. 2.Department of Computer Science & Information EngineeringNational Taiwan UniversityTaipeiTaiwan
  3. 3.Department of Computer Science & Information EngineeringNational Chung Cheng UniversityChiayiTaiwan

Personalised recommendations