An Unsupervised Ensemble Approach for Emotional Scene Detection from Lifelog Videos

  • Hiroki NomiyaEmail author
  • Atsushi Morikuni
  • Teruhisa Hochin
Part of the Studies in Computational Intelligence book series (SCI, volume 569)


An emotional scene detection method is proposed in order to retrieve impressive scenes from lifelog videos. The proposed method is based on facial expression recognition considering that a wide variety of facial expression could be observed in impressive scenes. Most of conventional facial expression techniques adopt supervised learning methods. This is a crucial problem because preparing sufficient training data requires considerable human effort due to the diversity of facial expressions observed in lifelog videos. We thus propose a more efficient emotional scene detection method using an unsupervised facial expression recognition on the basis of cluster ensembles. Our approach does not require any training data sets and is able to detect various emotional scenes. The detection performance of the proposed method is evaluated through an emotional scene detection experiment.


Lifelog video retrieval facial expression recognition clustering ensemble learning 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aizawa, K., Hori, T., Kawasaki, S., Ishikawa, T.: Capture and Efficient Retrieval of Life Log. In: Proc. of Pervasive 2004 Workshop on Memory and Sharing Experiences, pp. 15–20 (2004)Google Scholar
  2. 2.
    Gemmell, J., Bell, G., Luederand, R., Drucker, S., Wong, C.: MyLifeBits: Fulfilling the Memex Vision. In: Proc. of the 10th ACM International Conference on Multimedia, pp. 235–238 (2002)Google Scholar
  3. 3.
    Datchakorn, T., Yamasaki, T., Aizawa, K.: Practical Experience Recording and Indexing of Life Log Video. In: Proc. of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences, pp. 61–66 (2005)Google Scholar
  4. 4.
    Mehrabian, A.: Silent Messages. Wadsworth, Belmont (1971)Google Scholar
  5. 5.
    Datcu, D., Rothkrantz, L.: Facial Expression Recognition in Still Pictures and Videos Using Active Appearance Models: A Comparison Approach. In: Proc. of the 2007 International Conference on Computer Systems and Technologies, pp. 1–6 (2007)Google Scholar
  6. 6.
    Fanelli, G., Yao, A., Noel, P.-L., Gall, J., Van Gool, L.: Hough Forest-based Facial Expression Recognition from Video Sequences. In: Kutulakos, K.N. (ed.) ECCV 2010 Workshops, Part I. LNCS, vol. 6553, pp. 195–206. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  7. 7.
    Littlewort, G., Bartlett, M.S., Fasel, I., Susskind, J., Movellan, J.: Dynamics of Facial Expression Extracted Automatically from Video. Journal of Image and Vision Computing 24(6), 615–625 (2004)CrossRefGoogle Scholar
  8. 8.
    Strehl, A., Ghosh, J.: Cluster Ensembles – a Knowledge Reuse Framework for Combining Multiple Partitions. Journal of Machine Learning Research 3, 583–617 (2003)zbMATHMathSciNetGoogle Scholar
  9. 9.
    Tian, Y., Kanade, T., Cohn, J.F.: Facial Expression Recognition. In: Li, S.Z., Jain, A.K. (eds.) Handbook of Face Recognition, pp. 487–519. Springer, London (2011)CrossRefGoogle Scholar
  10. 10.
    Lyons, M., Akamatsu, S.: Coding Facial Expressions with Gabor Wavelets. In: Proc. of the 3rd International Conference on Automatic Face and Gesture Recognition, pp. 200–205 (1998)Google Scholar
  11. 11.
    Feng, X., Pietikäinen, M.: Facial Expression Recognition with Local Binary Patterns and Linear Programming. Pattern Recognition and Image Analysis 15(2), 546–548 (2005)Google Scholar
  12. 12.
    Wang, J., Yin, L., Wei, X., Sun, Y.: 3D Facial Expression Recognition Based on Primitive Surface Feature Distribution. In: Proc. of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1399–1406 (2006)Google Scholar
  13. 13.
    Soyel, H., Demirel, H.O.: Facial Expression Recognition Using 3D Facial Feature Distances. In: Kamel, M.S., Campilho, A. (eds.) ICIAR 2007. LNCS, vol. 4633, pp. 831–838. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  14. 14.
    Esau, N., Wetzel, E., Kleinjohann, L., Kleinjohann, B.: Real-time Facial Expression Recognition Using a Fuzzy Emotion Model. In: Proc. of International Conference on Fuzzy Systems, pp. 1–6 (2007)Google Scholar
  15. 15.
    Hupont, I., Cerezo, E., Baldassarri, S.: Sensing Facial Emotion in a Continuous 2D Affective Space. In: Proc. of International Conference on Systems, Man, and Cybernetics, pp. 2045–2051 (2010)Google Scholar
  16. 16.
    Nomiya, H., Morikuni, A., Hochin, T.: Emotional Video Scene Detection from Lifelog Videos using Facial Feature Selection. In: Proc. of 4th International Conference on Applied Human Factors and Ergonomics, pp. 8500–8509 (2012)Google Scholar
  17. 17.
    Hoey, J.: Hierarchical Unsupervised Learning of Facial Expression Categories. In: Proc. of IEEE Workshop on Detection and Recognition of Events in Video, pp. 99–106 (2001)Google Scholar
  18. 18.
    Gholami, B., Haddad, W.M., Tannenbaum, A.R.: An Unsupervised Learning Approach for Facial Expression Recognition Using Semi-Definite Programming and Generalized Principal Component Analysis. In: Proc. of Image Processing: Algorithms and Systems, pp. 1–10 (2010)Google Scholar
  19. 19.
    Nomiya, H., Morikuni, A., Hochin, T.: Impressive Scene Detection from Lifelog Videos by Unsupervised Facial Expression Recognition. In: Proc. of 14th International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, pp. 444–449 (2013)Google Scholar
  20. 20.
    Luxand Inc., Luxand FaceSDK 4.0, (April 18, 2014)
  21. 21.
    Ekman, P., Friesen, W.: Unmasking the Face: A Guide to Recognizing Emotions from Facial Clues. Prentice Hall, Englewood Cliffs (1975)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Hiroki Nomiya
    • 1
    Email author
  • Atsushi Morikuni
    • 1
  • Teruhisa Hochin
    • 1
  1. 1.Department of Information ScienceKyoto Institute of TechnologySakyo-kuJapan

Personalised recommendations