Emotional Scene Retrieval from Lifelog Videos Using Evolutionary Feature Creation

  • Hiroki NomiyaEmail author
  • Teruhisa Hochin
Conference paper
Part of the Studies in Computational Intelligence book series (SCI, volume 612)


For the purpose of promoting the utilization of a large amount of lifelog videos, an emotional scene retrieval framework is proposed. It detects emotional scenes on the basis of facial expression recognition assuming that a kind of emotion will be aroused with a certain facial expression in an important scene which is likely to be a target of the retrieval. The emotional scene retrieval has a critical issue that it is quite hard to accurately and efficiently detect the emotional scenes because of the difficulty in discriminating spontaneous facial expressions. One of the most effective way to enhance the performance of the retrieval is to select discriminative facial features used for the facial expression recognition. It is, however, not easy to manually select good facial features because very subtle and complex movements of several facial parts will be observed in the appearance of a facial expression. We thus propose a method to automatically generate discriminative facial features on the basis of genetic programming. It produces discriminative facial features by combining a number of points on some salient facial parts using various arithmetic operators. The proposed method is evaluated through an emotional scene detection experiment using a lifelog video dataset containing spontaneous facial expressions.


Lifelog Video retrieval Facial expression recognition Genetic programming 



This research is supported by Japan Society for the Promotion of Science, Grant-in-Aid for Young Scientists (B), 15K15993.


  1. 1.
    Aizawa, K., Hori, T., Kawasaki, S., Ishikawa, T.: Capture and efficient retrieval of life log. In: Proceedings of Pervasive 2004 Workshop on Memory and Sharing Experiences, pp. 15-20 (2004)Google Scholar
  2. 2.
    Gemmell, J., Bell, G., Luederand, R., Drucker, S., Wong, C.: MyLifeBits: fulfilling the memex vision. In: Proceedings of the 10th ACM International Conference on Multimedia, pp. 235-238 (2002)Google Scholar
  3. 3.
    Datchakorn, T., Yamasaki, T., Aizawa, K.: Practical experience recording and indexing of life log video. Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences, pp. 61-66 (2005)Google Scholar
  4. 4.
    Nomiya, H., Morikuni, A., Hochin, T.: An unsupervised ensemble approach for emotional scene detection from lifelog videos. Softw. Eng. Artif. Intell. Netw. Parallel/Distrib. Comput. Stud. Comput. Intell. 569, 145-159 (2015)Google Scholar
  5. 5.
    Datcu, D., Rothkrantz, L.: Facial expression recognition in still pictures and videos using active appearance models: a comparison approach. In: Proceedings of the 2007 International Conference on Computer Systems and Technologies, pp. 1-6 (2007)Google Scholar
  6. 6.
    Fanelli, G., Yao, A., Noel, P.-L., Gall, J., Gool, L.V.: Hough forest-based facial expression recognition from video sequences. In: Proceedings of the 11th European Conference on Trends and Topics in Computer Vision, pp. 195-206 (2010)Google Scholar
  7. 7.
    Koza, J.R.: Genetic programming: on the programming of computers by means of natural selection. The MIT Press, Cambridge (1992)zbMATHGoogle Scholar
  8. 8.
    Tian, Y., Kanade, T., Cohn, J.F.: In: Li, S.Z., Jain, A.K. (eds.) Handbook of face recognition. Facial Expression Recognition. Springer, London (2011)Google Scholar
  9. 9.
    Soyel, H., Demirel, H.: Facial expression recognition using 3d facial feature distances. In: Proceedings of the 4th International Conference on Image Analysis and Recognition, pp. 831-838 (2007)Google Scholar
  10. 10.
    Hupont, I., Cerezo, E., Baldassarri, S.: Sensing facial emotion in a continuous 2D affective space. In: Proceedings of International Conference on Systems, Man, and Cybernetics, pp. 2045-2051 (2010)Google Scholar
  11. 11.
    Luxand Inc.: Luxand FaceSDK 4.0 (2015). (The latest version is 5.0.1)
  12. 12.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273-297 (1995)zbMATHGoogle Scholar
  13. 13.
    Fraser, A., Weinbrenner, T.: GPC++ - Genetic Programming C++ Class Library, Version 0.5.2 (2015).
  14. 14.
    Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3, 27):1-27 (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  1. 1.Kyoto Institute of TechnologySakyo-ku, KyotoJapan

Personalised recommendations