Face and Gesture-Based Interaction for Displaying Comic Books

  • Hang-Bong Kang
  • Myung-Ho Ju
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4872)


In this paper, we present human robot interaction techniques such as face pose and hand gesture for efficient viewing comics through the robot. For the controlling of the viewing order of the panel, we propose a robust face pose recognition method using the pose appearance manifold. We represent each pose of a person’s face as connected low-dimensional appearance manifolds which are approximated by the affine plane. Then, face pose recognition is performed by computing the minimal distance from the given face image to the sub-pose manifold. To handle partially occluded faces, we generate an occlusion mask and then put the lower weights on the occluded pixels of the given image to recognize occluded face pose. For illumination variations in the face, we perform coarse normalization on skin regions using histogram equalization. To recognize hand gestures, we compute the center of gravity of the hand using skeleton algorithm and count the number of active fingers. Also, we detect index finger’s moving direction. The contents in the panel are represented by the scene graph and can be updated according to the user’s control. Based on the face pose and hand gesture recognition result, an audience can manipulate contents and finally appreciate the comics in his own style.


Face pose recognition Hand gesture recognition Human robot interaction 


  1. 1.
    Pentland, A., Moghaddam, B., Starner, B.: View-based and modular eigenspaces for face recognition. In: Proc. IEEE Conf. CVPR, IEEE Computer Society Press, Los Alamitos (1994)Google Scholar
  2. 2.
    Moghaddam, B., Pentland, A.: Probabilistic visual learning for object recognition. IEEE Trans. PAMI  (1997)Google Scholar
  3. 3.
    Moghaddam, B.: Principal manifold and probabilistic subspaces for visual recognition. IEEE Trans. PAMI  (2002)Google Scholar
  4. 4.
    Lee, K., Ho, J., Yang, M., Kriegman, D.: Video-based face recognition using probabilistic appearance manifolds. In: Proc. IEEE Conf. CVPR, IEEE Computer Society Press, Los Alamitos (2003)Google Scholar
  5. 5.
    Lee, K., Kriegman, D.: Online learning of probabilistic appearance manifold for video-based recognition and tracking. In: Proc. IEEE Conf. CVPR, IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  6. 6.
    Ju, M., Kang, H.-B.: A new partially occluded face pose recognition. In: ACIVS 2007. LNCS, vol. 4678, Springer, Heidelberg (2007)Google Scholar
  7. 7.
    Davis, J., Shah, M.: Visual Gesture Recognition. IEE Proc. Vis. Image Signal Processing 141(2) (1994)Google Scholar
  8. 8.
    Chang, C., Chen, I., Huang, Y.: Hand Pose Recognition Using Curvature Scale Space. IEEE ICPR  (2002)Google Scholar
  9. 9.
    Hasanuzzaman, Md., Zhang, T., Amporanaramveth, V., Bhuiyan, M., Shirai, Y., Ueno, H.: Face and gesture recognition using subspace method for human-robot interaction. In: Aizawa, K., Nakamura, Y., Satoh, S. (eds.) PCM 2004. LNCS, vol. 3331, Springer, Heidelberg (2004)Google Scholar
  10. 10.
    Malima, A., Ozgur, E., Cetin, M.: A Fast algorithm for vision-based hand gesture recognition for robot control. In: Proc. IEEE Conf. Signal Proc. And Comm. Applications (April 2006)Google Scholar
  11. 11.
    Arandjelovic, O., Shakhnarovich, G., Fisher, J., Cipolla, R., Darrell, T.: Face recognition with image sets using manifold density divergence. In: Proc. IEEE Conf. CVPR, IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  12. 12.
    Evolution robotics,
  13. 13.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proc. CVPR (2001)Google Scholar
  14. 14.
    ISO/IEC 14772-1, The Virtual Reality Modeling Language (VRML) (1997),

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Hang-Bong Kang
    • 1
  • Myung-Ho Ju
    • 1
  1. 1.Dept. of Computer Eng. Catholic University of Korea, #43-1 Yokkok 2-dong Wonmi-Gu, Bucheon, Gyonggi-DoKorea

Personalised recommendations