Robust Attentive Behavior Detection by Non-linear Head Pose Embedding and Estimation

  • Nan Hu
  • Weimin Huang
  • Surendra Ranganath
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3953)


We present a new scheme to robustly detect a type of human attentive behavior, which we call frequent change in focus of attention (FCFA), from video sequences. FCFA behavior can be easily perceived by people as temporal changes of human head pose (normally the pan angle). For recognition of this behavior by computer, we propose an algorithm to estimate the head pan angle in each frame of the sequence within a normal range of the head tilt angles. Developed from the ISOMAP, we learn a non-linear head pose embedding space in 2-D, which is suitable as a feature space for person-independent head pose estimation. These features are used in a mapping system to map the high dimensional head images into the 2-D feature space from which the head pan angle is calculated very simply. The non-linear person-independent mapping system is composed of two parts: 1) Radial Basis Function (RBF) interpolation, and 2) an adaptive local fitting technique. The results show that head orientation can be estimated robustly. Following the head pan angle estimation, an entropy-based classifier is used to characterize the attentive behaviors. The experimental results show that entropy of the head pan angle is a good measure, which is quite distinct for FCFA and focused attention behavior. Thus by setting an experimental threshold on the entropy value we can successfully and robustly detect FCFA behavior.


Radial Basis Function Video Sequence Focus Attention Locally Linear Embedding Head Orientation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Blumberg, B., et al.: Creature smarts: the art and architecture of a virtual brain. In: proc. Game Developers Conf., pp. 147–166 (2000)Google Scholar
  2. 2.
  3. 3.
    Rae, R., Ritter, H.: Recognition of human head orientation based on artificial neural networks. IEEE Trans. Neural Networks 9, 257–265 (1998)CrossRefGoogle Scholar
  4. 4.
    Zhao, L., Pingali, G., Carlbom, I.: Real-time head orientation estimation using neural networks. In: proc. Int’l Conf. Image Processing, vol. 1, pp. 297–300 (2002)Google Scholar
  5. 5.
    Matsumoto, Y., Zelinsky, A.: An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement. FG 2000, 499–505 (2000)Google Scholar
  6. 6.
    Srinivasan, S., Boyer, K.L.: Head pose estimation using view based eigenspaces. ICPR 2002 4, 302–305 (2002)Google Scholar
  7. 7.
    Everingham, M., Zisserman, A.: Identifying individuals in video by combining generative and discriminative head models. In: Proceedings of the International Conference on Computer Vision, pp. 1103–1110 (2005)Google Scholar
  8. 8.
    Stiefelhagen, R., Yang, J., Waibel, A.: Modeling focus of attention for meeting indexing. ACM Multimedia, 3–10 (1999)Google Scholar
  9. 9.
    Tenenbaum, J., de Silva, V., Langford, J.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar
  10. 10.
    Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  11. 11.
    Pless, R.: Image spaces and video trajectories: Using isomap to explore video sequences. ICCV 2003 2, 1433–1440 (2003)Google Scholar
  12. 12.
    Elgammal, A., Lee, C.: Separating style and content on a nonlinear manifold. CVPR 2004, 478–485 (2004)Google Scholar
  13. 13.
    Vlachos, M., et al.: Non-linear dimensionality reduction techniques for classification and visualization. In: proc. 8th ACM SIGKDD Int’l Conf. on Knowledge Discovery and Data Mining, pp. 645–651 (2002)Google Scholar
  14. 14.
    Efros, A., et al.: Seeing through water. In: Neural Information Processing Systems (NIPS 17), pp. 393–400 (2004)Google Scholar
  15. 15.
    Yang, M., Kriegman, D., Ahuja, N.: Detecting faces in images: A survey. IEEE Trans. PAMI 24, 34–58 (2002)Google Scholar
  16. 16.
    Fitzgibbon Pilu, M., Fisher, R.: Direct least-square fitting of ellipses. IEEE Trans. PAMI 21, 476–480 (1999)Google Scholar
  17. 17.
    Duda, R., Hart, P., Stork, D.: Pattern Classification, 2nd edn. John Wiley & Sons, Inc., Chichester (2000)Google Scholar
  18. 18.
    Hutcheson, M.: Trimmed Resistant Weighted Scatterplot Smooth. Master’s Thesis, Cornell University (1995)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Nan Hu
    • 1
  • Weimin Huang
    • 2
  • Surendra Ranganath
    • 3
  1. 1.Electrical & Computer EngineeringUniversity of KentuckyLexingtonUSA
  2. 2.Institute for Infocomm Research (I2R)Singapore
  3. 3.Electrical & Computer EngineeringNational University of SingaporeSingapore

Personalised recommendations