Dynamic World Modelling by Dichotomic Information Sets and Graphical Inference

With Focus on 3D Facial Pose Tracking
  • Markus Steffens
  • Werner Krybus
  • Christine Kohring
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6725)


This report establishes a novel concept for tracking complex and articulated objects in the presence of high observation uncertainties utilising Markov random fields Markov chains (MRFMCs) and a novel paradigm of modelling visual perception. The approach is rooted in ideas from information fusion and cognitive sciences. The problem is to track non-rigid and articulated objects in the 3D space. The aim is to precisely estimate landmarks with high certainty for fitting accurate object models and secondary states like the orientation under partial occlusions. The targeted system is characterised by a high degree of generality. Previous solutions are relatively limited in robustness and accuracy. The new concept is motivated by the fact that all previous tracking approaches rely on semantic information, that is classified signal signatures, while neglecting all further non-classifiable and thus semantically unrelated information present in the scene herein abstracted as structure. By observing salient cues in structure and by learning and incorporating topological relations between salient cues and semantic features it is intended to tackle the major problem of visual tracking, namely accurate and robust inference in the presence of high observation uncertainties. The notion of the dichotomy of semantic and structure is not covered in previous literature. The new concept constitutes a novel direction in the design and implementation of visual perception and tracking networks. While the ideas of dynamic world modelling and intelligent forgetting stem from principles of information fusion, the principle of fusing semantical with structural information from intelligent exploring is an entirely original contribution and is inspired by ideas from cognitive sciences and linguistics. It is deduced from the inherent yet unrevealed principle of appearance modelling, which is based on incorporating object-related appearance information without classification. In this report the presented system is applied to high-level facial pose tracking and compared to a state-of-the-art reference method.


Visual Tracking Information Fusion Topological Relation Gaussian Graphical Model Virtual Plane 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Agnew, D., Constable, C.: Geophysical data analysis: Multivariate random variables, correlation and error propagation (2008)Google Scholar
  2. 2.
    Allen, P.K.: 3d photography: Point based rigid registration. Technical report, Columbia Department of Computer Science, Columbia University (2005)Google Scholar
  3. 3.
    Arun, K.S., Huang, T.S., Blostein, S.D.: Least-squares fitting of two 3-d point sets. IEEE Trans. PAMI 9, 698–700 (1987)CrossRefGoogle Scholar
  4. 4.
    Bishop, C.M.: A new framework for machine learning. In: Zurada, J.M., Yen, G.G., Wang, J. (eds.) Computational Intelligence: Research Frontiers. LNCS, vol. 5050, pp. 1–24. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  5. 5.
    Bishop, C.M.: Pattern Recognition and Machine Learning: Graphical Models. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  6. 6.
    Blake, A., Isard, M.: Active Contours. Springer, Heidelberg (1997)Google Scholar
  7. 7.
    Chang, W.-Y., Chen, C.-S., Hung, Y.-P.: Tracking by parts: A bayesian approach with component collaboration. IEEE Transactions on Systems, Man, and Cybernetics 39, 375–388 (2009)CrossRefGoogle Scholar
  8. 8.
    Constable, C., Agnew, D.C.: Geophysical data analysis: Statistics (2005)Google Scholar
  9. 9.
    Crowley, J.L., Demazeau, Y.: Principle and techniques for sensor data fusion. Signal Processing 32, 5–27 (1993)CrossRefGoogle Scholar
  10. 10.
    Del Bue, A., Agapito, L.: Non-rigid 3d shape recovery using stereo factorization. In: Asian Conference of Computer Vision (ACCV), vol. 1, pp. 25–30 (2004)Google Scholar
  11. 11.
    Del Bue, A., Smeraldi, F., Agapito, L.: Non-rigid structure from motion using ranklet-based tracking and non-linear optimization. IVC 25(3), 297–310 (2007)CrossRefGoogle Scholar
  12. 12.
    Doucet, A., Johansen, A.M.: A tutorial on particle filtering and smoothing: Fiteen years later (2009)Google Scholar
  13. 13.
    Du, W., Piater, J.: A probabilistic approach to integrating multiple cues in visual tracking. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part II. LNCS, vol. 5303, pp. 225–238. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Gales, M.J.F., Airey, S.S.: Product of gaussians for speech recognition. In: Computer Speech and Language (2006)Google Scholar
  15. 15.
    Hall, D.: Mathematical Techniques in Multisensor Data Fusion. Artech House, Boston (1992)Google Scholar
  16. 16.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2004)CrossRefzbMATHGoogle Scholar
  17. 17.
    Haug, A.J.: A tutorial on bayesian estimation and tracking techniques applicable to nonlinear and non-gaussian processes. Technical report, The MITRE Corporation (2005)Google Scholar
  18. 18.
    Isard, M.: Pampas: Real-valued graphical models for computer vision. Technical report, Microsoft Research (2003)Google Scholar
  19. 19.
    Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking (1998)Google Scholar
  20. 20.
    Johnson, J.K.: Estimation of gmrfs by recursive cavity modeling. Technical report, EECS Dept., MIT (2004)Google Scholar
  21. 21.
    Jordan, M.I., Weiss, Y.: The Handbook of Brain Theory and Neural Networks. Graphical models: Probabilistic inference. MIT Press, Cambridge (2002)Google Scholar
  22. 22.
    Jordan, M.I.: An introduction to probabilistic graphical models. Technical report, University of California, Berkeley (2003)Google Scholar
  23. 23.
    Kropatsch, W.: Tracking with structure in computer vision twist-cv. Technical report, Patter Recognition and Image Processing Group, TU Wien (2005)Google Scholar
  24. 24.
    Li, T., Kallem, V., Singaraju, D., Vidal, R.: Projective factorization of multiple rigid-body motions. In: CVPR 2007, pp. 1–6 (2007)Google Scholar
  25. 25.
    Malioutov, D.M.: Approximate Inference in Gaussian Graphical Models. PhD thesis, Dept. of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge (2008)Google Scholar
  26. 26.
    Malioutov, D.M., Johnson, J.K., Willsky, A.S.: Walk-sums and belief propagation in gaussian graphical models. Journal of Machine Learning Research 7, 2031–2064 (2006)zbMATHGoogle Scholar
  27. 27.
    Mills, S.: Stereo-motion analysis of image sequences. In: Proceedings of Digital Image & Vision Computing: Techniques and Applications (DICTA), pp. 515–520 (1997)Google Scholar
  28. 28.
    Mills, S., Novins, K.: Graph-based object hypothesis. New Zealand Journal of Computing 7, 21–29 (1998)Google Scholar
  29. 29.
    Mills, S., Novins, K.: Motion segmentation in long image sequences. In: Proceedings of the British Machine Vision Conference, pp. 162–171 (2000)Google Scholar
  30. 30.
    Murphy, K.P.: An introduction to graphical models. Technical report, University of British Columbia, Vancouver, Canada (2001)Google Scholar
  31. 31.
    Newman, P., Leonard, J.: A matrix oriented note on joint, marginal, and conditional multivariate gaussian distributions. Technical report, Massachusetts Institute of Technology (2006)Google Scholar
  32. 32.
    Ristic, B., Arulampalam, S., Gordon, N.: Beyond the Kalman Filter: Particle Filters for Tracking Applications. Artech House Publishers, Boston (2004)zbMATHGoogle Scholar
  33. 33.
    Rong Li, X., Jilkov, V.P.: Survey of maneuvering target tracking. part i: Dynamic models. IEEE Transactions on Aerospace and Electronic Systems 39, 1333–1364 (2003)CrossRefGoogle Scholar
  34. 34.
    Sanfeliu, A., Serratosa, F.: Learning and recognising 3d models represented by multiple views by means of methods based on random graphs. In: Proceedings International Conference on Image Processing, ICIP (2003)Google Scholar
  35. 35.
    Sigal, L., Zhu, Y., Comaniciu, D., Black, M.J.: Tracking complex objects using graphical object models. In: Jähne, B., Mester, R., Barth, E., Scharr, H. (eds.) IWCM 2004. LNCS, vol. 3417, pp. 223–234. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  36. 36.
    Sinha, A., Chen, H., Danu, D.G., Kirubarajan, T., Farooq, M.: Estimation and decision fusion: A survey. Neurocomputing 71, 2650–2656 (2008)CrossRefGoogle Scholar
  37. 37.
    Smith, D., Singh, S.: Approaches to multisensor data fusion in target tracking: A survey. IEEE Transactions on Knowledge and Data Engineering 18(12), 1696–1710 (2006)CrossRefGoogle Scholar
  38. 38.
    Steffens, M., Krybus, W., Kohring, C.: Linear gaussian error models from component matrices for 3d graphical tracking networks. In: Submitted to SAMT 2010 (2010)Google Scholar
  39. 39.
    Steffens, M., Krybus, W., Kohring, C.: Spatio-temporal gaussian graphical models as tracking networks. In: Submitted to SAMT 2010 (2010)Google Scholar
  40. 40.
    Su, C., Huang, L.: Spatio-temporal graphical-model-based multiple facial feature tracking. EURASIP Journal on Applied Signal Processing 13, 2091–2100 (2005)CrossRefGoogle Scholar
  41. 41.
    Sudderth, E.B., Ihler, A.T., Freeman, W.T., Willsky, A.S.: Nonparametric belief propagation. In: Conference Proceedings Computer Vision and Pattern Recognition. IEEE, Los Alamitos (2003)Google Scholar
  42. 42.
    Tang, C.-Y., Hung, Y.-P., Shih, S.-W., Chen, Z.: A 3d feature-based tracker for multiple object tracking. In: Proceedings of the National Science Council, Republic of China, Part A: Physical Science and Engineering, pp. 151–168 (1999)Google Scholar
  43. 43.
    Taycher, L., Fisher III, J.W., Darrell, T.: Combining object and feature dynamics in probabilistic tracking. Computer Vision and Image Understanding 108, 243–260 (2007)CrossRefGoogle Scholar
  44. 44.
    Vidal, R., Abretske, D.: Nonrigid shape and motion from multiple perspective views. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 205–218. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  45. 45.
    Vidal, R., Ma, Y.: A unified algebraic approach to 2-d and 3-d motion segmentation. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3021, pp. 1–15. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  46. 46.
    Vidal, R., Ma, Y., Soatto, S., Sastry, S.: Two-view multibody structure from motion. IJCV 68(1), 7–25 (2006)CrossRefGoogle Scholar
  47. 47.
    Vidal, R., Ravichandran, A.: Optical flow estimation and segmentation of multiple moving dynamic textures. In: CVPR 2005, pp. II: 516–521 (2005)Google Scholar
  48. 48.
    Vidal, R., Singaraju, D.: A closed form solution to direct motion segmentation. In: CVPR 2005, pp. II: 510–515 (2005)Google Scholar
  49. 49.
    Wang, P., Ji, Q.: Robust face tracking via collaboration of generic and specific models. IEEE Transactions on Image Processing 17, 1189–1199 (2008)CrossRefGoogle Scholar
  50. 50.
    Weingarten, J.W., Gruener, G., Siegwart, R.: Probabilistic plane fitting in 3d and an application to robotic mapping. In: IEEE International Conference on Robotics and Automation (2004)Google Scholar
  51. 51.
    Yang, M., Wu, Y.: Granularity and elasticity adaptation in visual tracking. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2008)Google Scholar
  52. 52.
    Yedidia, J.S., Freeman, W.T., Weiss, Y.: Understanding belief propagation and its generalizations. Technical report. MIT, Cambridge (2001)Google Scholar
  53. 53.
    Yu, T., Wu, Y.: Decentralized multiple target tracking using netted collaborative autonomous trackers. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 939–946 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Markus Steffens
    • 1
  • Werner Krybus
    • 1
  • Christine Kohring
    • 1
  1. 1.University of Applied Sciences South WestphaliaGermany

Personalised recommendations