Abstract
We propose a model of the shape, motion and appearance of a scene, seen through a sequence of images, that captures occlusions, scene deformations, unconstrained viewpoint variations and changes in its radiance. This model is based on a collection of overlapping layers that can move and deform, each supporting an intensity function that can change over time. We discuss the generality and limitations of this model in relation to existing ones such as traditional optical flow or motion segmentation, layers, deformable templates and deformotion. We then illustrate how this model can be used for inference of shape, motion, deformation and appearance of the scene from a collection of images. The layering structure allows for automatic inpainting of partially occluded regions. We illustrate the model on synthetic and real sequences where existing schemes fail, and show how suitable choices of constants in the model yield existing schemes, from optical flow to motion segmentation and inpainting.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Alvarez, L., Weickert, J., & Sanchez, J. (1999). A scale-space approach to nonlocal optical flow calculations. In ScaleSpace ’99 (pp. 235–246).
Baker, S., Matthews, I., & Schneider, J. (2003). Image coding with active appearance models (Technical report). Carnegie Mellon University, The Robotics Institute.
Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, C. (2000). Image inpainting. In K. Akeley (Ed.), Siggraph 2000, computer graphics proceedings (pp. 417–424). Longman: Addison-Wesley.
Blake, A., & Isard, M. (1998). Active contours. Berlin: Springer.
Caselles, V., Kimmel, R., & Sapiro, G. (1997). Geodesic active contours. International Journal of Computer Vision, 22(1), 61–79.
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (1998). Active appearance models. In Proceedings of the European conference on computer vision (pp. 484–496).
Cremers, D. (2003). Multiphase levelset framework for variational motion segmentation. In International conference on scale-space theories in computer vision (pp. 599–614), June 2003.
Deriche, R., Kornprobst, P., & Aubert, G. (1995). Optical flow estimation while preserving its discontinuities: a variational approach. In Proceedings of ACCV (Vol. 2, pp. 290–295).
Grenander, U. (1993). General pattern theory. Oxford: Oxford University Press.
Haussecker, H. W., & Fleet, D. J. (2001). Computing optical flow with physical models of brightness variation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 661–673.
Horn, B. K. P., & Schunk, B. G. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203.
Hsu, S., Anandan, P., & Peleg, S. (1992). Accurate computation of optical flow by using layered motion representations. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1621–1626).
Kichenassamy, S., Kumar, A., Olver, P., Tannenbaum, A., & Yezzi, A. (1995). Gradient flows and geometric active contour models. In ICCV ’95: proceedings of the fifth international conference on computer vision (pp. 810–815), Washington, DC, USA. New York: IEEE Computer Society.
Marks, T., Hershey, J., Roddey, J., & Movellan, J. (2005). Joint tracking of pose, expression, and texture using conditionally Gaussian filters. In Advances in neural information processing systems (Vol. 17). Cambridge: MIT Press.
Miller, M. I., & Younes, L. (1999). Group action, diffeomorphism and matching: a general framework. In Proceedings of SCTV.
Osher, S., & Sethian, J. (1988). Fronts propagating with curvature-dependent speed: algorithms based on Hamilton–Jacobi equations. Journal of Computational Physics, 79, 12–49.
Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (2006). Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision, 67(2), 141–158.
Paragios, N., & Deriche, R. (2000). Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(3), 266–280.
Paragios, N., Rousson, M., & Ramesh, V. (2003). Non-rigid registration using distance functions. Computer Vision and Image Understanding, 89(2–3), 142–165.
Schnörr, C. (1992). Computation of discontinuous optical flow by domain decomposition and shape optimization. International Journal of Computer Vision, 8(2), 153–165.
Soatto, S., & Yezzi, A. (2002). Deformotion: deforming motion, shape average and the joint segmentation and registration of images. In Proceedings of the European conference on computer vision (ECCV) (Vol. 3, pp. 32–47).
Trouve, A., & Younes, L. (2005). Metamorphoses through lie group action. Foundations of Computational Mathematics, 5(2), 173–198.
Wang, J., & Adelson, E. (1994). Representing moving images with layers. IEEE Transactions on Image Processing, 3(5), 625–638.
Yezzi, A., & Soatto, S. (2001). Stereoscopic segmentation. In Proceedings of the international conference on computer vision (pp. 59–66).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Jackson, J.D., Yezzi, A.J. & Soatto, S. Dynamic Shape and Appearance Modeling via Moving and Deforming Layers. Int J Comput Vis 79, 71–84 (2008). https://doi.org/10.1007/s11263-007-0097-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-007-0097-1