Skip to main content
SpringerLink
Log in

Robust Temporally Coherent Laplacian Protrusion Segmentation of 3D Articulated Bodies

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In motion analysis and understanding it is important to be able to fit a suitable model or structure to the temporal series of observed data, in order to describe motion patterns in a compact way, and to discriminate between them. In an unsupervised context, i.e., no prior model of the moving object(s) is available, such a structure has to be learned from the data in a bottom-up fashion. In recent times, volumetric approaches in which the motion is captured from a number of cameras and a voxel-set representation of the body is built from the camera views, have gained ground due to attractive features such as inherent view-invariance and robustness to occlusions. Automatic, unsupervised segmentation of moving bodies along entire sequences, in a temporally-coherent and robust way, has the potential to provide a means of constructing a bottom-up model of the moving body, and track motion cues that may be later exploited for motion classification. Spectral methods such as locally linear embedding can be useful in this context, as they preserve “protrusions”, i.e., high-curvature regions of the 3D volume, of articulated shapes, while improving their separation in a lower dimensional space, making them in this way easier to cluster. In this paper we therefore propose a spectral approach to unsupervised and temporally-coherent body-protrusion segmentation along time sequences. Volumetric shapes are clustered in an embedding space, clusters are propagated in time to ensure coherence, and merged or split to accommodate changes in the body’s topology. Experiments on both synthetic and real sequences of dense voxel-set data are shown. This supports the ability of the proposed method to cluster body-parts consistently over time in a totally unsupervised fashion, its robustness to sampling density and shape quality, and its potential for bottom-up model construction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27

Similar content being viewed by others

Notes

  1. The embeddings \(c'_j(t+1)\) of the previous 3D centroids \(X_{i_j}(t)\) in the new embedded cloud can also be computed by out of sample extension (Bengio et al. 2003).

  2. These data are available on-line at http://4drepository.inrialpes.fr/public/datasets.

References

  • Agarwal, S., Lim, J., Zelnik-Manor, L., Perona, P., Kriegman, D., Belongie, S. (2005). Beyond pairwise clustering. In Computer Vision and Pattern Recognition, pp. 838–845.

  • Ali, S., Basharat, A., Shah, M. (2007). Chaotic invariants for human action recognition. In International Conference on Computer Vision.

  • Alpert, A. B., Kahng, C. J., & Yao, S.-Z. (1999). Spectral partitioning with multiple eigenvectors. Discrete Applied Mathematics, 90(1–3), 3–26.

    Article  MATH  MathSciNet  Google Scholar 

  • Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2006). Generalized multidimensional scaling: A framework for isometry-invariant partial surface matching. Proceedings of the National Academy of Sciences, 103(5), 1168–1172.

    Article  MATH  MathSciNet  Google Scholar 

  • Bronstein, A. M., Bronstein, M. M., & Kimmel, R. (2009). Topology-invariant similarity of nonrigid shapes. International Journal of Computer Vision, 81(3), 281–301.

    Article  Google Scholar 

  • Belkin, M., & Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Computation, 15, 1373–1396.

    Article  MATH  Google Scholar 

  • Bengio, Y., Delalleau, O., Le Roux, N., Paiement, J.-F., Vincent, P., & Ouimet, M. (2004). Learning eigenfunctions links spectral embedding and kernel PCA. Neural Computation, 16(10), 2197–2219.

    Article  MATH  Google Scholar 

  • Bengio, Y., Paiement, J.-F., & Vincent, P. (2003). Out-of-sample extensions for LLE, Isomap, MDS, eigenmaps, and spectral clustering. Technical report: Université Montreal.

  • Bissacco, A., Chiuso, A., & Soatto, S. (2007). Classification and recognition of dynamical models: The role of phase, independent components, kernels and optimal transport. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(11), 1958–1972.

    Article  Google Scholar 

  • Biyikoglu, J., Leydold, T., & Stadler, P. F. (2007). Laplacian eigenvectors of graphs. Berlin: Springer.

    Book  MATH  Google Scholar 

  • Blank, M., Gorelick, L., Shechtman, E., Irani, M., Basri, R. (2005). Actions as space-time shapes. In International Conference on Computer Vision, pp. 1395–1402.

  • Brand, M. (1999) Shadow puppetry. In International Conference on Computer Vision.

  • Brostow, G.-J., Essa, I., Steedly, D., Kwatra, V. (2004). Novel skeletal representation for articulated creatures. In European Conference on Computer Vision.

  • Carter, N., Young, D., Ferryman. J. (2006) Supplementing Markov chains with additional features for behavioural analysis. In Proceedings of AVSBS.

  • Chang, W., Zwicker, M. (2008). Automatic registration for articulated shapes. In Eurographics Symposium on Geometry Processing.

  • Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R. (2009) Histograms of oriented optical flow and Binet–Cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In Computer Vision and Pattern Recognition, pp. 1932–1939.

  • Chu, C.-W., Jenkins, O. C., Mataric, M. J. (2003) Markerless kinematic model and motion capture from volume sequences. In Computer Vision and Pattern Recognition, pp. 475–482.

  • Chung, F. (1997). Spectral Graph Theory. Providence: American Mathematical Society.

    MATH  Google Scholar 

  • Cuzzolin, F., Sarti, A., & Tubaro, S. (2004). Action modeling with volumetric data. In Proceedings of ICIP, 2, 881–884.

    Google Scholar 

  • Cuzzolin, F., Sarti, A., Tubaro, S. (2004). Invariant action classification with volumetric data. In International Workshop on Multimedia Signal Processing, pp. 395–398.

  • Cuzzolin, F., Mateus, D., Boyer, E., & Horaud, R. (2007). Robust spectral 3D-bodypart segmentation along time in Workshop on Human Motion—Understanding, Modeling. Capture and Animation, 196–211, 2007.

    Google Scholar 

  • Cuzzolin, F., Mateus, D., Knossow, D., Boyer, E., Horaud, R. (2008). Coherent Laplacian 3-D protrusion segmentation. In Computer Vision and Pattern Recognition.

  • Cvetkovic, M., Doob, D. M., & Sachs, H. (1998). Spectra of graphs, theory and application. Berlin: Vch Verlagsgesellschaft Mbh.

  • Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Pang, H., Davis, J. (2004). The correlated correspondence algorithm for unsupervised registration of non-rigid surfaces. In Neural Information Processing Systems.

  • Anguelov, D., Koller, D., Pang, H.-C., Srinivasan, P., Thrun, S. (2004). Recovering articulated object models from 3D range data. In Proceedings of UAI.

  • Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the em algorithm. Journal of the Royal Statistical Society, 39, 1–38.

    MATH  MathSciNet  Google Scholar 

  • Dhillon, Y., Guan, I. S., Kulis, B. (2004). Kernel kmeans: spectral clustering and normalized cuts. In Proceedings of KDD, pp. 551–556.

  • Elgammal, A., Lee, C. (2004). Inferring 3D body pose from silhouettes using activity manifold learning. In Computer Vision and Pattern Recognition.

  • Elliot, R., Aggoun, L., & Moore, J. (1995). Hidden Markov models: Estimation and control. New York: Springer.

    Google Scholar 

  • Fouss, A. R.-J. M., Pirotte, F., & Saerens, M. (2007). Random-walk computation of similarities between nodes of a graph, with application to collaborative recommendation. IEEE Transactions on Knowledge and Data Engineering, 19(3), 355–369.

    Article  Google Scholar 

  • Franco, J. S., & Boyer, E. (2009). Efficient polyhedral modeling from Silhouettes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(3), 414–427.

    Article  Google Scholar 

  • Franco, J. S., Boyer, E. (2011). Learning Temporally Consistent Rigidities. In Computer Vision and Pattern Recognition.

  • Furukawa, Y., Ponce, J. (2005) Carved visual hulls for high accuracy image-based modeling. In Technical Sketch at SIGGRAPH.

  • Golovinskiy, A., Funkhouser, T. (2009). Consistent segmentation of 3d models. In SIGGRAPH.

  • Grauman, K., Shakhnarovich, G., Darrell, T. (2003). Inferring 3D structure with a statistical image-based shape model. In International Conference on Computer Vision, pp. 641–648.

  • Gupta, A., Srinivasan, P., Shi, J., Davis, L. S. (2009). Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos. In Computer Vision and Pattern Recognition, pp. 2012–2019.

  • Hayashi, C. (1972). Two dimensional quantification based on the measure of dissimilarity among three elements. Annals of the Institute of Statistical Mathematics, 24, 251–257.

    Article  MATH  MathSciNet  Google Scholar 

  • Heiser, W. J., & Bennani, M. (1997). Triadic distance models: Axiomatization and least squares representation. Journal of Mathematical Psychology, 41, 189–206.

    Article  MATH  MathSciNet  Google Scholar 

  • Hernandez, C., & Schmitt, F. (2004). Silhouette and stereo fusion for 3d object modeling. Computer Vision and Image Understanding, 96(3), 367–392.

    Article  Google Scholar 

  • Jain, V., Zhang, H. (2006). Robust 3d shape correspondence in the spectral domain. In Shape Modeling International.

  • Jakobson, N., & Toth, J. (2001). Geometric properties of eigenfunctions. Russian Mathematical Surveys, 56(6), 1085–1106.

    Article  MATH  MathSciNet  Google Scholar 

  • Jenkins, O., Mataric, M. (2004). A spatio-temporal extension to isomap nonlinear dimension reduction. In International Conference on Machine Learning.

  • Katz, S., Leifman, G., & Tal, A. (2005). Mesh segmentation using feature point and core extraction. The Visual Computer, 21, 649–658.

    Article  Google Scholar 

  • Kim, T., & Cipolla, R. (2009). Canonical correlation analysis of video volume tensors for action categorization and detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(8), 1415–1428.

    Article  Google Scholar 

  • Knossow, D., Ronfard, R., & Horaud, R. (2008). Human motion tracking with a kinematic parameterization of extremal contours. International Journal of Computer Vision, 79(3), 247–269.

    Article  Google Scholar 

  • Lafon, S., & Lee, A. B. (2006). Diffusion maps and coarse-graining: A unified framework for dimensionality reduction, graph partitioning, and data set parameterization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(9), 1393–1403.

  • Levi, B. (2006) Laplace–Beltrami eigenfunctions: Towards and algorithm that understands geometry. In Shape Modeling International.

  • Lien, J.-M., Amanto, N. (2007). Approximate convex decomposition of polyhedra. In Proceedings of the ACM Symposium on Solid and Physical Modeling, pp. 121–131.

  • Lin, R., Liu, C.-B., Yang, M.-H., Ahuja, N., Levinson, S. (2006) Learning nonlinear manifolds from time series. In European Conference on Computer Vision.

  • Liu, R., Zhang, H. (2004). Segmentation of 3D meshes through spectral clustering. In Proceedings of Computer Graphics and Applications, pp. 298–305.

  • Luxburg, U. (2007). A tutorial on spectral clustering. Statistics and Computing, 17(4), 395–416.

    Article  MathSciNet  Google Scholar 

  • MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, 1, 281–297.

    MathSciNet  Google Scholar 

  • Mateus, D., Horaud, R., Knossow, D., Cuzzolin, F., Boyer, E. (2008). Articulated shape matching using laplacian eigenfunctions and unsupervised point registration. In Computer Vision and Pattern Recognition.

  • Meila, M., Shi, J. (2001). A random walks view of spectral segmentation. In Artificial Intelligence and Statistics.

  • Ng, A., Jordan, M., Weiss, Y. (2002). On spectral clustering: Analysis and an algorithm. In Neural Information Processing Systems.

  • Huang, Q.-X., Wicke, M., Adams, B., Guibas, L. (2009). Shape decomposition using modal analysis. In EUROGRAPHICS, volume 28.

  • Pons, J.-P., Keriven, R., & Faugeras, O. (2007). Multi-view stereo reconstruction and scene flow estimation with a global image-based matching score. International Journal of Computer Vision, 72(2), 179–193.

    Article  Google Scholar 

  • Ralaivola, L., & d’Alche Buc, F. (2004). Dynamical modeling with kernels for nonlinear time series prediction. Information Processing Systems, 16, 129–136.

    Google Scholar 

  • Roweis, S., & Saul, L. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500), 2323–2326.

    Article  Google Scholar 

  • Saerens, M., Fouss, F., Yen, L., Dupont, P. (2004). The principal components analysis of a graph, and its relationships to spectral clustering. In European Conference on Machine Learning, pp. 371–383.

  • Shamir, A. (2008). A survey on mesh segmentation techniques. Computer Graphics Forum, 26, 1539–1556.

  • Shapira, L., Shamir, A., & Cohen-Or, D. (2008). Consistent mesh partitioning and skeletonisation using the shape diameter function. The Visual Computer, 24, 249–259.

    Article  Google Scholar 

  • Sharma, A., Horaud, R., Knossow, D., von Lavante, E. (2009). Mesh segmentation using Laplacian eigenvectors and Gaussian mixtures. In Proceedings of the AAAI Fall Symposium on Manifold Learning and its Applications.

  • Shi, J., & Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 888–905.

    Article  Google Scholar 

  • Shi, Q., Wang, L., Cheng, L., Smola, A. (2008). Discriminative human action segmentation and recognition using semi-Markov model. In Computer Vision and Pattern Recognition.

  • Starck, J., Hilton, A. (2007). Correspondence labeling for widetimeframe free-form surface matching. In International Conference on Computer Vision.

  • Sundaresan, A., Chellappa, R. (2006). Segmentation and probalistic registration of articulated body models. In International Conference on Pattern Recognition.

  • Tenenbaum, J. B., de Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2319–2323.

    Article  Google Scholar 

  • Varanasi, K., Boyer, E. (2010). Temporally coherent segmentation of 3D reconstructions. In Proceedings of 3DPTV.

  • Zaharescu, A., Boyer, E., & Horaud, R. (2011). Topology-adaptive mesh deformation for surface evolution, morphing, and multi-view reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(4), 823–837.

    Article  Google Scholar 

  • Zhou, K., Huang, J., Snyder, J., Liu, X., Bao, H., Guo, B., et al. (2005). Large mesh deformation using the volumetric graph Laplacian. ACM Transactions on Graphics, 24(3), 496–503.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computing and Communication Technologies, Oxford Brookes University, Oxford, UK

    Fabio Cuzzolin

  2. Institut für Informatik, Technische Universität München, Garching b. München, Germany

    Diana Mateus

  3. INRIA Grenoble Rhône-Alpes, 655 avenue de l’Europe, Montbonnot Saint-Martin, France

    Radu Horaud

Authors
  1. Fabio Cuzzolin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diana Mateus
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Radu Horaud
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fabio Cuzzolin.

Additional information

Communicated by k. Ikeuchi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cuzzolin, F., Mateus, D. & Horaud, R. Robust Temporally Coherent Laplacian Protrusion Segmentation of 3D Articulated Bodies. Int J Comput Vis 112, 43–70 (2015). https://doi.org/10.1007/s11263-014-0754-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-014-0754-0

Keywords

Search

Navigation