Machine Learning for Joint Classification and Segmentation

  • Jeremy Lerner
  • Romeil Sandhu
  • Yongxin Chen
  • Allen TannenbaumEmail author
Part of the Lecture Notes in Control and Information Sciences - Proceedings book series (LNCOINSPRO)


In this note, we consider the use of 3D models for visual tracking in controlled active vision. The models are used for a joint 2D segmentation/3D pose estimation procedure in which we automatically couple the two processes under one energy functional. Further, employing principal component analysis or locally linear embedding from statistical learning, one can train our tracker on a catalog of 3D shapes, giving a priori shape information. The segmentation itself is information based, which allows us to track in uncertain adversarial environments. Our methodology is demonstrated on realistic scenes, which illustrate its robustness on challenging scenarios.


  1. 1.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15, 1373–139615 (2003)CrossRefzbMATHGoogle Scholar
  2. 2.
    Chan, T., Sandberg, B., Vese, L.: Active contours without edges for vector-valued images. J. Visual Commun. Image Represent. 11(2), 130–141 (2000)CrossRefGoogle Scholar
  3. 3.
    Cipolla, R., Blake, A.: Surface shape from the deformation of apparent contours. Int. J. Comp. Vis. 9(2), 83–112 (1992)CrossRefGoogle Scholar
  4. 4.
    Dambreville, S., Sandhu, R., Yezzi, A., Tannenbaum, A.: Robust 3D pose estimation and efficient 2D region-based segmentation from a 3D shape prior. In: Proceedings of ECCV, pp. 169–182 (2008)Google Scholar
  5. 5.
    Dambreville, S., Sandhu, R., Yezzi, A., Tannenbaum, A.: A geometric approach to joint 2D region-based segmentation and 3D pose estimation using a 3D shape prior. SIAM Imaging Sci. 3, 110–132 (2010)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Donoho, D.L., Grimes, C.: "Hessian eigenmaps: new locally linear embedding techniques for high-dimensional data," Technical Report TR-2003-08. Stanford University, Department of Statistics (2003)zbMATHGoogle Scholar
  7. 7.
    Flanders, H.: Differentiation under the integral sign. Am. Math. Monthly 80(6), 615–627 (1973)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Hein, M., Maier, M.: Manifold denoising. NIPS:561–568 (2006)Google Scholar
  9. 9.
    Leventon, M., Grimson, E., Faugeras, O.: Statistical shape influence in geodesic active contours. In: IEEE CVPR, pp. 1316–1324 (2000)Google Scholar
  10. 10.
    Ma, Y., Stefano, S., Kosecka, J., Sastry, S.: An invitation to 3-d vision: from images to geometric models. In: Springer Science and Business Media, p. 26 (2012)Google Scholar
  11. 11.
    Michailovich, O., Rathi, Y., Tannenbaum, A.: Image segmentation using active contours driven by the Bhattacharyya gradient flow. IEEE Trans. Image Process. 16, 2787–2801 (2007)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Osher, S., Fedkiw, R.: Level Set Methods and Dynamic Implicit Surfaces. Springer (2003)Google Scholar
  13. 13.
    Rathi, Y., Tannenbaum, A.: A generic framework for tracking using particle filter with dynamic shape prior. IEEE Trans. Image Process. 16, 1370–1382 (2007)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Rathi, Y., Vaswani, N., Tannenbaum, A.: A generic framework for tracking using particle filter with dynamic shape prior. IEEE Trans. Image Process. 16(5), 1370–1382 (2007)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Ridder, D., Duin, R.: Locally linear embedding for classification. Technical report. PH-2002-01, Pattern Recognition Group, Delft University of Technology (2002)Google Scholar
  16. 16.
    Roweis, S., Saul, L.: Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500), 2323–2326 (2000)CrossRefGoogle Scholar
  17. 17.
    Sandhu, R., Dambreville, S., Tannenbaum, A.: Particle filtering for registration of 2D and 3D point sets with stochastic dynamics. In: IEEE CVPR (2008)Google Scholar
  18. 18.
    Sandhu, R., Dambreville, S., Yezzi, A., Tannenbaum, A.: Non-rigid 2D-3D pose estimation and 2D image segmentation. In: IEEE CVPR (2009)Google Scholar
  19. 19.
    Sethian, J.A.: Level Set Methods and Fast Marching Methods, 2nd edn. Cambridge University Press (1999)Google Scholar
  20. 20.
    Tenenbaum, J.B., de Silva, V., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar
  21. 21.
    Yezzi, A., Soatto, S.: Deformotion: Deforming motion, shape average and the joint registration and approximation of structures in images. Int. J. Comput. Vis. 53, 153–167 (2003)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Jeremy Lerner
    • 1
  • Romeil Sandhu
    • 2
  • Yongxin Chen
    • 3
  • Allen Tannenbaum
    • 4
    Email author
  1. 1.Department of Applied Mathematics & StatisticsStony Brook UniversityNew YorkUSA
  2. 2.Department of Biomedical InformaticsStony Brook UniversityNew YorkUSA
  3. 3.Department of Medical PhysicsMemorial Sloan Kettering Cancer CenterNew YorkUSA
  4. 4.Department of Computer Science at Stony BrookNew YorkUSA

Personalised recommendations