Machine Vision and Applications

, Volume 25, Issue 4, pp 1077–1087 | Cite as

Shape from interaction

  • Damien Michel
  • Xenophon Zabulis
  • Antonis A. Argyros
Original Paper

Abstract

We present “shape from interaction” (SfI), an approach to the problem of acquiring 3D representations of rigid objects through observing the activity of a human who handles a tool. SfI relies on the fact that two rigid objects cannot share the same physical space. The 3D reconstruction of the unknown object is achieved by tracking the known 3D tool and by carving out the space it occupies as a function of time. Due to this indirection, SfI reconstructs rigid objects regardless of their material and appearance properties and proves particularly useful for the cases of textureless, transparent, translucent, refractive and specular objects for which there exists no practical vision-based 3D reconstruction method. Additionally, object concavities that are not directly observable can also be reconstructed. The 3D tracking of the tool is formulated as an optimization problem that is solved based on visual input acquired by a multicamera system. Experimental results from a prototype implementation of SfI support qualitatively and quantitatively the effectiveness of the proposed approach.

Keywords

3D reconstruction Object tracking  3D pose estimation Transparent objects 

Supplementary material

138_2014_602_MOESM1_ESM.mp4 (32.3 mb)
Supplementary material 1 (mp4 33064 KB)

References

  1. 1.
    Adato, Y., Vasilyev, Y., Ben-Shahar, O., Zickler, T.: Toward a theory of shape from specular flow. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007)Google Scholar
  2. 2.
    Agarwal, S., Mallick, S., Kriegman, D., Belongie, S.: On refractive optical flow. In: European Conference on Computer Vision, pp. 483–494 (2004)Google Scholar
  3. 3.
    Aloimonos, Y.: Shape from texture. Biol. Cybern. 58(5), 345–360 (1988)CrossRefMATHGoogle Scholar
  4. 4.
    Argyros, A.A., Lourakis, M.: Real-time tracking of multiple skin-colored objects with a possibly moving camera. In: European Conference on Computer Vision, pp. 368–379. Springer, New York (2004)Google Scholar
  5. 5.
    Atcheson, B., Ihrke, I., Bradley, D., Heidrich, W., Magnor, M., Seidel, H.-P.: Imaging and 3D tomographic reconstruction of time-varying inhomogeneous refractive index fields. Tech. Rep. University of British Columbia, UBC CS TR-2007-06 (2007)Google Scholar
  6. 6.
    Ben-Ezra, M., Nayar, S.: What does motion reveal about transparency? In: IEEE International Conference on Computer Vision (2003)Google Scholar
  7. 7.
    Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)Google Scholar
  8. 8.
    Blais, F.: Review of 20 years of range sensor development. J. Electron. Imaging 13, 231 (2004)CrossRefGoogle Scholar
  9. 9.
    Bouguet, J., Perona, P.: 3D photography on your desk. In: IEEE International Conference on Computer Vision, pp. 43–50 (1998)Google Scholar
  10. 10.
    Chen, T., Goesele, M., Seidel, H.: Mesostructure from specularity. In: IEEE Conference on Computer Vision and Pattern Recognition. vol. 2, pp. 1825–1832 (2006)Google Scholar
  11. 11.
    Chen, T., Lensch, H., Fuchs, C., Seidel, H., Informatik, M.: Polarization and phaseshifting for 3D scanning of translucent objects. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  12. 12.
    Clerc, M., Kennedy, J.: The particle swarm—explosion, stability, and convergence in a multidimensional complex space. Trans. Evol. Comput. 6(1), 58–73 (2002)CrossRefGoogle Scholar
  13. 13.
    Collins, R.: A space-sweep approach to true multi-image matching. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 358–363 (1996)Google Scholar
  14. 14.
    Eren, G., Aubreton, O., Meriaudeau, F., Secades, S., Fofi, D., Naskali, T., Truchetet, F., Ercil, A.: Scanning from heating: 3D shape estimation of transparent objects from local surface heating. Opt. Express 17(14), 11457–11468 (2009)CrossRefGoogle Scholar
  15. 15.
    Faugeras, O., Luong, Q., Papadopoulos, T.: The Geometry of Multiple Images. MIT Press, Cambridge (2001)MATHGoogle Scholar
  16. 16.
    Guan, L., Franco, J., Pollefeys, M.: Multi-view occlusion reasoning for probabilistic silhouette-based dynamic scene reconstruction. Int. J. Comput. Vis. 90(3), 283–303 (2010)Google Scholar
  17. 17.
    Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004). ISBN: 0521540518CrossRefMATHGoogle Scholar
  18. 18.
    Helwig, S., Wanka, R.: Particle swarm optimization in high-dimensional bounded search spaces. In: Proceedings of IEEE Swarm Intelligence Symposium, pp. 198–205 (2007)Google Scholar
  19. 19.
    Ihrke, I., Kutulakos, K., Lensch, H., Magnor, M., Heidrich, W.: Transparent and specular object reconstruction. Comput. Graph. Forum 29(8), 2400–2426 (2010)CrossRefGoogle Scholar
  20. 20.
    Ikeuchi, K.: Determining surface orientations of specular surfaces by using the photometric stereo method. IEEE Trans. Pattern Anal. Mach. Intell. 3(6), 661–669 (1981)Google Scholar
  21. 21.
    John, V., Ivekovic, S., Trucco, E.: Articulated human motion tracking with HPSO. In: Proceedings of International Conference on Computer Vision Theory and Applications (2009)Google Scholar
  22. 22.
    Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks, vol. 4. pp. 1942–1948 (1995)Google Scholar
  23. 23.
    Klank, U., Carton, D., Beetz, M.: Transparent object detection and reconstruction on a mobile platform. In: IEEE International Conference on Robotics and Automation, pp. 5971–5978 (2011)Google Scholar
  24. 24.
    Kutulakos, K., Seitz, S.: A theory of shape by space carving. Int. J. Comput. Vis. 38, 307–314 (2000)Google Scholar
  25. 25.
    Kutulakos, K., Steger, E.: A theory of refractive and specular 3D shape by light-path triangulation. In: IEEE International Conference on Computer Vision, pp. 1448–1455 (2005)Google Scholar
  26. 26.
    Kutulakos, K., Vallino, J.: Calibration-free augmented reality. IEEE Trans. Vis. Comput. Graph. 4(1), 1–20 (1998)CrossRefGoogle Scholar
  27. 27.
    Kyriazis, N., Argyros, A.A.: Physically plausible 3d scene tracking: the single actor hypothesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9–16 (2013)Google Scholar
  28. 28.
    Li, Y., Lin, S., Lu, H., Kang, S., Shum, H.: Multibaseline stereo in the presence of specular reflections. In: International Conference on Pattern Recognition, pp. 573–576 (2002)Google Scholar
  29. 29.
    Lorensen, W., Cline, H.: Marching cubes: a high resolution 3D surface construction algorithm. In: SIGGRAPH, pp. 163–169 (1987)Google Scholar
  30. 30.
    Matsuyama, T., Wu, X., Takai, T., Nobuhara, S.: Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video. CVIU 96(3), 1077–3142 (2004)Google Scholar
  31. 31.
    Zhang, Z.: Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2), 4–12 (2012)Google Scholar
  32. 32.
    Miyazaki, D., Kagesawa, M., Ikeuchi, K.: Transparent surface modeling from a pair of polarization images. IEEE Trans. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 73–82 (2004)Google Scholar
  33. 33.
    Moravec, H.: Towards automatic visual obstacle avoidance. In: International Joint Conference on Artificial Intelligence, pp. 584–594 (1977)Google Scholar
  34. 34.
    Nayar, S., Fang, X., Boult, T.: Removal of specularities using color and polarization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 583–590 (1993)Google Scholar
  35. 35.
    Nayar, S., Sanderson, A., Weiss, L., Simon, D.: Specular surface inspection using structured highlight and gaussian images. IEEE Trans. Robot. Autom. 6(2), 208–218 (1990)CrossRefGoogle Scholar
  36. 36.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Markerless and efficient 26-DOF hand pose recovery. In: Asian Conference on Computer Vision, pp. 744–757 (2010)Google Scholar
  37. 37.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Efficient model-based 3D tracking of hand articulations using Kinect. In: British Machine Vision Conference, Dundee (2011)Google Scholar
  38. 38.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In: IEEE International Conference on Computer Vision, pp. 2088–2095 (2011b)Google Scholar
  39. 39.
    Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Tracking the articulated motion of two strongly interacting hands. In: IEEE Conference on Computer Vision and Pattern Recognition (2012)Google Scholar
  40. 40.
    Padeleris, P., Zabulis, X., Argyros, A.: Head pose estimation on depth data based on particle swarm optimization. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, Workshop on Human Activity Understanding from 3D Data (HAU3D) (2012)Google Scholar
  41. 41.
    Remondino, F., El-Hakim, S.: Image-based 3D modelling: a review. Photogramm. Rec. 21, 269–291 (2006)CrossRefGoogle Scholar
  42. 42.
    Sarmis, T., Zabulis, X., Argyros, A.A.: A checkerboard detection utility for intrinsic and extrinsic camera cluster calibration. Tech. Rep. 397, FORTH-ICS (2009)Google Scholar
  43. 43.
    Savarese, S., Chen, M., Perona, P.: Local shape from mirror reflections. Int. J. Comput. Vis. 64(1), 31–67 (2005)Google Scholar
  44. 44.
    Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 195–202 (2003)Google Scholar
  45. 45.
    Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)Google Scholar
  46. 46.
    Tarini, M., Lensch, H., Goesele, M., Seidel, H.: 3D acquisition of mirroring objects using striped patterns. Graph. Models 67(4), 233–259 (2005)CrossRefGoogle Scholar
  47. 47.
    Wetzstein, G., Roodnick, D., Raskar, R., Heidrich, W.: Refractive shape from light field distortion. In: IEEE International Conference on Computer Vision (2011)Google Scholar
  48. 48.
    Woodham, R.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 513–531 (1980)Google Scholar
  49. 49.
    Zhang, R., Tsai, P., Cryer, J., Shah, M.: Shape from shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)Google Scholar
  50. 50.
    Zheng, J., Murata, A.: Acquiring a complete 3D model from specular motion under the illumination of circular-shaped light sources. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 913–920 (2000)Google Scholar
  51. 51.
    Zivkovic, Z.: Improved adaptive Gaussian mixture model for background subtraction. In: International Conference on Pattern Recognition, pp. 28–31 (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Damien Michel
    • 1
  • Xenophon Zabulis
    • 1
  • Antonis A. Argyros
    • 1
    • 2
  1. 1.Institute of Computer ScienceFORTHHeraklionGreece
  2. 2.Computer Science DepartmentUniversity of CreteHeraklionGreece

Personalised recommendations