Skip to main content
SpringerLink
Log in

Shape from interaction

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

We present “shape from interaction” (SfI), an approach to the problem of acquiring 3D representations of rigid objects through observing the activity of a human who handles a tool. SfI relies on the fact that two rigid objects cannot share the same physical space. The 3D reconstruction of the unknown object is achieved by tracking the known 3D tool and by carving out the space it occupies as a function of time. Due to this indirection, SfI reconstructs rigid objects regardless of their material and appearance properties and proves particularly useful for the cases of textureless, transparent, translucent, refractive and specular objects for which there exists no practical vision-based 3D reconstruction method. Additionally, object concavities that are not directly observable can also be reconstructed. The 3D tracking of the tool is formulated as an optimization problem that is solved based on visual input acquired by a multicamera system. Experimental results from a prototype implementation of SfI support qualitatively and quantitatively the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Adato, Y., Vasilyev, Y., Ben-Shahar, O., Zickler, T.: Toward a theory of shape from specular flow. In: IEEE International Conference on Computer Vision, pp. 1–8 (2007)

  2. Agarwal, S., Mallick, S., Kriegman, D., Belongie, S.: On refractive optical flow. In: European Conference on Computer Vision, pp. 483–494 (2004)

  3. Aloimonos, Y.: Shape from texture. Biol. Cybern. 58(5), 345–360 (1988)

    Article  MATH  Google Scholar 

  4. Argyros, A.A., Lourakis, M.: Real-time tracking of multiple skin-colored objects with a possibly moving camera. In: European Conference on Computer Vision, pp. 368–379. Springer, New York (2004)

  5. Atcheson, B., Ihrke, I., Bradley, D., Heidrich, W., Magnor, M., Seidel, H.-P.: Imaging and 3D tomographic reconstruction of time-varying inhomogeneous refractive index fields. Tech. Rep. University of British Columbia, UBC CS TR-2007-06 (2007)

  6. Ben-Ezra, M., Nayar, S.: What does motion reveal about transparency? In: IEEE International Conference on Computer Vision (2003)

  7. Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)

    Google Scholar 

  8. Blais, F.: Review of 20 years of range sensor development. J. Electron. Imaging 13, 231 (2004)

    Article  Google Scholar 

  9. Bouguet, J., Perona, P.: 3D photography on your desk. In: IEEE International Conference on Computer Vision, pp. 43–50 (1998)

  10. Chen, T., Goesele, M., Seidel, H.: Mesostructure from specularity. In: IEEE Conference on Computer Vision and Pattern Recognition. vol. 2, pp. 1825–1832 (2006)

  11. Chen, T., Lensch, H., Fuchs, C., Seidel, H., Informatik, M.: Polarization and phaseshifting for 3D scanning of translucent objects. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)

  12. Clerc, M., Kennedy, J.: The particle swarm—explosion, stability, and convergence in a multidimensional complex space. Trans. Evol. Comput. 6(1), 58–73 (2002)

    Article  Google Scholar 

  13. Collins, R.: A space-sweep approach to true multi-image matching. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 358–363 (1996)

  14. Eren, G., Aubreton, O., Meriaudeau, F., Secades, S., Fofi, D., Naskali, T., Truchetet, F., Ercil, A.: Scanning from heating: 3D shape estimation of transparent objects from local surface heating. Opt. Express 17(14), 11457–11468 (2009)

    Article  Google Scholar 

  15. Faugeras, O., Luong, Q., Papadopoulos, T.: The Geometry of Multiple Images. MIT Press, Cambridge (2001)

    MATH  Google Scholar 

  16. Guan, L., Franco, J., Pollefeys, M.: Multi-view occlusion reasoning for probabilistic silhouette-based dynamic scene reconstruction. Int. J. Comput. Vis. 90(3), 283–303 (2010)

    Google Scholar 

  17. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004). ISBN: 0521540518

    Book  MATH  Google Scholar 

  18. Helwig, S., Wanka, R.: Particle swarm optimization in high-dimensional bounded search spaces. In: Proceedings of IEEE Swarm Intelligence Symposium, pp. 198–205 (2007)

  19. Ihrke, I., Kutulakos, K., Lensch, H., Magnor, M., Heidrich, W.: Transparent and specular object reconstruction. Comput. Graph. Forum 29(8), 2400–2426 (2010)

    Article  Google Scholar 

  20. Ikeuchi, K.: Determining surface orientations of specular surfaces by using the photometric stereo method. IEEE Trans. Pattern Anal. Mach. Intell. 3(6), 661–669 (1981)

    Google Scholar 

  21. John, V., Ivekovic, S., Trucco, E.: Articulated human motion tracking with HPSO. In: Proceedings of International Conference on Computer Vision Theory and Applications (2009)

  22. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: IEEE International Conference on Neural Networks, vol. 4. pp. 1942–1948 (1995)

  23. Klank, U., Carton, D., Beetz, M.: Transparent object detection and reconstruction on a mobile platform. In: IEEE International Conference on Robotics and Automation, pp. 5971–5978 (2011)

  24. Kutulakos, K., Seitz, S.: A theory of shape by space carving. Int. J. Comput. Vis. 38, 307–314 (2000)

    Google Scholar 

  25. Kutulakos, K., Steger, E.: A theory of refractive and specular 3D shape by light-path triangulation. In: IEEE International Conference on Computer Vision, pp. 1448–1455 (2005)

  26. Kutulakos, K., Vallino, J.: Calibration-free augmented reality. IEEE Trans. Vis. Comput. Graph. 4(1), 1–20 (1998)

    Article  Google Scholar 

  27. Kyriazis, N., Argyros, A.A.: Physically plausible 3d scene tracking: the single actor hypothesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9–16 (2013)

  28. Li, Y., Lin, S., Lu, H., Kang, S., Shum, H.: Multibaseline stereo in the presence of specular reflections. In: International Conference on Pattern Recognition, pp. 573–576 (2002)

  29. Lorensen, W., Cline, H.: Marching cubes: a high resolution 3D surface construction algorithm. In: SIGGRAPH, pp. 163–169 (1987)

  30. Matsuyama, T., Wu, X., Takai, T., Nobuhara, S.: Real-time 3D shape reconstruction, dynamic 3D mesh deformation, and high fidelity visualization for 3D video. CVIU 96(3), 1077–3142 (2004)

    Google Scholar 

  31. Zhang, Z.: Microsoft kinect sensor and its effect. IEEE MultiMedia 19(2), 4–12 (2012)

    Google Scholar 

  32. Miyazaki, D., Kagesawa, M., Ikeuchi, K.: Transparent surface modeling from a pair of polarization images. IEEE Trans. IEEE Trans. Pattern Anal. Mach. Intell. 26(1), 73–82 (2004)

    Google Scholar 

  33. Moravec, H.: Towards automatic visual obstacle avoidance. In: International Joint Conference on Artificial Intelligence, pp. 584–594 (1977)

  34. Nayar, S., Fang, X., Boult, T.: Removal of specularities using color and polarization. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 583–590 (1993)

  35. Nayar, S., Sanderson, A., Weiss, L., Simon, D.: Specular surface inspection using structured highlight and gaussian images. IEEE Trans. Robot. Autom. 6(2), 208–218 (1990)

    Article  Google Scholar 

  36. Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Markerless and efficient 26-DOF hand pose recovery. In: Asian Conference on Computer Vision, pp. 744–757 (2010)

  37. Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Efficient model-based 3D tracking of hand articulations using Kinect. In: British Machine Vision Conference, Dundee (2011)

  38. Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In: IEEE International Conference on Computer Vision, pp. 2088–2095 (2011b)

  39. Oikonomidis, I., Kyriazis, N., Argyros, A.A.: Tracking the articulated motion of two strongly interacting hands. In: IEEE Conference on Computer Vision and Pattern Recognition (2012)

  40. Padeleris, P., Zabulis, X., Argyros, A.: Head pose estimation on depth data based on particle swarm optimization. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, Workshop on Human Activity Understanding from 3D Data (HAU3D) (2012)

  41. Remondino, F., El-Hakim, S.: Image-based 3D modelling: a review. Photogramm. Rec. 21, 269–291 (2006)

    Article  Google Scholar 

  42. Sarmis, T., Zabulis, X., Argyros, A.A.: A checkerboard detection utility for intrinsic and extrinsic camera cluster calibration. Tech. Rep. 397, FORTH-ICS (2009)

  43. Savarese, S., Chen, M., Perona, P.: Local shape from mirror reflections. Int. J. Comput. Vis. 64(1), 31–67 (2005)

    Google Scholar 

  44. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 195–202 (2003)

  45. Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)

  46. Tarini, M., Lensch, H., Goesele, M., Seidel, H.: 3D acquisition of mirroring objects using striped patterns. Graph. Models 67(4), 233–259 (2005)

    Article  Google Scholar 

  47. Wetzstein, G., Roodnick, D., Raskar, R., Heidrich, W.: Refractive shape from light field distortion. In: IEEE International Conference on Computer Vision (2011)

  48. Woodham, R.: Photometric method for determining surface orientation from multiple images. Opt. Eng. 19(1), 513–531 (1980)

    Google Scholar 

  49. Zhang, R., Tsai, P., Cryer, J., Shah, M.: Shape from shading: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 21(8), 690–706 (1999)

    Google Scholar 

  50. Zheng, J., Murata, A.: Acquiring a complete 3D model from specular motion under the illumination of circular-shaped light sources. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 913–920 (2000)

    Google Scholar 

  51. Zivkovic, Z.: Improved adaptive Gaussian mixture model for background subtraction. In: International Conference on Pattern Recognition, pp. 28–31 (2004)

Download references

Acknowledgments

This work was partially supported by the EU IST-FP7-IP-288533 project RoboHow.Cog.

Author information

Authors and Affiliations

  1. Institute of Computer Science, FORTH, Heraklion, Greece

    Damien Michel, Xenophon Zabulis & Antonis A. Argyros

  2. Computer Science Department, University of Crete, Heraklion, Greece

    Antonis A. Argyros

Authors
  1. Damien Michel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xenophon Zabulis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonis A. Argyros
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonis A. Argyros.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 33064 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Michel, D., Zabulis, X. & Argyros, A.A. Shape from interaction. Machine Vision and Applications 25, 1077–1087 (2014). https://doi.org/10.1007/s00138-014-0602-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00138-014-0602-9

Keywords

Search

Navigation