Virtual Reality

, Volume 19, Issue 1, pp 45–56 | Cite as

Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture

  • Parinya PunpongsanonEmail author
  • Daisuke Iwai
  • Kosuke Sato
Original Article


In this paper, we propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics are deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light, but is invisible to the human eye, is used to provide the surface texture. Consequently, the texture does not degrade the image quality of the augmented graphics. The proposed technique measures individually the surface motion between two successive frames. Therefore, it does not suffer from occlusions caused by interactions and allows touching, pushing, pulling, and pinching, etc. The moving least squares technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that, with the proposed method, we can visualize the surface deformation of various materials by projected illumination, even when the user’s hand occludes the surface from the camera.


Projection-based mixed reality User interaction Deformable surface 


  1. Alvarez L, Weickert J, Snchez J (2000) Reliable estimation of dense optical flow fields with large displacements. Int J Comput Vis 39(1):41–56CrossRefzbMATHGoogle Scholar
  2. Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of the IEEE/ACM international symposium on augmented reality, pp 207–216Google Scholar
  3. Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A. K. Peters, Ltd., USAGoogle Scholar
  4. Bimber O, Iwai D (2008) Superimposing dynamic range. ACM Trans Graph 27(5):15:1–15:8CrossRefGoogle Scholar
  5. Bluteau J, Kitahara I, Kameda Y, Noma H, Kogure K, Ohta Y (2005) Visual support for medical communication by using projector-based augmented reality and thermal markers. In: Proceedings of the international conference on artificial reality and telexistence, pp 98–105Google Scholar
  6. Chang RC, Tseng FC (2010) Automatic detection and correction for glossy reflections in digital photograph. In: 3rd IEEE international conference on Ubi-media computing (U-Media), pp 44–49Google Scholar
  7. Follmer S, Johnson M, Adelson E, Ishii H (2011) deform: An interactive malleable surface for capturing 2.5d arbitrary objects, tools and touch. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, New York, UIST’11, pp 527–536Google Scholar
  8. Fujimoto Y, Smith R, Taketomi T, Yamamoto G, Miyazaki J, Kato H, Thomas B (2014) Geometrically-correct projection-based texture mapping onto a deformable object. IEEE Trans Vis Comput Graph 20(4):540–549CrossRefGoogle Scholar
  9. Haouchine N, Dequidt J, Kerrien E, Berger MO, Cotin S (2012) Physics-based augmented reality for 3d deformable object. In: Workshop on virtual reality interaction and physical simulationGoogle Scholar
  10. Heo S, Lee G (2013) Indirect shear force estimation for multi-point shear force operations. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, CHI’13, pp 281–284Google Scholar
  11. Hisada M, Yamamoto K, Kanaya I, Sato K (2006) Free-form shape design system using stereoscopic projector—hyperreal 2.0. In: SICE-ICASE international joint conference, pp 4832–4835Google Scholar
  12. Ito Y, Kim Y, Obinata G (2014) Acquisition of contact force and slippage using a vision-based tactile sensor with a fluid-type touchpad for the dexterous handling of robots. Adv Robot Autom 3(116)Google Scholar
  13. Iwai D, Sato K (2010) Document search support by making physical documents transparent in projection-based mixed reality. Virtual Real 15(2–3):147–160Google Scholar
  14. Johnson MK, Cole F, Raj A, Adelson EH (2011) Microgeometry capture using an elastomeric sensor. ACM Trans Graph 30(4):46:1–46:8CrossRefGoogle Scholar
  15. Jones BR, Benko H, Ofek E, Wilson AD (2013) Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the ACM annual conference on human factors in computing systems, pp 869–878Google Scholar
  16. Kamiyama K, Vlack K, Mizota T, Kajimoto H, Kawakami N, Tachi S (2005) Vision-based sensor for real-time measuring of surface traction fields. IEEE Comput Graph Appl 25(1):68–75CrossRefGoogle Scholar
  17. Kocev B, Ritter F, Linsen L (2013) Projector-based surgeon–computer interaction on deformable surfaces. Int J Comput Assist Radiol Surg 8(6):1015–1025CrossRefGoogle Scholar
  18. Matoba Y, Sato T, Takahashi N, Koike H (2012) Claytricsurface: an interactive surface with dynamic softness control capability. In: ACM SIGGRAPH emerging technologies, p 6:1Google Scholar
  19. Mine M, van Baar J, Grundhofer A, Rose D, Yang B (2012) Projection-based augmented reality in disney theme parks. IEEE Comput 45(7):32–40CrossRefGoogle Scholar
  20. Ni T, Karlson AK, Wigdor D (2011) AnatOnMe: facilitating doctor-patient communication using a projection-based handheld device. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 3333–3342Google Scholar
  21. Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122CrossRefGoogle Scholar
  22. Piper B, Ratti C, Ishii H (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 355–362Google Scholar
  23. Raskar R, Welch G, Low KL, Bandyopadhyay D (2001) Shader lamps: animating real objects with image-based illumination. In: Proceedings of the eurographics workshop on rendering, pp 89–102Google Scholar
  24. Raskar R, Ziegler R, Willwacher T (2006) Cartoon dioramas in motion. In: Proceedings of the international symposium on non-photorealistic animation and rendering, pp 7–12Google Scholar
  25. Rivers A, Adams A, Durand F (2012) Sculpting by numbers. ACM Trans Graph 31(6):157:1–157:7CrossRefGoogle Scholar
  26. Saakes D, Chiu K, Hutchison T, Buczyk BM, Koizumi N, Inami M, Raskar R (2010) Slow display. In: ACM SIGGRAPH emerging technologies, p 22Google Scholar
  27. Sato T, Mamiya H, Koike H, Fukuchi K (2009) PhotoelasticTouch: transparent rubbery tangible interface using an LCD and photoelasticity. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology—UIST ’09. ACM Press, New York, pp 43–50Google Scholar
  28. Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. ACM Trans Graph 25(3):533–540CrossRefGoogle Scholar
  29. Shi J, Tomasi C (1994) Good features to track. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 593–600Google Scholar
  30. Shimazu S, Iwai D, Sato K (2011) 3d high dynamic range display system. In: Proceedings of the 10th IEEE/ACM international symposium on mixed and augmented reality, pp 235–236Google Scholar
  31. Shimizu N, Yoshida T, Hayashi T, de Sorbier F, Saito H (2013) Non-rigid surface tracking for virtual fitting system. In: International conference on computer vision theory and applications, pp 1–7Google Scholar
  32. Steimle J, Jordt A, Maes P (2013) Flexpad: highly flexible bending interactions for projected handheld displays. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 237–246Google Scholar
  33. Uchiyama H, Marchand E (2011) Deformable random dot markers. In: Proceedings of IEEE international symposium on mixed and augmented reality, pp 237–238Google Scholar
  34. Uchiyama H, Saito H (2011) Random dot markers. In: 2014 IEEE virtual reality (VR), pp 35–38Google Scholar
  35. Vlack K, Mizota T, Kawakami N, Kamiyama K, Kajimoto H, Tachi S (2005) GelForce: a vision-based traction field computer interface. In: CHI ’05 extended abstracts on human factors in computing systems. ACM, New York, pp 1154–1155Google Scholar
  36. Wu HY, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph 31(4):65–72CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Parinya Punpongsanon
    • 1
    Email author
  • Daisuke Iwai
    • 1
  • Kosuke Sato
    • 1
  1. 1.Osaka UniversityToyonakaJapan

Personalised recommendations