Abstract
In this paper, we propose a projection-based mixed reality system that visualizes the tangential deformation of a nonrigid surface by superimposing graphics directly onto the surface by projected imagery. The superimposed graphics are deformed according to the surface deformation. To achieve this goal, we develop a computer vision technique that estimates the tangential deformation by measuring the frame-by-frame movement of an infrared (IR) texture on the surface. IR ink, which can be captured by an IR camera under IR light, but is invisible to the human eye, is used to provide the surface texture. Consequently, the texture does not degrade the image quality of the augmented graphics. The proposed technique measures individually the surface motion between two successive frames. Therefore, it does not suffer from occlusions caused by interactions and allows touching, pushing, pulling, and pinching, etc. The moving least squares technique interpolates the measured result to estimate denser surface deformation. The proposed method relies only on the apparent motion measurement; thus, it is not limited to a specific deformation characteristic, but is flexible for multiple deformable materials, such as viscoelastic and elastic materials. Experiments confirm that, with the proposed method, we can visualize the surface deformation of various materials by projected illumination, even when the user’s hand occludes the surface from the camera.
Similar content being viewed by others
Notes
See http://en.wikipedia.org/wiki/Flubber_(material) for more information.
References
Alvarez L, Weickert J, Snchez J (2000) Reliable estimation of dense optical flow fields with large displacements. Int J Comput Vis 39(1):41–56
Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of the IEEE/ACM international symposium on augmented reality, pp 207–216
Bimber O, Raskar R (2005) Spatial augmented reality: merging real and virtual worlds. A. K. Peters, Ltd., USA
Bimber O, Iwai D (2008) Superimposing dynamic range. ACM Trans Graph 27(5):15:1–15:8
Bluteau J, Kitahara I, Kameda Y, Noma H, Kogure K, Ohta Y (2005) Visual support for medical communication by using projector-based augmented reality and thermal markers. In: Proceedings of the international conference on artificial reality and telexistence, pp 98–105
Chang RC, Tseng FC (2010) Automatic detection and correction for glossy reflections in digital photograph. In: 3rd IEEE international conference on Ubi-media computing (U-Media), pp 44–49
Follmer S, Johnson M, Adelson E, Ishii H (2011) deform: An interactive malleable surface for capturing 2.5d arbitrary objects, tools and touch. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, New York, UIST’11, pp 527–536
Fujimoto Y, Smith R, Taketomi T, Yamamoto G, Miyazaki J, Kato H, Thomas B (2014) Geometrically-correct projection-based texture mapping onto a deformable object. IEEE Trans Vis Comput Graph 20(4):540–549
Haouchine N, Dequidt J, Kerrien E, Berger MO, Cotin S (2012) Physics-based augmented reality for 3d deformable object. In: Workshop on virtual reality interaction and physical simulation
Heo S, Lee G (2013) Indirect shear force estimation for multi-point shear force operations. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, CHI’13, pp 281–284
Hisada M, Yamamoto K, Kanaya I, Sato K (2006) Free-form shape design system using stereoscopic projector—hyperreal 2.0. In: SICE-ICASE international joint conference, pp 4832–4835
Ito Y, Kim Y, Obinata G (2014) Acquisition of contact force and slippage using a vision-based tactile sensor with a fluid-type touchpad for the dexterous handling of robots. Adv Robot Autom 3(116)
Iwai D, Sato K (2010) Document search support by making physical documents transparent in projection-based mixed reality. Virtual Real 15(2–3):147–160
Johnson MK, Cole F, Raj A, Adelson EH (2011) Microgeometry capture using an elastomeric sensor. ACM Trans Graph 30(4):46:1–46:8
Jones BR, Benko H, Ofek E, Wilson AD (2013) Illumiroom: peripheral projected illusions for interactive experiences. In: Proceedings of the ACM annual conference on human factors in computing systems, pp 869–878
Kamiyama K, Vlack K, Mizota T, Kajimoto H, Kawakami N, Tachi S (2005) Vision-based sensor for real-time measuring of surface traction fields. IEEE Comput Graph Appl 25(1):68–75
Kocev B, Ritter F, Linsen L (2013) Projector-based surgeon–computer interaction on deformable surfaces. Int J Comput Assist Radiol Surg 8(6):1015–1025
Matoba Y, Sato T, Takahashi N, Koike H (2012) Claytricsurface: an interactive surface with dynamic softness control capability. In: ACM SIGGRAPH emerging technologies, p 6:1
Mine M, van Baar J, Grundhofer A, Rose D, Yang B (2012) Projection-based augmented reality in disney theme parks. IEEE Comput 45(7):32–40
Ni T, Karlson AK, Wigdor D (2011) AnatOnMe: facilitating doctor-patient communication using a projection-based handheld device. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 3333–3342
Pilet J, Lepetit V, Fua P (2008) Fast non-rigid surface detection, registration and realistic augmentation. Int J Comput Vis 76(2):109–122
Piper B, Ratti C, Ishii H (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 355–362
Raskar R, Welch G, Low KL, Bandyopadhyay D (2001) Shader lamps: animating real objects with image-based illumination. In: Proceedings of the eurographics workshop on rendering, pp 89–102
Raskar R, Ziegler R, Willwacher T (2006) Cartoon dioramas in motion. In: Proceedings of the international symposium on non-photorealistic animation and rendering, pp 7–12
Rivers A, Adams A, Durand F (2012) Sculpting by numbers. ACM Trans Graph 31(6):157:1–157:7
Saakes D, Chiu K, Hutchison T, Buczyk BM, Koizumi N, Inami M, Raskar R (2010) Slow display. In: ACM SIGGRAPH emerging technologies, p 22
Sato T, Mamiya H, Koike H, Fukuchi K (2009) PhotoelasticTouch: transparent rubbery tangible interface using an LCD and photoelasticity. In: Proceedings of the 22nd annual ACM symposium on User interface software and technology—UIST ’09. ACM Press, New York, pp 43–50
Schaefer S, McPhail T, Warren J (2006) Image deformation using moving least squares. ACM Trans Graph 25(3):533–540
Shi J, Tomasi C (1994) Good features to track. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 593–600
Shimazu S, Iwai D, Sato K (2011) 3d high dynamic range display system. In: Proceedings of the 10th IEEE/ACM international symposium on mixed and augmented reality, pp 235–236
Shimizu N, Yoshida T, Hayashi T, de Sorbier F, Saito H (2013) Non-rigid surface tracking for virtual fitting system. In: International conference on computer vision theory and applications, pp 1–7
Steimle J, Jordt A, Maes P (2013) Flexpad: highly flexible bending interactions for projected handheld displays. In: Proceedings of ACM SIGCHI conference on human factors in computing systems, pp 237–246
Uchiyama H, Marchand E (2011) Deformable random dot markers. In: Proceedings of IEEE international symposium on mixed and augmented reality, pp 237–238
Uchiyama H, Saito H (2011) Random dot markers. In: 2014 IEEE virtual reality (VR), pp 35–38
Vlack K, Mizota T, Kawakami N, Kamiyama K, Kajimoto H, Tachi S (2005) GelForce: a vision-based traction field computer interface. In: CHI ’05 extended abstracts on human factors in computing systems. ACM, New York, pp 1154–1155
Wu HY, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W (2012) Eulerian video magnification for revealing subtle changes in the world. ACM Trans Graph 31(4):65–72
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Punpongsanon, P., Iwai, D. & Sato, K. Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture. Virtual Reality 19, 45–56 (2015). https://doi.org/10.1007/s10055-014-0256-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-014-0256-y