Dynamic Projection Mapping on Multiple Non-rigid Moving Objects for Stage Performance Applications

  • Ryohei NakatsuEmail author
  • Ningfeng Yang
  • Hirokazu Takata
  • Takashi Nakanishi
  • Makoto Kitaguchi
  • Naoko Tosa
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11112)


In this paper we have proposed and developed a projection mapping system that can project 3D images on multiple moving objects that change their positions and shapes. Although projection mapping has become a new type of image projection method, so far most of the projection mappings have focused on the projection of 3D images on objects such as buildings that stay static without changing their shapes. If projection mapping on non-rigid moving objects such as human performers is realized, application area of projection mapping will become far wider.

To demonstrate this, a simple test of integrating projection mapping and Noh performance was carried out. Although Noh performance has been considered difficult to understand for beginners and foreigners because of its minimalism, by projecting 2D images that suggest inner emotion of a main character and also ongoing story, the evaluation result show that this new type of Noh performance could make it easier to be understood. This means that the integration of projection mapping and various stage performances could open-up new possibilities for various kinds of performing arts to add their values.

To realize this, we have proposed a projection mapping system consisting of multiple depth sensors and multiple projectors. In the first step, using multiple depth sensors the position and shape of multiple objects are detected in real time and the obtained 3D models of the multiple objects are sent to the projection phase. In the projection phase, the images to be projected on the objects are rendered on the obtained 3D model of the object. Then using multiple projectors real-time 3D projection mapping is carried out on the multiple moving objects. The system is now under development and demonstration of actual Noh performance using this technology will be achieved.


Projection mapping Depth sensor Moving object Performing arts Noh performance Japanese culture Inner emotion Media art 


  1. 1.
    Murayama, S., Torii, I., Ishii, N.: Development of projection mapping with utility of digital signage. In: IIAI 3rd International Conference on Advanced Applied Informatics, pp. 895–900 (2014)Google Scholar
  2. 2.
    Rowe, A.: Designing for engagement in mixed reality experiences that combine projection mapping and camera-based interaction. Digit. Creat. 25(2), 155–168 (2013)CrossRefGoogle Scholar
  3. 3.
    Bimber, O., Raskar, R.: Spatial Augmented Reality; Merging Real and Virtual Worlds. CRC Press, Boca Raton (2005)CrossRefGoogle Scholar
  4. 4.
    Mine, M., Rose, D., Yang, B., van Baar, J., Grundhofer, A.: Projection-based augmented reality in Disney theme parks. Computer 45(7), 32–40 (2012)CrossRefGoogle Scholar
  5. 5.
    Maniello, D.: Augmented Reality in Public Spaces Basic Techniques for Video Mapping, vol. 1. Paperback (2015)Google Scholar
  6. 6.
    Tosa, N., Nakatsu, R., Yunian, P.: Creation of media art utilizing fluid dynamics. In: 2017 International Conference on Culture and Computing, pp. 130–135 (2017)Google Scholar
  7. 7.
    Yunian, P, Liang, Z., Nakatsu, R., Tosa, N.: A study of variable control of Sound Vibration Form (SVF) for media art creation. In: 2017 International Conference on Culture and Computing, pp. 137–142 (2017)Google Scholar
  8. 8.
    Yasumura, T.: Rimpa: Decorative Japanese Painting. PIE Books (2011)Google Scholar
  9. 9.
    Tosa, N., Nakatsu, R., Yunian, P.: Projection mapping celebrating RIMPA 400th Anniversary. In: 2015 International Conference on Culture and Computing (2015)Google Scholar
  10. 10.
    Sadler, A.L.: Japanese Plays: Classic Noh, Kyogen and Kabuki Works. Tuttle Publishing, Clarendon (2011)Google Scholar
  11. 11.
    Raskar, R., Welch, G., Low, K.L., Bandyopadhyay, D.: Shader lamps: animating real objects with image-based illumination. In: Gortler, S.J., Myszkowski, K. (eds.) Rendering Techniques 2001. Eurographics. Springer, Vienna (2001)Google Scholar
  12. 12.
    Bimber, O., Iwai, D., Wetzstein, G., Grundhofer, A.: The visual computing of projector-camera systems. Comput. Graph. Forum 27(8), 2219–2245 (2008)CrossRefGoogle Scholar
  13. 13.
    Fua, P., Leclerc, Y.G.: Object-centered surface reconstruction: combining multi-image stereo and shading. Int. J. Comput. Vis. 16(1), 35–56 (1995)CrossRefGoogle Scholar
  14. 14.
    Gay-Bellile, V., Bartoli, A., Sayd, P.: Deformable surface augmentation in spite of self-occlusions. In: Proceedings of 6th IEEE ACM International Symposium on Mixed Augmented Reality, pp. 235–238 (2007)Google Scholar
  15. 15.
    Uchiyama, H., Marchand, E.: Object detection and pose tracking for augmented reality: recent approaches. In: Proceedings of 18th Korea-Japan Joint Workshop Frontiers Computer Vision, pp. 1–8 (2012)Google Scholar
  16. 16.
    Park, H., Park, J.-I.: Invisible marker based augmented reality system. In: Proceedings of SPIE Visual Communication Image Processing, Art. No. 59601 (2006)Google Scholar
  17. 17.
    Guskov, I.: Efficient tracking of regular patterns on non-ridig geometry. In: Proceedings of 16th International Conference on Pattern Recognition, vol. 2, pp. 1057–1060 (2002)Google Scholar
  18. 18.
    Narita, G., Watanabe, Y., Ishikawa, M.: Dynamic projctin mapping onto deforming non-ridig surface using deformable dot cluster marker. IEEE Trans. Visual. Comput. Graph. 23(3), 1235–1248 (2017)CrossRefGoogle Scholar
  19. 19.
    Jones, B.R., Benko, H., Ofek, E., Wilson, A.D.: Illumiroom: peripheral projected illusions for interactive experiences. In: ACM SIGGRAPH 2013 Emerging Technologies, SIGGRAPH 2013, p. 7:1. ACM, New York (2013)Google Scholar
  20. 20.
    Siegl, C., et al.: Real-time pixel luminance optimization for dynamic multi-projection mapping. ACM Trans. Graph. 34(6), 237:1–237:11 (2015)CrossRefGoogle Scholar

Copyright information

© IFIP International Federation for Information Processing 2018

Authors and Affiliations

  • Ryohei Nakatsu
    • 1
    Email author
  • Ningfeng Yang
    • 1
  • Hirokazu Takata
    • 2
  • Takashi Nakanishi
    • 2
  • Makoto Kitaguchi
    • 3
  • Naoko Tosa
    • 4
  1. 1.NT & AssociatesKyotoJapan
  2. 2.TakumiVisionKyotoJapan
  3. 3.TECHMACKyotoJapan
  4. 4.Graduate School of Advanced Integrated Studies in Human SurvivabilityKyoto UniversityKyotoJapan

Personalised recommendations