Machine Vision and Applications

, Volume 20, Issue 2, pp 85–92 | Cite as

Augmented reality on cloth with realistic illumination

  • Derek BradleyEmail author
  • Gerhard Roth
  • Prosenjit Bose
Original Paper


Augmented reality (AR) is the concept of inserting virtual objects into real scenes. Often, augmentations are aligned with rigid planar objects in the scene. However, a more difficult task is to align non-rigid augmentations with flexible objects like cloth. To address this problem, we present a method to perform real-time flexible augmentations on cloth. Our method involves sparse cloth-tracking in video images using a new vision-based marker system with temporal coherence. We include an image-based method to automatically acquire real world illumination and shadows from the input frame. Non-rigid augmentations are achieved by rendering a textured 2D mesh aligned with the cloth surface, and combining the illumination result. The ability to perform realistic augmentations on cloth leads to applications in fashion, advertising, home decor and entertainment. We demonstrate our cloth augmentations with an application to interactively design T-shirts by demonstrating different virtual logos on a physical shirt in real-time.


Augmented reality Non-rigid object tracking Common illumination Marker systems 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Ballester C., Bertalmio M., Caselles V., Sapiro G. and Verdera J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE Trans. on Image Process. 10(8): 1200–1211 zbMATHCrossRefMathSciNetGoogle Scholar
  3. 3.
    Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.: Image inpainting. In: K. Akeley (ed.) Siggraph 2000, Computer Graphics Proceedings, pp. 417–424. ACM Press/ACM SIGGRAPH/Addison Wesley Longman, New York (2000).
  4. 4.
    Billinghurst M., Kato H. and Poupyrev I. (2001). The magicbook—moving seamlessly between reality and virtuality. IEEE Comput. Graph. Appl. 21(3): 6–8 Google Scholar
  5. 5.
    Bradley, D., Roth, G.: Augmenting non-rigid objects with realistic lighting. Technical Report NRC 47398, National Research Council of Canada (2004)Google Scholar
  6. 6.
    Bradley D. and Roth G. (2007). Adaptive thresholding using the integral image. ACM J. Graph. Tools 12(2): 13–21 Google Scholar
  7. 7.
    Bridson, R., Marino, S., Fedkiw, R.: Simulation of clothing with folds and wrinkles. In: SCA ’03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, pp. 28–36. Eurographics Association, Aire-la-Ville, Switzerland, Switzerland (2003)Google Scholar
  8. 8.
    Comport, A.I., Marchand, E., Chaumette, F.: A real-time tracker for markerless augmented reality. In: IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2003), pp. 36–45. Tokyo, Japan (2003)Google Scholar
  9. 9.
    Cormen T.H., Stein C., Rivest R.L. and Leiserson C.E. (2001). Introduction to Algorithms. McGraw-Hill Higher Education, New York zbMATHGoogle Scholar
  10. 10.
    Debevec, P.: Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the 25th annual conference on Computer graphics and interactive techniques, pp. 189–198. ACM Press, New York (1998).
  11. 11.
    Drettakis, G., Robert, L., Bougnoux, S.: Interactive common illumination for computer augmented reality. In: Proceedings of the Eurographics Workshop on Rendering Techniques ’97, pp. 45–56. Springer, Heidelberg (1997)Google Scholar
  12. 12.
    Ehara, J., Saito, H.: Texture overlay for virtual clothing based on pca of silhouettes. In: IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2006) (2006)Google Scholar
  13. 13.
    Fournier, A., Gunawan, A.S., Romanzin, C.: Common illumination between real and computer generated scenes. In: Proceedings of Graphics Interface ’93, pp. 254–262. Toronto, ON, Canada (1993)Google Scholar
  14. 14.
    Genc, Y., Riedel, S., Souvannavong, F., Akinlar, C., Navab, N.: Marker-less tracking for ar: a learning-based approach. In: IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR 2002), pp. 295–304, Darmstadt, Germany (2002)Google Scholar
  15. 15.
    Gibson, S., Cook, J., Howard, T., Hubbold, R.: Rapid shadow generation in real-world lighting environments. In: Proceedings of the 14th Eurographics workshop on Rendering, pp. 219–229. Eurographics Association (2003)Google Scholar
  16. 16.
    Gibson, S., Murta, A.: Interactive rendering with real-world illumination. In: Proceedings of the Eurographics Workshop on Rendering Techniques 2000, pp. 365–376. Springer, Heidelberg (2000)Google Scholar
  17. 17.
    Guskov, I.: Efficient tracking of regular patterns on non-rigid geometry. In: Proceedings of ICPR, pp. 1057–1060 (2002)Google Scholar
  18. 18.
    Guskov, I., Klibanov, S., Bryant, B.: Trackable surfaces. In: ACM/EG Symposium on Computer Animation (2003)Google Scholar
  19. 19.
    Haller, M., Drab, S., Hartmann, W.: A real-time shadow approach for an augmented reality application using shadow volumes. In: ACM Symposium on Virtual Reality Software and Technology (VRST 2003), pp. 56–65 (2003)Google Scholar
  20. 20.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge ISBN: 0521623049 (2000)Google Scholar
  21. 21.
    Madsen, C.B., Sørensen, M.K.D., Vittrup, M.: The importance of shadows in augmented reality.In: Proceedings: 6th Annual International Workshop on Presence, Aalborg, Denmark (2003)Google Scholar
  22. 22.
    Okabe A., Boots B., Sugihara K. and Chiu S.N. (2000). Spatial Tessellations: Concepts and Applications of Voronoi Diagrams, 2nd Edn. Wiley, New York Google Scholar
  23. 23.
    Pilet, J., Lepetit, V., Fua, P.: Augmenting deformable objects in real-time. In: ISMAR ’05: Proceedings of the Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 134–137. IEEE Computer Society, Los Alamitos (2005)Google Scholar
  24. 24.
    Pilet, J., Lepetit, V., Fua, P.: Real-time non-rigid surface detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2005), pp. 822–828 (2005)Google Scholar
  25. 25.
    Pritchard D. and Heidrich W. (2003). Cloth motion capture. Computer Graphics Forum (Eurographics 2003) 22(3): 263–271 CrossRefGoogle Scholar
  26. 26.
    Scholz, V., Magnor, M.: Texture replacement of garments in monocular video sequences. In: Heidrich, W., Akenine-Moeller, T. (eds.) Rendering Techniques 2006, Eurographics Symposium on Rendering, pp. 305–312. Eurographics (EG), ACM SIGGRAPH, Eurographics, Nicosia, Cyprus (2006)Google Scholar
  27. 27.
    Scholz, V., Magnor, M.A.: Cloth motion from optical flow. In: Girod, B., Magnor, M., Seidel H.P. (eds.) Proceedings of Vision, Modeling and Visualization 2004, pp. 117–124 (2004)Google Scholar
  28. 28.
    Scholz V., Stich T., Keckeisen M., Wacker M. and Magnor M. (2005). Garment motion capture using color-coded patterns. Computer Graphics Forum (Proc. Eurographics EG’05) 24(3): 439–448 CrossRefGoogle Scholar
  29. 29.
    Simon, G., Fitzgibbon, A.W., Zisserman, A.: Markerless tracking using planar structures in the scene. In: Proceedings of International Symposium on Augmented Reality, pp. 120–128 (2000).
  30. 30.
    Stauder, J.: Augmented reality with automatic illumination control incorporating ellipsoidal models. IEEE Trans. Mult. 1(2), 136–143 (1999) Google Scholar
  31. 31.
    White, R., Forsyth, D.: Retexturing single views using texture and shading. In: European Conference on Computer Vision, vol. LNCS 3954, pp. 70–81. Springer, Heidelberg (2006)Google Scholar

Copyright information

© Springer-Verlag 2007

Authors and Affiliations

  1. 1.Department of Computer ScienceCarleton UniversityOttawaCanada
  2. 2.University of British ColumbiaVancouverCanada
  3. 3.Institute for Information TechnologyNational Research Council of CanadaOttawaCanada

Personalised recommendations