Live facial expression generation based on mixed reality

Abstract

Virtual reality technology provides a new methodology for visualization with realistic sensation, and has attracted special interests of human interface, visual communication communities. The key issue there is how to represent and reconstruct human naturally and realistically. Accordingly, recent study of facial expression has been received growing attention and intensively investigated. In this paper, We propose a hybrid approach to Live facial expression generation based on mixed reality. We first propose a novel approach to mixed reality, we call Augmented Virtuality, which enhances and augments the reality of complex and delicate live motions of the object in the virtual space, by projecting portions of live video images observing the deformation and motion of the object onto the surface of its static CG model. We also propose a new method of adapting color properties for smooth merging of real and virtual spaces, and also propose a new method of extraction of the region effective for merging, based on the optical flow analysis of both range and color images in which in the virtual space, shape and texture changes are observed. We apply this technique to real time generation of realistic eye expression. We then propose the homotopy sweep method for surface deformation using 3D control vectors, and apply this technique to the animation of mouth/lips expression. Our approach has the advantages of descripting the geometric shapes and the deformation of circular muscle simply, and of reconstructing realistic deformation efficiently. Experimental results demonstrate the effectiveness of the proposed hybrid approach in representive and visualizing live facial expression in real time.