Cross-Modal Predictive Coding for Talking Head Sequences

Abstract

Predictive coding of video has traditionally used information from previous video frames to help construct an estimate of the current frame. The difference between the original and estimated signal can then be transmitted to allow the receiver to fully reconstruct the original video frame. In this paper, we explore a new algorithm for use in coding the shape of a person’s lips in a head-and-shoulder video sequence. This algorithm uses the same predictive coding loop, but instead of forming an estimate of the lip image using motion compensation and previous video frames, it forms an estimate from the associated acoustic data. Since the acoustic data is also transmitted, the receiver is able to reconstruct the video with very little side information. In this paper, we will describe our predictive coding system and analyze methods for converting from the acoustic data to visual estimates.