Alignment of 3D Models to Images Using Region-Based Mutual Information and Neighborhood Extended Gaussian Images

Abstract

Mutual information has been used for matching and registering 3D models to 2D images. However, in Viola’s original framework [1], surface albedo variance is assumed to be minimal when measuring similarity between 3D models and 2D image data using mutual information. In reality, most objects have textured surfaces with different albedo values across their surfaces, and direct application of this method in such circumstances will fail. To solve this problem, we propose to include spatial information into the original formulation by using histogram-based features of local regions that are robust to local but significant albedo variation. Neighborhood Extended Gaussian Images (NEGI) are used as descriptors to represent local surface regions on the 3D model, while pixel intensity data are considered within corresponding region windows on the image. Experiments on aligning 3D car models in cluttered scenes using this new framework demonstrate substantial improvement as compared to the original pixel-wise mutual information approach.