An Image Based Approach to Hand Occlusions in Mixed Reality Environments

Abstract

The illusion of the co-existence of virtual objects in the physical world, which is the essence of MR paradigm, is typically made possible by superimposing virtual contents onto the surrounding environment captured through a camera. This works well until the order of the planes to be composited is coherent to their distance from the observer. But, whenever an object of the real world is expected to occlude the virtual contents, the illusion vanishes. What should be seen behind a real object could be visualized over it instead, generating a “cognitive dissonance” that may compromise scene comprehension and, ultimately, the interaction capabilities during the MR experience. This paper describes an approach to handle hand occlusions in MR/AR interaction contexts by means of an optimized stereo matching technique based on the belief propagation algorithm.