Document search support by making physical documents transparent in projection-based mixed reality

Abstract

This paper presents Limpid Desk that supports document search on a physical desktop by making the upper layer of a document stack transparent in a projection-based mixed reality environment. A user can visually access a lower-layer document without physically removing the upper documents. This is accomplished by superimposition of cover textures of lower-layer documents on the upper documents by projected imagery. This paper introduces a method of generating projection images that make physical documents transparent. Furthermore, a touch sensing method based on thermal image processing is proposed for the system’s input interface. Areas touched by a user on physical documents can be detected without any user-worn or handheld devices. This interface allows a user to select a stack to be made transparent by a simple touch gesture. Three document search support techniques are realized using the system. User studies are conducted, and the results show the effectiveness of the proposed techniques.

Appendix: Radiometric compensation

In the real world, most objects have spatially varying reflectance properties that disturb the appearance of a projected image. We apply a radiometric compensation method (Yoshida et al. 2003), so as not to decrease the visibility of the appearance content of the projected thermal image. Note that the method assumes the spectral response of the visible camera is same as one of the human eye.

The method uses the visible camera of the system to obtain an affine correlation in color space between the projection and the reflectance captured by the camera for each camera pixel. This affine correlation can transform the desired color appearance on a physical objects surface into a projected color value. When the input RGB value for the projector is represented as (I _R , I _G , I _B ) and the captured RGB value of the visible camera is represented as (C _R , C _G , C _B ), the correlation between them can be represented by the following equation in the affine transformation.

$$ \left[ \begin{array}{ccc} I_R & I_G & I_B\\ \end{array}\right]^{t}= {\bf K} \left[ \begin{array}{cccc} C_R & C_G & C_B & 1\\ \end{array} \right]^t. $$

(7)

K is a 3 × 4 matrix that transforms a camera’s color space to that of a projector. Therefore, K is called a color mixing matrix that takes into account the projector’s spectral characteristics, the camera’s spectral sensitivity, and the spectral reflectance of the object’s surface. K has to be calibrated for each camera pixel. Once at least four correspondences between (C _R , C _G , C _B ) and (I _R , I _G , I _B ) are obtained, K is calculated by a least-squares method. In the calibration process, more than four simple color patterns (e.g., red, green, blue, yellow, magenta, and cyan) are projected, and the reflectance of each projected pattern is captured. After this color calibration, images of the desired color can be displayed on surfaces under consideration of their reflectance. The color of the compensated projection image (I _R , I _G , I _B ) is calculated by (7) for each pixel of the projection image where RGB values of the desired color are assigned to (C _R , C _G , C _B ). As described above, the color mixing matrix K can be calibrated without any prior information about the spectral characteristics of the projector, the camera, and the object’s surface.