Single Image Based Three-Dimensional Scene Reconstruction Using Semantic and Geometric Priors

Abstract

Single image based three-dimensional (3D) scene reconstruction has become an important research topic for computer vision and computer graphics fields to provide machine vision systems with near human visual perception. Previous approaches for 3D scene reconstruction and depth estimation from single images required many factors including motion parallax, stereoscopic parallels, and various monocular depth cues adopted from known geometric priors. Deep learning based depth estimation techniques have advanced single image based depth estimation by aggregating various complexity information from RGB depth image datasets for training images to drive the process. This paper proposes an effective 3D scene estimation methodology by automatically extracting vanishing point and semantic information including 3D geometric characteristics without prior assumptions. The vanishing point is extracted from line segments and minimum spanning tree clustering to remove spurious noisy edges. Retracting geometric and semantic information from a given image is achieved by a generative adversarial network trained on the created training set. We verified the proposed approach’s efficiency and effectiveness experimentally with a large database created by directly recovering 3D scenes from from an input image.

Appendix: Geometric Quantities and Image Transformations

In this appendix, we obtain various geometric quantities including a vanishing point representation and several transformations related to the camera settings. As seen in Fig. 4, we take a 2D image from an optical center $C_h(0,c_h, 0)$ with angle $\theta$ focusing at $F_c(0,b,c)$ in a 3D space. Then we correspond the focusing point $F_c$ to the origin $O_I$ of the image plane $I_{2D}$ with orthogonal coordinate system. We can see that $\overrightarrow{C_hF_c}=(0,b-c_h,c)$ so that the normal vector $\mathbf {n}$ to the image plane $I_{2D}$ lying in the 3D space is given as

$$\begin{aligned} \mathbf {n}=(n_1,n_2,n_3)=\frac{\overrightarrow{C_hF_c}}{\Vert \overrightarrow{C_hF_c}\Vert } =\frac{(0,b-c_h,c)}{\sqrt{(b-c_h)^2+c^2}}. \end{aligned}$$

(4)

1.1 Gemmetric Quantities and Vanishing Point Representation

Now, we calculate 3D coordinates of several points related to the 2D image obtained from the projection. Since the vanishing point $(0, v_h)$, we can calculate the camera focusing angle $\theta$ using the focal and focusing points as

$$\begin{aligned} \theta =\arctan \frac{v_h}{\sqrt{(b-c_h)^2+c^2}}=\arctan \frac{c_h-b}{c}. \end{aligned}$$

(5)

From the projection given in Fig. 8, we obtain the angle relation

$$\begin{aligned} \sin \theta =\frac{c_h-b}{\sqrt{(c_h-b)^2+c^2}} \end{aligned}$$

(6)

and

$$\begin{aligned} \cos \theta =\frac{c}{\sqrt{(c_h-b)^2+c^2}}=\frac{\sqrt{(c_h-b)^2+c^2}}{v_3}. \end{aligned}$$

(7)

Fig. 8

Vanishing point estimation in three-dimensional (3D) space from Eqs. 4 and 6

Applying the relations to the trigonometric values, we can obtain the z coordinate of the vanishing point $v=(0,c_h, v_3)$ in 3D space as

$$\begin{aligned} v_3=\frac{(c_h-b)^2+c^2}{c} =c+\frac{v_h(c_h-b)}{\sqrt{(c_h-b)^2+c^2}}. \end{aligned}$$

(8)

1.2 3D Representation of Points in the Two-Dimensional Image Plane $I_{2D}$

In Sect. 2.4, we proposed a depth estimation from the 2D image $I_{2D}$ using the vanishing point. For the depth estimation, it is convenient to have a transform of the 2D image onto the plane containing the image plane in 3D. The plane $I_{3D}$ containing the image is given as $I_{3D}=\{\mathbf{x}\in \mathbb {R}^3: \mathbf {n}\cdot (\mathbf{x}-F_c)=n_1 x +n_2(y-b)+n_3(z-c)=0\}$ with normal vector $\mathbf {n}$ in (1). Even though we can find a clue for the transform in the proof given for the projection into the 2D image plane, we are to give a concrete form to the transform.

Fig. 9

Mapping the two-dimensional (2D) image plane onto three dimensional (3D) plane

Let Q(u, v) be a point in $I_{2D}.$ From the orthogonal coordinate system given to $I_{2D},$ this means $\overrightarrow{O_IQ}=u\mathbf {e}_1+ v\mathbf {e}_2.$

In 3D, the point Q is represented in vector sum from Fig. 9 as

$$\begin{aligned} \overrightarrow{OQ}&=\overrightarrow{OF_c}+\overrightarrow{F_cQ}\nonumber \\&=b\mathbf {e}_y+c\mathbf {e}_z+u\mathbf {e}_1+ v\mathbf {e}_2 \end{aligned}$$

(9)

On the other hand, we have found the relations between $\{\mathbf {e}_1, \mathbf {e}_2\}$ and $\{\mathbf {e}_x, \mathbf {e}_y, \mathbf {e}_z\}$ in (11) as $\mathbf {e}_1=\mathbf {e}_x\quad \text{ and }\quad \mathbf {e}_2=\cos \theta \mathbf {e}_y+\sin \theta \mathbf {e}_z.$ Applying these relations to (9) and using the fact $x=\overrightarrow{OQ}\cdot \mathbf {e}_x,\quad y=\overrightarrow{OQ}\cdot \mathbf {e}_y, \quad z=\overrightarrow{OQ}\cdot \mathbf {e}_z,$ we find the 3D coordinate (x, y, z) for the point Q as

$$\begin{aligned} (x,y,x)&=(0,b,c)+(u,0,0)+(0,v\cos \theta ,v\sin \theta )\\ {}&=(u, b+v\cos \theta ,c+v\sin \theta ) \end{aligned}$$

Here $\cos \theta$ and $\sin \theta$ are given in (7) and (6), respectively. Thus, the transform $T_{I_{2D}\rightarrow I_{3D}}: I_{2D}\rightarrow I_{3D}$ is found to be

$$\begin{aligned} \left( \begin{array}{c} x\\ y\\ z\end{array}\right)&=T_{I_{2D}\rightarrow I_{3D}}\left( \begin{array}{c} u\\ v\end{array}\right) =\left( \begin{array}{c} u\\ b+v\cos \theta \\ c+v\sin \theta \end{array}\right) \\ {}&=\left( \begin{array}{cc} 1 &{} 0\\ 0 &{}\cos \theta \\ 0&{} \sin \theta \end{array}\right) \left( \begin{array}{c} u\\ v\end{array}\right) +\left( \begin{array}{c} 0\\ b\\ c\end{array}\right) . \end{aligned}$$

Conversely, we assume to be given the 3D coordinate (x, y, z) for the point Q. From Fig. 9, the vector sum in (9) gives the relation

$$\begin{aligned} \overrightarrow{O_IQ}&=u\mathbf {e}_1+v\mathbf {e}_2\\ {}&=\overrightarrow{OQ}-\overrightarrow{OF_c}\\&=(x,y,z)-(0,b,c)=(x, y-b,z-c)\\&=x\mathbf {e}_x+(y-b)\mathbf {e}_y+(z-c)\mathbf {e}_z. \end{aligned}$$

Applying the relations (10) and the orthogonality gives the inverse transform $T_{I_{3D}\rightarrow I_{2D}}: I_{3D}\rightarrow I_{2D}$ as

$$\begin{aligned} \left( \begin{array}{c} u\\ v\end{array}\right)&=T_{I_{3D}\rightarrow I_{2D}}\left( \begin{array}{c} x\\ y\\ z\end{array}\right) \\&=\left( \begin{array}{c} x\\ (y-b)\cos \theta +(z-c)\sin \theta \end{array}\right) \\&=\left( \begin{array}{ccc} 1 &{} 0 &{} 0\\ 0 &{}\cos \theta &{}\sin \theta \end{array}\right) \left( \begin{array}{c} x\\ y\\ z\end{array}\right) +\left( \begin{array}{c} 0\\ -b\cos \theta -c\sin \theta \end{array}\right) . \end{aligned}$$

We note that it is not difficult to show that

$$\begin{aligned} T_{I_{3D}\rightarrow I_{2D}}\circ T_{I_{2D}\rightarrow I_{3D}}=id_{I_{2D}} \end{aligned}$$

and

$$\begin{aligned} T_{I_{2D}\rightarrow I_{3D}}\circ T_{I_{3D}\rightarrow I_{2D}}=id_{I_{3D}} \end{aligned}$$

by using the fact that $(x, y-b, z-c)\cdot \mathbf {n}=0$ for (x, y, z) in $I_{3D}$ with the normal vector $\mathbf {n}$ given in (4).

1.3 Projection Onto a Two-Dimensional Image $I_{2D}$

The 3D plane $I_{3D}$ corresponding to the image plane $I_{2D}$ is the set of all points $\mathbf{x}=(x,y,z)$ such that $\mathbf {n}\cdot (\mathbf{x}-F_c)=n_1 x +n_2(y-b)+n_3(z-c)=0.$ In this last section, we find the projection of 3D points in $\mathbb {R}^3$ into the image plane $I_{2D}.$ Let P(x, y, z) be a 3D point in $\{\mathbf{x}=(x,y,z): \mathbf {n}\cdot (\mathbf{x}-C_h)\ne 0\}$, then we are to find the coordinate (u, v) of the corresponding point Q in image plane $I_{2D}$. We plan to find u and v using the relation

$$\begin{aligned} u=\overrightarrow{O_IQ}\cdot \mathbf {e}_1=\overrightarrow{F_cQ}\cdot \mathbf {e}_1, ~v=\overrightarrow{O_IQ}\cdot \mathbf {e}_2=\overrightarrow{F_cQ}\cdot \mathbf {e}_2. \end{aligned}$$

(10)

To do this, we need to represent the two unit vectors $\mathbf {e}_1$ and $\mathbf {e}_2$ in terms of $\mathbf {e}_x, \mathbf {e}_y, \mathbf {e}_y.$ Using the angle $\theta ,$ we see that

$$\begin{aligned} \mathbf {e}_1=\mathbf {e}_x \quad \text{ and } \quad \mathbf {e}_2=\cos \theta \mathbf {e}_y+\sin \theta \mathbf {e}_z. \end{aligned}$$

(11)

On the other hand, the two vectors $\overrightarrow{C_hQ}$ and $\overrightarrow{C_hP}$ are parallel so that there exists a constant $\alpha$ such that $\overrightarrow{C_hQ}=\alpha \overrightarrow{C_hP}.$ And $\overrightarrow{OQ}$ lies in the plane so that it satisfies $\mathbf {n}\cdot (\overrightarrow{OQ}-\overrightarrow{OF_c})=0.$ And we have $\overrightarrow{OQ}=\overrightarrow{OC_h}+\alpha \overrightarrow{C_hP}$ and $\overrightarrow{C_hP}=\overrightarrow{OP}-\overrightarrow{OC_h}=(x,y-c_h,z).$ Combining these relations, we get

$$\begin{aligned} \alpha&=\frac{(\overrightarrow{OF_c}-\overrightarrow{OC_h})\cdot \mathbf {n}}{(\overrightarrow{OP}-\overrightarrow{OC_h})\cdot \mathbf {n}}=\frac{(0,b-c_h,c)\cdot (n_1,n_2,n_3)}{(x,y-c_h,z)\cdot (n_1,n_2,n_3)}\nonumber \\&=\frac{(b-c_h)^2+c^2}{(b-c_h)(y-c_h)+cz}. \end{aligned}$$

(12)

Also, $\overrightarrow{O_IQ}$ is represented in vector sum as $\overrightarrow{F_cQ}=\overrightarrow{C_hQ}+\overrightarrow{F_cC_h}=\alpha (x,y-c_h,z)+(0, c_h-b,-c) =(\alpha x, \alpha (y-c_h)+c_h-b, \alpha z-c),$ Applying this vector sum and (11) to (10), we obtain $u=\alpha x$ and $v=(\alpha (y-c_h)+c_h-b)\cos \theta +(\alpha z-c)\sin \theta .$ Finally, we calculate the projection mapping $T_{{3D}\rightarrow I_{2D}}: \mathbb {R}^3\rightarrow I_{2D}$ as

$$\begin{aligned} \left( \begin{array}{c} u\\ v\end{array}\right)&=T_{3D \rightarrow I_{2D}}\left( \begin{array}{c} x\\ y\\ z\end{array}\right) \\&=\left( \begin{array}{c} \alpha x\\ (\alpha (y-c_h)+c_h-b)\cos \theta +(\alpha z-c)\sin \theta \end{array}\right) \end{aligned}$$

where the parameters $\alpha$, $\sin \theta ,$ and $\cos \theta$ are given in (12), (6) and (7), respectively.