On the Well-Posedness of Uncalibrated Photometric Stereo Under General Lighting

Abstract

Uncalibrated photometric stereo aims at estimating the 3D-shape of a surface, given a set of images captured from the same viewing angle, but under unknown, varying illumination. While the theoretical foundations of this inverse problem under directional lighting are well-established, there is a lack of mathematical evidence for the uniqueness of a solution under general lighting. On the other hand, stable and accurate heuristical solutions of uncalibrated photometric stereo under such general lighting have recently been proposed. The quality of the results demonstrated therein tends to indicate that the problem may actually be well-posed, but this still has to be established. The present paper addresses this theoretical issue, considering first-order spherical harmonics approximation of general lighting. Two important theoretical results are established. First, the orthographic integrability constraint ensures uniqueness of a solution up to a global concave–convex ambiguity, which had already been conjectured, yet not proven. Second, the perspective integrability constraint makes the problem well-posed, which generalizes a previous result limited to directional lighting. Eventually, a closed-form expression for the unique least-squares solution of the problem under perspective projection is provided, allowing numerical simulations on synthetic data to empirically validate our findings.

Appendices

Appendix 1: Proof of Proposition 5.1

Proposition 5.1 characterizes the integrability of a normal field in terms of the coefficients $m_1$, $m_2$ and $m_3$ of $\mathbf {m}:=\rho \mathbf {n}$. The following proof of this proposition is largely inspired by [25].

Proof

According to Eqs. (5.15)–(5.17), integrability of the normal field under perspective projection can be written as:

$$\begin{aligned}&\hat{p}_{v} = \hat{q}_{u}\text { over }\Omega , \nonumber \\ \iff&\left( \frac{p}{f - u p - v q} \right) _v = \left( \frac{q}{f - u p - v q}\right) _u\text { over }\Omega , \nonumber \\ \iff&f p_{v} - v q p_{v} + v p q_{v} - f q_{u} + u p q_{u} - u q p_{u} = 0\text { over }\Omega , \nonumber \\ \iff&\begin{pmatrix} p_v \\ q_v \\ 0 \end{pmatrix}^\top \begin{pmatrix} 0 \\ -f \\ v \end{pmatrix} \times \begin{pmatrix} p \\ q \\ -1 \end{pmatrix} + \begin{pmatrix} p_u \\ q_u \\ 0 \end{pmatrix}^\top \begin{pmatrix} -f \\ 0 \\ u \end{pmatrix} \times \begin{pmatrix} p \\ q \\ -1 \end{pmatrix} = 0\text { over }\Omega , \end{aligned}$$

(5.75)

where $\times$ denotes the cross-product.

Besides, $-\dfrac{\mathbf {m}}{m_3} = \begin{pmatrix} p \\ q \\ -1 \end{pmatrix}$ according to (5.14). If we denote $\mathbf {w}_1 = \left[ 0,-f,v\right] ^\top$ and $\mathbf {w}_2 = \left[ -f,0,u\right] ^\top$, then (5.75) yields the following equation over $\Omega$:

$$\begin{aligned}&\left( \dfrac{-\mathbf {m}}{m_3}\right) _v^\top \mathbf {w}_1 \times \left( \dfrac{-\mathbf {m}}{m_3}\right) + \left( \dfrac{-\mathbf {m}}{m_3}\right) _u^\top \mathbf {w}_2 \times \left( \dfrac{-\mathbf {m}}{m_3}\right) = 0,\nonumber \\ \iff&\left( -\dfrac{m_3\mathbf {m}_v - m_{3v}\mathbf {m}}{{m_3}^2}\right) ^\top \mathbf {w}_1 \times \left( \dfrac{-\mathbf {m}}{m_3}\right) +\left( -\dfrac{m_3 \mathbf {m}_u - m_{3u}\mathbf {m}}{{m_3}^2}\right) ^\top \mathbf {w}_2 \times \left( \dfrac{-\mathbf {m}}{m_3}\right) = 0. \end{aligned}$$

(5.76)

Multiplying Eq. (5.76) by ${m_3}^3$:

$$\begin{aligned} (m_3\mathbf {m}_v - m_{3v}\mathbf {m})^\top \mathbf {w}_1 \times \mathbf {m} + (m_3\mathbf {m}_u - m_{3u}\mathbf {m})^\top \mathbf {w}_2 \times \mathbf {m} = 0 \quad \text { over }\Omega . \end{aligned}$$

(5.77)

In addition, $(\mathbf {w}_1 \times \mathbf {m}) \perp \mathbf {m}$ and $(\mathbf {w}_2 \times \mathbf {m}) \perp \mathbf {m}$, thus the following relationship holds over $\Omega$:

$$\begin{aligned}&m_3\mathbf {m}_v^\top (\mathbf {w}_1 \times \mathbf {m}) + m_3\mathbf {m}_u^\top (\mathbf {w}_2 \times \mathbf {m}) = 0, \nonumber \\ \iff&\mathbf {m}_v^\top (\mathbf {w}_1 \times \mathbf {m}) + \mathbf {m}_u^\top (\mathbf {w}_2 \times \mathbf {m}) = 0, \nonumber \\ \iff&u(m_{1u}m_{2} - m_{1}m_{2u} ) + v(m_{1v}m_{2} - m_{1}m_{2v}) \nonumber \\&+ f(m_{1v}m_{3} - m_{1}m_{3v}) - f(m_{2u}m_{3} - m_{2}m_{3u}) = 0. \end{aligned}$$

(5.78)

which concludes the proof.   $\square$

Appendix 2: Proof of Theorem 5.1

Theorem 5.1 characterizes scaled Lorentz transformations. Its proof relies on the following Propositions 5.2–5.4 from Lorentz’ group theory (proofs of these propositions can be found in [14]).

Proposition 5.2

For any proper and orthochronous Lorentz transformation $\mathbf {A} \in L^{p}_{o}$, there exists a unique couple $(\mathbf {v}, \mathbf {O}) \in B(\mathbf {0},1) \times SO(3,\mathbb {R})$ such that

$$\begin{aligned} \mathbf {A} = \mathbf {S}(\mathbf {v})\, \mathbf {R}(\mathbf {O}) = \begin{pmatrix} \gamma &{} \gamma \, \mathbf {v}^\top \mathbf {O}\\ \\ \gamma \, \mathbf {v} &{} (\mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top ) \mathbf {O} \\ \\ \end{pmatrix}, \end{aligned}$$

(5.79)

where $\gamma = \frac{1}{\sqrt{1-\left\Vert \mathbf {v}\right\Vert ^2}}$, and

$$\begin{aligned} \mathbf {S}(\mathbf {v}) = \begin{pmatrix} \gamma &{} \gamma \, \mathbf {v}^\top \\ \\ \gamma \, \mathbf {v} &{} \mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top \\ \\ \end{pmatrix}, \qquad&\mathbf {R}(\mathbf {O}) = \begin{pmatrix} 1 &{} 0 &{} 0 &{} 0\\ 0 \\ 0 &{} &{} \mathbf {O} \\ 0 \\ \end{pmatrix}. \end{aligned}$$

(5.80)

Proposition 5.3

The product of two proper/improper transformations is a proper one, and the product of a proper and improper transformation is an improper one. The same for the orthochronous property.

Proposition 5.4

Matrix $\mathbf {T} = \begin{pmatrix} -1 &{} 0 &{} 0 &{} 0\\ 0 &{} 1 &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 1 \end{pmatrix}$ is improper and non-orthochronous, and

matrix $\mathbf {P} = \begin{pmatrix} 1 &{} 0 &{} 0 &{} 0\\ 0 &{} -1 &{} 0 &{} 0 \\ 0 &{} 0 &{} -1 &{} 0 \\ 0 &{} 0 &{} 0 &{} -1 \end{pmatrix}$ is improper and orthochronous.

Using these already known results, we propose the following characterization of Lorentz transformations:

Proposition 5.5

For any Lorentz transformation $\mathbf {A} \in L$, there exists a unique couple $(\mathbf {v}, \mathbf {O}) \in B(\mathbf {0},1) \times SO(3,\mathbb {R})$ such that

$$\begin{aligned} \mathbf {A} = \begin{pmatrix} \epsilon _{1}(\mathbf {A})\,\gamma &{} \epsilon _{1}(\mathbf {A})\,\gamma \, \mathbf {v}^{\top }\mathbf {O}\\ \\ \epsilon _{2}(\mathbf {A})\,\gamma \, \mathbf {v} &{} \epsilon _{2}(\mathbf {A})(\mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^{\top }) \mathbf {O} \end{pmatrix}. \end{aligned}$$

(5.81)

Proof

We first assume that $\mathbf {A} \in L^{i}_{n}$.

According to Proposition 5.4, $\mathbf {T} \in L^{i}_{n}$. Thus using Proposition 5.3: $\mathbf {T}\mathbf {A} \in L^{p}_{o}$. Therefore, according to Proposition 5.2, there exists a unique couple $(\mathbf {v}, \mathbf {O}) \in B(\mathbf {0},1) \times SO(3,\mathbb {R})$ such that $\mathbf {T}\mathbf {A} = \mathbf {S}(\mathbf {v})\mathbf {R}(\mathbf {O})$. Since $\mathbf {T}\mathbf {T} = \mathbf {I_4}$, this implies that $\mathbf {A} = \mathbf {T}\mathbf {S}(\mathbf {v})\mathbf {R}(\mathbf {O})$. In addition, $\epsilon _1(\mathbf {A}) = -1$ and $\epsilon _2(\mathbf {A}) = 1$, hence:

$$\begin{aligned} \mathbf {A} = \begin{pmatrix} \epsilon _{1}(\mathbf {A})\gamma &{} \epsilon _{1}(\mathbf {A})\gamma \mathbf {v}^\top \mathbf {O}\\ \epsilon _{2}(\mathbf {A})\gamma \mathbf {v} &{} \epsilon _{2}(\mathbf {A})(\mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top ) \mathbf {O} \end{pmatrix}. \end{aligned}$$

(5.82)

With the same reasoning, we get the result for all the other transformations.   $\square$

Combining Proposition 5.5 and the definition (5.24) of scaled Lorentz transformations, we get Theorem 5.1.

Appendix 3: Some Useful Results on GBR and Lorentz Matrices, and Corollary 5.1

The aim of this section is to prove Corollary 5.1, which was used in the proofs of Theorems 5.2 and 5.3. Its proof relies on a few results on GBR and Lorentz matrices, which we provide in the following.

Let us denote by G the group of GBR transformations, and by $G_s$ that of scaled GBR transformations defined by

$$\begin{aligned} G_s = \lbrace s\mathbf {A} | s \in \mathbb {R}\backslash \{0\} \text { and } \mathbf {A} \in G \rbrace . \end{aligned}$$

(5.83)

Both are subgroups of $GL(3,\mathbb {R})$ under the matrix product. For all $\mathbf {B} = s\mathbf {A} \in G_s$, we call s the scale part of $\mathbf {B}$, and $\mathbf {A}$ its GBR part.

Let $\mathbf {C} \in \mathbb {R}^{n \times n}$ with $n > 1$ and $C_{ij}$ its entries. We will use the following notation for a minor of size two:

$$\begin{aligned} C^{i,j}_{k,l} = C_{ij}C_{kl} - C_{kj}C_{il}, \end{aligned}$$

(5.84)

where $1 \le i < k \le n$ and $1 \le j < l \le n$.

Such minors allow to characterize scaled GBR matrices:

Proposition 5.6

Let $\mathbf {A} \in \mathbb {R}^{3 \times 3}, A_{ij}$ the entries of $\mathbf {A}$. Then, $\mathbf {A}$ is a scaled GBR transformation iff $\mathbf {A}$ is invertible and fulfills the following equations:

$$\begin{aligned} \left\{ \begin{array}{l} A^{2,1}_{3,2} = A^{2,1}_{3,3} = A^{1,2}_{3,3} = A^{1,1}_{3,2} = 0,\\ A^{2,2}_{3,3} = A^{1,1}_{3,3}. \end{array} \right. \end{aligned}$$

(5.85)

Proof

See [4].   $\square$

Proposition 5.7

Let $\mathbf {v} \in B_{0}(1), \gamma = \frac{1}{\sqrt{1-\left\Vert \mathbf {v}\right\Vert ^2}}$, then $\mathbf {C} = \mathbf {I_3} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top$ is positive definite.

Proof

Let $\mathbf {B} = \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top$. We note $E_\lambda (\mathbf {B})$ the eigenspace associated to the eigenvalue $\lambda$ of $\mathbf {B}$. $\mathbf {B}$ is symmetric, thus according to the spectral theorem, all the eigenvalues of $\mathbf {B}$ are real, and $\mathbb {R}^3 = \bigoplus \limits _{i=1}^r E_{\lambda _i}(\mathbf {B})$ with $r \le 3$ the number of eigenvalues, and $\lbrace \lambda _i \rbrace _{i=1 \ldots r}$ the eigenvalues of $\mathbf {B}$. Hence: $\text {dim}(\mathbb {R}^3) = \sum \limits _{i=1}^r \text {dim}(E_{\lambda _i}(\mathbf {B}))$. According to the rank-nullity theorem, $\text {dim}(Ker(\mathbf {B})) + rank(\mathbf {B}) = 3$, and by definition $rank(\mathbf {B}) = 1$, thus $\text {dim}(Ker(\mathbf {B})) = \text {dim}(E_0(\mathbf {B})) = 2$. We deduce that there exists a unique nonzero eigenvalue $\lambda \in \mathbb {R}\backslash \{0\}$ such that $\mathbb {R}^3 = E_0(\mathbf {B}) \bigoplus E_\lambda (\mathbf {B})$ with $\text {dim}(E_\lambda (\mathbf {B})) = 1$.

Let $\Pi _{\mathbf {v}}$ the orthogonal projection onto $span \lbrace \mathbf {v} \rbrace$, and let $\mathbf {x} \in \mathbb {R}^3$ : $\Pi _{\mathbf {v}}(\mathbf {x}) = \frac{\mathbf {v}\mathbf {v}^\top }{\left\Vert \mathbf {v}\right\Vert ^2} \mathbf {x}$ and $\Pi _{\mathbf {v}}(\mathbf {v}) = \mathbf {v}$. We have: $\mathbf {B} = \frac{\gamma ^{2}}{1 + \gamma }\left\Vert \mathbf {v}\right\Vert ^2\Pi _{\mathbf {v}}$ and $\mathbf {B}\mathbf {v} = (\frac{\gamma ^{2}}{1 + \gamma }\left\Vert \mathbf {v}\right\Vert ^2)\mathbf {v}$. Thus, $\frac{\gamma ^{2}}{1 + \gamma }\left\Vert \mathbf {v}\right\Vert ^2$ is an eigenvalue of $\mathbf {B}$ and $\lambda = \frac{\gamma ^{2}}{1 + \gamma }\left\Vert \mathbf {v}\right\Vert ^2$ . Besides, $\frac{\lambda }{\gamma -1} = \frac{\gamma ^{2}}{(\gamma - 1)(\gamma + 1)}\left\Vert \mathbf {v}\right\Vert ^2 = \frac{\gamma ^{2}}{\gamma ^2 - 1}\left\Vert \mathbf {v}\right\Vert ^2 = \frac{1}{1 - \frac{1}{\gamma ^2}}\left\Vert \mathbf {v}\right\Vert ^2 = \frac{1}{1-(1-\left\Vert \mathbf {v}\right\Vert ^2)}\left\Vert \mathbf {v}\right\Vert ^2=1$. Therefore, $\lambda = \gamma - 1$ and the eigenvalues of $\mathbf {B}$ are 0 and $(\gamma - 1)$. Let $\alpha \in \lbrace 0,\gamma - 1 \rbrace$ and $\mathbf {u} \in E_\alpha (\mathbf {B})$. We have:

$$\begin{aligned} \mathbf {B}\mathbf {u}= \alpha \mathbf {u} \iff \mathbf {u} + \mathbf {B}\mathbf {u} = \mathbf {u} + \alpha \mathbf {u} \iff \mathbf {C} \mathbf {u} = (\alpha + 1)\mathbf {u}. \end{aligned}$$

(5.86)

Thus, $1>0$ and $\gamma > 0$ are the eigenvalues of $\mathbf {C}$ with $E_1(\mathbf {C}) = E_0(\mathbf {B})$ and $E_\gamma (\mathbf {C}) = E_{\gamma -1}(\mathbf {B})$. Consequently, $\mathbf {C}$ is positive definite.   $\square$

Proposition 5.8

Let $\mathbf {A}_s \in L_s$ a scaled Lorentz transformation. The submatrix $\mathbf {B}$ formed by the last 3 rows and 3 columns of $\mathbf {A}_s$ is invertible.

Proof

By definition of $\mathbf {A}_s$, there exists a unique couple $(s,{\tilde{\mathbf {A}}}) \in \mathbb {R}\backslash \{0\} \times L$ such that $\mathbf {A}_s = s{\tilde{\mathbf {A}}}$. Hence, from Proposition 5.5, there exists a unique couple $(\mathbf {v}, \mathbf {O}) \in B(\mathbf {0},1) \times SO(3,\mathbb {R})$ such that $\mathbf {B} = s \epsilon _{2}({\tilde{\mathbf {A}}})(\mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top ) \mathbf {O}$. Since $\mathbf {O} \in SO(3,\mathbb {R})$, $\text {det}(\mathbf {O}) = 1$. In addition, Proposition 5.7 implies $\text {det}(\mathbf {I_{3}} + \frac{\gamma ^{2}}{1 + \gamma } \mathbf {v} \mathbf {v}^\top ) {>} 0$, thus $\text {det}(\mathbf {B}) \ne 0$.   $\square$

Corollary 5.1

Let $\mathbf {A}_s \in L_s$ a scaled Lorentz transformation. If its entries $A_{ij}$ fulfill

$$\begin{aligned} \left\{ \begin{array}{l} A^{3,2}_{4,3} = A^{3,2}_{4,4} = A^{2,3}_{4,4} = A^{2,2}_{4,3} = 0,\\ A^{3,3}_{4,4} = A^{2,2}_{4,4}, \end{array} \right. \end{aligned}$$

(5.87)

then the submatrix $\mathbf {B}$ of $\mathbf {A}_s$ formed by the last 3 rows and 3 columns is a scaled GBR, i.e., there exists a unique quadruple $(\lambda ,\mu ,\nu ,\beta ) \in \mathbb {R}^4$ with $\lambda \ne 0, \beta \ne 0$ such that

$$\begin{aligned} \mathbf {A}_s=\begin{pmatrix} A_{11} &{} A_{12} &{} A_{13} &{} A_{14} \\ A_{21} &{} \beta \lambda &{} 0 &{} -\beta \mu \\ A_{31} &{} 0 &{} \beta \lambda &{} -\beta \nu \\ A_{41} &{} 0 &{} 0 &{} \beta \\ \end{pmatrix}. \end{aligned}$$

(5.88)

Proof

According to Proposition 5.8, $\mathbf {B}$ is invertible. Besides, $\mathbf {A}_s$ fulfill Eqs. (5.87) iff $\mathbf {B}$ fulfill Eqs. (5.85), thus according to Proposition 5.6, $\mathbf {B} \in G_s$.   $\square$