Linear Time Illumination Invariant Stereo Matching

Abstract

This paper proposes a new similarity measure that is invariant to global and local affine illumination changes. Unlike existing methods, its computational complexity is very low. When used for stereo correspondence estimation, its computational complexity is linear in the number of image pixels and disparity searching range. It also outperforms the current state of the art similarity measures in terms of accuracy on the Middlebury benchmark (with radiometric differences).

Appendix

1.1 Appendix 1: Derivation of Eq. 12

The matching cost measured from two corresponding pixels p and $p'$ in two grayscale images $I_L$ and $I_R$ is:

$$\begin{aligned}&\sum _{q\in I_L}\mathcal {W}(p,q)\cdot (I_L(q)- a_p\cdot I_{R,\varDelta }(q)-b_p)^2\nonumber \\&\quad =\sum _{q\in I_L}\mathcal {W}(p,q)(a_p^2I_{R,\varDelta }(q)^2+2a_pb_pI_{R,\varDelta } (q)\nonumber \\&\qquad -2a_pI_L(q)I_{R,\varDelta }(q) -2b_pI_L(q)+b_p^2+I_L(q)^2)\nonumber \\&\quad =a_p^2\sum _{q\in I_L}\mathcal {W}(p,q)I_{R,\varDelta }(q)^2\!+\!2a_pb_p\sum _{q\in I_L}\mathcal {W}(p,q)I_{R,\varDelta }(q)\nonumber \\&\qquad -\,2a_p\sum _{q\in I_L}\mathcal {W}(p,q) I_L(q)I_{R,\varDelta }(q)\nonumber \\&\qquad -\,2b_{p}\sum _{q\in I_L}\mathcal {W}(p,q)I_L(q)+b_p^2+\sum _{q\in I_L}\mathcal {W}(p,q)I_L(q)^2\nonumber \\&\quad =a_p^2\cdot \mathcal {F}_{I_{R,\varDelta }\cdot I_{R,\varDelta }}(p)+2a_pb_p\cdot \mathcal {F}_{I_{R,\varDelta }}(p)\nonumber \\&\qquad -2a_p\cdot \mathcal {F}_{I_L\cdot I_{R,\varDelta }}(p)-\,2b_p\cdot \mathcal {F}_{I_{L}}(p)+b_p^2+\mathcal {F}_{I_L\cdot I_L}(p).\nonumber \\ \end{aligned}$$

(27)

1.2 Appendix 2: Derivation of Eq. 22

Similar to Eq. 4, we can extend Eq. 20 for color images as follows:

$$\begin{aligned} \tilde{A}\cdot \tilde{\mathcal {X}} =\tilde{B}. \end{aligned}$$

(28)

$\tilde{\mathcal {X}}$ is defined in Eq. 21, and

$$\begin{aligned} \tilde{A}=\begin{pmatrix} \cdots &{}\mathcal {W}(p,q)\cdot I_{R,\varDelta }^1(q)&{}\cdots \\ \cdots &{}\mathcal {W}(p,q)\cdot I_{R,\varDelta }^2(q)&{}\cdots \\ \cdots &{}\mathcal {W}(p,q)\cdot I_{R,\varDelta }^3(q)&{}\cdots \\ \cdots &{}\mathcal {W}(p,q)\cdot 1&{}\cdots \\ \end{pmatrix}^T, \end{aligned}$$

(29)

and

$$\begin{aligned} \tilde{B}=\begin{pmatrix} \cdots &{}\mathcal {W}(p,q)\cdot I_L^1(q)&{}\cdots \\ \cdots &{}\mathcal {W}(p,q)\cdot I_L^2(q)&{}\cdots \\ \cdots &{}\mathcal {W}(p,q)\cdot I_L^3(q)&{}\cdots \\ \end{pmatrix}^T. \end{aligned}$$

(30)

The linear system presented in Eq. 28 can be rewritten as:

$$\begin{aligned} \tilde{A}^T\tilde{A}\cdot \tilde{\mathcal {X}} =\tilde{A}^T\tilde{B}, \end{aligned}$$

(31)

where

$$\begin{aligned}&\tilde{A}^T\tilde{A}\nonumber \\&\quad =\begin{pmatrix} \mathcal {F}_{I_{R,\varDelta }^1\cdot I_{R,\varDelta }^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^1\cdot I_{R,\varDelta }^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^1\cdot I_{R,\varDelta }^3}(p)&{}\mathcal {F}_{I_{R,\varDelta }^1}(p)\\ \mathcal {F}_{I_{R,\varDelta }^2\cdot I_{R,\varDelta }^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2\cdot I_{R,\varDelta }^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2\cdot I_{R,\varDelta }^3}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2}(p)\\ \mathcal {F}_{I_{R,\varDelta }^3\cdot I_{R,\varDelta }^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3\cdot I_{R,\varDelta }^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3\cdot I_{R,\varDelta }^3}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3}(p)\\ \mathcal {F}_{I_{R,\varDelta }^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3}(p)&{}1\\ \end{pmatrix}\nonumber \\ \end{aligned}$$

(32)

and

$$\begin{aligned} \tilde{A}^T\tilde{B}= \begin{pmatrix} \mathcal {F}_{I_{R,\varDelta }^1\cdot I_L^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^1\cdot I_L^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^1\cdot I_L^3}(p)\\ \mathcal {F}_{I_{R,\varDelta }^2\cdot I_L^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2\cdot I_L^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^2\cdot I_L^3}(p)\\ \mathcal {F}_{I_{R,\varDelta }^3\cdot I_L^1}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3\cdot I_L^2}(p)&{}\mathcal {F}_{I_{R,\varDelta }^3\cdot I_L^3}(p)\\ \mathcal {F}_{I_L^1}(p)&{}\mathcal {F}_{I_L^2}(p)&{}\mathcal {F}_{I_L^3}(p)\\ \end{pmatrix}. \end{aligned}$$

(33)

1.3 Appendix 3: Derivation of Eq. 23

The matching cost for color images is:

$$\begin{aligned}&\sum _{{c1}=1}^3\sum _{q\in I_L}\mathcal {W}(p,q)\cdot (I_L^{c1}(q)-\sum _{{c2}=1}^3a_p^{{c1}{c2}}I_{R,\varDelta }^{c2}(q)-b_p^{c1})^2\nonumber \\&\quad =\sum _{{c1}=1}^3\sum _{q\in I_L}\mathcal {W}(p,q)\cdot ((I_L^{c1}(q))^2+(\sum _{{c2}=1}^3a_p^{{c1}{c2}}I_{R,\varDelta }^{c2}(q))^2\nonumber \\&\qquad +\,(b_p^{c1})^2-\,2b_p^{c1}I_L^{c1}(q)-2\sum _{{c2}=1}^3a_p^{{c1}{c2}}I_{R,\varDelta }^{c2}(q)I_L^{c1}(q)\nonumber \\&\qquad +\,2\sum _{{c2}=1}^3a_p^{{c1}{c2}}b_p^{c1}I_{R,\varDelta }^{c2}(q))\nonumber \\&\quad =\sum _{{c1}=1}^3(\sum _{q\in I_L}\mathcal {W}(p,q)\cdot I_L^{c1}(p)I_L^{c1}(p))\nonumber \\&\qquad +\sum _{{c3}=1}^3\sum _{{c2}=1}^3(\sum _{{c1}=1}^3 a_p^{c1c2}\cdot a_p^{c1c3})(\sum _{q\in I_L}\mathcal {W}(p,q)\nonumber \\&\qquad \cdot I_{R,\varDelta }^{c2}(p)I_{R,\varDelta }^{c3}(p))\nonumber \\&\qquad +\sum _{{c1}=1}^3(b_p^{c1})^2-2\sum _{{c1}=1}^3 b_p^{c1}(\sum _{q\in I_L}\mathcal {W}(p,q)\cdot I_L^{c1}(p))\nonumber \end{aligned}$$

$$\begin{aligned}&\qquad -2\sum _{{c1}=1}^3\sum _{{c2}=1}^3 a_p^{c1c2}(\sum _{q\in I_L}\mathcal {W}(p,q)\cdot I_L^{c1}(p)I_{R,\varDelta }^{c2}(p))\nonumber \\&\qquad +2\sum _{{c2}=1}^3\left( (\sum _{{c1}=1}^3a_p^{{c1}{c2}}\cdot b_p^{{c1}})\cdot (\sum _{q\in I_L}\mathcal {W}(p,q)\cdot I_{R,\varDelta }^{c2}(p))\right) \nonumber \\&\quad =\sum _{{c1}=1}^3\mathcal {F}_{I_L^{c1}\cdot I_L^{c1}}(p)\!+\!\sum _{{c3}=1}^3\sum _{{c2}=1}^3(\sum _{{c1}=1}^3 a_p^{c1c2}\cdot a_p^{c1c3}) \mathcal {F}_{I_{R,\varDelta }^{c2}\cdot I_{R,\varDelta }^{c3}}(p)\nonumber \\&\qquad +\,\sum _{{c1}=1}^3(b_p^{c1})^2 -2\sum _{{c1}=1}^3 b_p^{c1}\mathcal {F}_{I_L^{c1}}(p)\nonumber \\&\qquad -\,2\sum _{{c1}=1}^3\sum _{{c2}=1}^3 a_p^{c1c2}\mathcal {F}_{I_L^{c1}\cdot I_{R,\varDelta }^{c2}}(p)\nonumber \\&\qquad +\,2\sum _{{c2}=1}^3\left( (\sum _{{c1}=1}^3a_p^{{c1}{c2}}\cdot b_p^{{c1}})\cdot \mathcal {F}_{I_{R,\varDelta }^{c2}}(p)\right) \end{aligned}$$

(34)