Automatic exposure selection and fusion for high-dynamic-range photography via smartphones

Abstract

High-dynamic-range (HDR) photography involves fusing a bracket of images taken at different exposure settings in order to compensate for the low dynamic range of digital cameras such as the ones used in smartphones. In this paper, a method for automatically selecting the exposure settings of such images is introduced based on the camera characteristic function. In addition, a new fusion method is introduced based on an optimization formulation and weighted averaging. Both of these methods are implemented on a smartphone platform as an HDR app to demonstrate the practicality of the introduced methods. Comparison results with several existing methods are presented indicating the effectiveness as well as the computational efficiency of the introduced solution.

Appendix: Optimization solution

This appendix provides the solution of the optimization problem with only one gradient term. The derivation for two terms is straightforward and not included here to save space. The optimization formulation for one gradient term is given by

$$\begin{aligned} \widehat{\mathbf{Y }}=\text {argmin}_\mathbf{Y }\big \lbrace \Vert \mathbf Y -\mathbf X \Vert _F^2+ \lambda \Vert \nabla \mathbf Y -\varvec{\Lambda } \Vert _F^2\big \rbrace \end{aligned}$$

(14)

where $\varvec{\Lambda }$ represents the gradient of Y and $\nabla$ indicates the gradient operator. In vector form, Equation (14) can be written as:

$$\begin{aligned} \widehat{\mathbf{y }}=\text {argmin}_\mathbf{y }\big \lbrace \Vert \mathbf y -\mathbf x \Vert ^2+ \lambda \Vert \mathbf C {} \mathbf y -\varvec{\delta } \Vert ^2\big \rbrace \end{aligned}$$

(15)

where y, x, and $\varvec{\delta }$ represent the column vector versions of Y, X, and $\varvec{\Lambda }$, respectively, and C denotes the block-circulant matrix representation of $\nabla$. By taking the derivative with respect to y, the following solution is obtained

$$\begin{aligned} \widehat{\mathbf{y }}= \big (\mathbf I +\lambda \mathbf C ^T\mathbf C \big )^{-1} \big (\mathbf x +\lambda \mathbf C ^T\varvec{\delta } \big ) \end{aligned}$$

(16)

where I denotes the identity matrix. Since C is a block-circulant matrix, it can be represented in diagonal form as:

$$\begin{aligned} \mathbf C =\mathbf W {} \mathbf E {} \mathbf W ^{-1} \end{aligned}$$

(17)

where E is the diagonal version of C and W is the DFT (discrete Fourier transform) operator. The diagonal values of E correspond to the DFT coefficients of $\nabla$ ($\nabla$ should be zero-padded properly before applying DFT). Hence, (16) can be rewritten as:

$$\begin{aligned} \widehat{\mathbf{y }}= \big (\mathbf I +\lambda \mathbf W {} \mathbf E ^{\dagger }{} \mathbf E {} \mathbf W ^{-1} \big )^{-1} \big (\mathbf x +\lambda \mathbf W {} \mathbf E ^{\dagger }{} \mathbf W ^{-1} \varvec{\delta } \big ) \end{aligned}$$

(18)

By multiplying both sides of (18) by $\mathbf W ^{-1}$, the following equation is resulted

$$\begin{aligned} \mathbf W ^{-1}\widehat{\mathbf{y }}= \big (\mathbf I +\lambda \mathbf E ^{\dagger }{} \mathbf E \big )^{-1} \big (\mathbf W ^{-1}{} \mathbf x +\lambda \mathbf E ^{\dagger }{} \mathbf W ^{-1} \varvec{\delta } \big ) \end{aligned}$$

(19)

Since $\mathbf W ^{-1}\widehat{\mathbf{y }}$, $\mathbf W ^{-1}{} \mathbf x$, and $\mathbf W ^{-1}\varvec{\delta }$ correspond to the DFT of $\widehat{\mathbf{y }}$, x, and $\varvec{\delta }$, respectively, (19) can be stated as follows:

$$\begin{aligned} \mathcal {F}\lbrace \widehat{\mathbf{Y }}_{u,v}\rbrace =\frac{ \mathcal {F}\lbrace \mathbf X \rbrace _{u,v}+ \lambda \mathcal {F}\lbrace \nabla \rbrace _{u,v}^{\dagger } \mathcal {F}\lbrace \varvec{\Lambda }\rbrace _{u,v} }{ 1+\lambda \Vert \mathcal {F}\lbrace \nabla \rbrace _{u,v} \Vert ^2 } \end{aligned}$$

(20)