QRnet: fast learning-based QR code image embedding

Abstract

Quick Response (QR) codes usage in e-commerce is on the rise due to their versatility and ability to connect offline and online content, taking over almost every aspect of a business from posters to payments. Thus, many efforts have aimed at improving the visual quality of QR codes to be easily included in publicity designs in billboards and magazines. The most successful approaches, however, are slow since optimization algorithms are required for the generation of each beautified QR code, hindering its online customization. The aim of this paper is the fast generation of visually pleasant and robust QR codes. The proposed framework leverages state-of-the-art deep-learning algorithms to embed a color image into a baseline QR code in seconds while keeping a maximum probability of error during the decoding procedure. Halftoning techniques that exploit the human visual system (HVS) are used to smooth the embedding of the QR code structure in the final QR code image while reinforcing the decoding robustness. Compared to optimization-based methods, our framework provides similar qualitative results but is 3 orders of magnitude faster.

Appendices

Appendix A: Encoding procedure

QR codes have a well-defined structure for correcting geometric transformations and achieving quick machine detection and decoding. Some encoders analyze the input message to find the most suitable encoding mode (numeric, alphanumeric, byte or Kanji), version and, error correction capability, while in others, these are chosen by the user. Although the method used to convert the input text into bits varies according to the encoding mode, a general procedure is followed to complete the bit stream s_o. That is, the encoded message is always located in the first m-bits of the bit stream which is padded with zeros to be later broken up into 8-bit codewords.

Due to QR codes structure, the QR code standard requires to add specific padding bits p after the message to completely fill the total capacity of the QR code. Finally, u error correction bits are generated for the message m and padding bits p, generating an n-bits Reed Solomon (RS) code. This bit stream is distributed through the QR code, starting from the bottom right as it is shown in Fig. 12.

Fig. 12

QR Code Encoding. Each part of the bit stream is shown and highlighted into the QR code

Appendix B: Decoding procedure

As illustrated in Fig. 13, after an image of a QR code is captured by the decoder, it is binarized to a black and white image. The QR code standard [24] does not define a specific binarization method, thus, there is not an unique manner to perform this step. However, many widely used QR code decoders apply local thresholding strategies to overcome the effects of uneven lighting conditions. For instance, the widely used open source ZXing [19] library for QR code generation and reading computes local thresholds for non-overlapping blocks B_m,n as the mean value of the image intensity in a local window W as

$$ t_{m,n} = \frac{1}{N} \sum\limits_{(i,j)\in \mathbf{W}} Y_{i,j}, $$

(5)

where Y_i,j is the image luminance value at a pixel location (i,j), t_m,n is the local threshold for the block B_m,n, and N is the number of pixels in the local window W with size of W_a × W_a pixels. This binarization method has been used before to design a probability of error model on beautified QR codes [8, 30] and to evaluate their robustness [31].

The decoder then searches for the finder patterns and the alignment pattern regions, which possess a black-white-black-white-black pattern with a ratio of 1:1:3:1:1, and black-white-black pattern with ratio 1:1:1, respectively. Based on the three finder patterns and the closest-to-corner alignment pattern, a perspective transform is performed to compensate for geometric distortions from the scanning process. Once the detected barcode region is converted into a square region, a sampling grid is estimated, using as reference the detected coordinates of the finder and alignment pattern, as well as, the timing patterns. As shown in Fig. 13, the grid samples the central pixel from each binarized module and demodulate these data as ‘1’ or ‘0’ for white and black, respectively. After the error correction algorithm is applied, the message is extracted from the QR code.

Fig. 13

QR Code Decoding Process. Luminance Y from the scanned image is first determined and then binarize for the detection of the QR code using the finder patterns. Next, only the center of each module is sample to decode the message

Appendix C: Probability of error models

The luminance modification in the original binary structure of the QR code distorts the binarization threshold, increasing the probability of error during the decoding process. To consider this error, a normalized version of the probability of error model proposed in [8] is used. In this model, two independent probability models are developed based on the sampling accuracy of the central pixels of the QR code modules. Then, these two models are combined through a probability of sampling error model.

1.1 C.1 Probability of binarization error model

This first model considers the probability that any given pixel inside a QR code module is binarized in the wrong direction during the decoding procedure. Then, the probability of error at the detected binary pixel (i,j) is given by

$$ \begin{array}{@{}rcl@{}} P_{Berr} &=& P(\hat{Q}_{i,j}=1|Q_{i,j}=0) p_{0} \\ && + P(\hat{Q}_{i,j}=0|Q_{i,j}=1) p_{1}, \end{array} $$

(6)

where p₀ = P(Q_i,j = 0) and p₁ = P(Q_i,j = 1) are the probabilities of having a black or a white pixel in the QR code Q, respectively. As mentioned before, the calculation of the threshold t_m,n depends on the decoder. In this case, a decoder built on ZXing [19] library open source code is considered. Consequently, the threshold depends on the local distribution of luminance values in the image, the distribution of pixels in the QR code, and the values of the parameters for the luminance transformation. Since $\hat {Q}_{i,j}$ represents the binary decision made by the decoder depending on the threshold, the conditional probabilities above can be written as

$$ \begin{array}{@{}rcl@{}} P(\hat{Q}_{i,j} = 1|Q_{i,j}=0) &=& P({Y}_{i,j}^{out}>t_{m,n}|Q_{i,j}=0),\\ P(\hat{Q}_{i,j} = 0|Q_{i,j}=1) &=& P(Y_{i,j}^{out}<t_{m,n}|Q_{i,j}=1). \end{array} $$

(7)

The exact solution to (7) requires the knowledge of the joint distribution of the image and threshold values. The problem can be simplified assuming that the components of the embedding luminance are independent and can be represented in function of the luminance transformation parameters as [8]

$$ {Y}_{i,j}^{out} = [\beta Q_{i,j}+\alpha (1-Q_{i,j})] I_{p_{c}(i,j)}+Y_{i,j} (1-I_{p_{c}(i,j)}). $$

(8)

Using (8) and (5) together with certain reasonable assumptions stated in [8], a model for the probability distribution of the threshold is given by

$$ f_{t}(x) = \sum\limits_{k=0}^{n} f (x-t_{k}) \left( \begin{array}{c}n\\ k \end{array}\right) {{p}_{1}^{k}} p_{0}^{n-k}, $$

(9)

where n = ⌊p_cN⌋ is the total number of pixels selected for modification in the window W, $t_{k}=\lfloor \frac {k\beta }{N}+\frac {(n-k)}{N}\alpha \rfloor$ are all the possible average contributions of n modified pixels to the total threshold, and f is the convolution of f_Y(x) and f_η(x). f_Y(x) represents the probability distribution of the unmodified pixels in the image, modeled as independent Gaussian random variables that depend on the window W, and f_η(x) accounts for the noise at the detector, modeled also as a Gaussian distribution. Thus, conditioning on the set of modified pixels, the conditional probabilities in (7) can be determined as follows

$$ \begin{array}{@{}rcl@{}} P(Y_{i,j}^{out}>t_{m,n}|Q_{i,j}=0) &=& P(\alpha>t_{m,n}) p_{c} \\ && + P(Y_{i,j}>t_{m,n}) (1- p_{c}), \end{array} $$

(10)

$$ \begin{array}{@{}rcl@{}} P(Y_{i,j}^{out}<t_{m,n}|Q_{i,j}=1) &=& P(\beta<t_{m,n}) p_{c} \\ && + P(Y_{i,j}<t_{m,n})(1- p_{c}), \end{array} $$

(11)

where

$$ \begin{array}{@{}rcl@{}} P(x>t_{m,n}) &=& \frac{{{\int}_{\!\!0}^{x}} f_{t}(t)dt}{{{\int}_{\!\!0}^{1}} f_{t}(t)dt}, \end{array} $$

(12)

$$ \begin{array}{@{}rcl@{}} P(x<t_{m,n}) &=& \frac{{{\int}_{\!\!x}^{1}} f_{t}(t)dt}{{{\int}_{\!\!0}^{1}} f_{t}(t)dt}. \end{array} $$

(13)

Substituting (10) and (11) in (6), we finally estimate the probability of binarization error P_Berr.

1.2 C.2 Probability of detection error

As in [8], assuming the central pixels in the QR code modules are precisely sampled, this model considers the probability of a given central pixel to be detected in the wrong direction during the decoding procedure. The probability of detection error per each QR code module is defined as

$$ P_{Derr} = P(\alpha_{c}>t)p_{0}+P(\beta_{c}<t)p_{1}. $$

(14)

In this case, the binarization threshold at the decoder can be decomposed as t = μ + b + η, considering not only the average contribution to the threshold from the modified pixels μ computed as

$$ \mu = \frac{\alpha n_{\alpha}+\beta n_{\beta}+\alpha_{c} n_{\alpha_{c}}+\beta_{c} n_{\beta_{c}}}{N}, $$

(15)

but also from the non-modified pixels b in the window, as well as the noise at the decoder η. In (15), $n_{\alpha }, n_{\beta }, n_{\alpha _{c}}$ y $n_{\beta _{c}}$ are the number of pixels per each modified luminance level and N is the total number of pixels in the window W.

Non-modified pixels are modeled as Gaussian random variables as before, with mean μ_b = E[Y_i,j](1 − p_c) and variance ${\sigma _{b}^{2}}=\frac {Var[Y_{i,j}]}{N} (1- p_{c})$. As a result, the probability distribution of the threshold t is Gaussian with mean μ_t = μ + μ_b and variance ${\sigma _{t}^{2}}={\sigma _{b}^{2}}+\sigma _{\eta }^{2}$ where $\sigma _{\eta }^{2}$ is the variance of the Gaussian noise at the decoder. Substituting this probability distribution of the threshold in the normalized probabilities defined in (12), we can estimate the probability of detection error P_Derr in (14).

1.3 C.3 Global probability of error

A probability of error sampling model is considered to combine the two independent stages in a global probability of error [8]. According to the decoding procedure, ideally, central pixels are more likely to be sampled. Thus, it is reasonable to model the sampling process by means of a Gaussian distribution around the center regions with σ = W_a/6. The probability of sampling out of the center region p_s can be determined by integrating outside the central region d_a × d_a. In this paper, this probability is also normalized. Once the probability of sampling error p_s is known, the global probability of error is determined by

$$ P_{err} = P_{Berr} p_{s}+P_{Derr}(1-p_{s}). $$

(16)