Training data augmentation using generative models with statistical guarantees for materials informatics

Abstract

In materials $\mathcal {z}$ science, the small data size problem occurs frequently when applying machine learning algorithms. This issue is primarily due to, for example, real experimental measurement cost and time-consuming simulations based on materials models, e.g., density functional theory. We address the small data issue by generating training data using generative models. The proposed training data augmentation method can generate data using kernel density estimation models while maintaining statistical guarantees using unbiased estimators of linear regression, which has low computational costs and easy hyper-parameter tuning compared to deep neural networks. In addition, we derive an upper bound for the $L_{1}$ loss of the proposed method relative to probability density functions. Experiments were conducted with four benchmark and three materials (i.e., binary compounds, oxide ionic conductivity, and phosphorescent materials) datasets. Regarding the generated data, we examined training and generalization performances using kernel ridge regression compared to those of generative adversarial networks and real-NVPs. For the materials datasets, we analyzed influential factors regarding material properties (conductivity and emission intensity). The experimental results demonstrate that the proposed method can generate data that can be used as new training data.

Appendices

A Proof of unbiased estimator of ordinary least squares

Assume the true model is ${\varvec{Y}} = {\varvec{X}}\beta + \epsilon$, where $\epsilon \sim {\mathcal {N}}(0, \sigma ^{2}I)$, ${\varvec{X}} \in {\mathbb {R}}^{\mathrm{n} \times \mathrm{d}}$, ${\varvec{Y}} \in {\mathbb {R}}^{\mathrm{n}}$, $\beta \in {\mathbb {R}}^{\mathrm{d}}$, n denotes data size, and d denotes the number of inputs. Then, we obtain the following:

$$\begin{aligned} \begin{aligned} {\mathbb {E}}_{\epsilon } [{\hat{\beta }}]&= {\mathbb {E}}_{\epsilon } [({\varvec{X}}^{{\mathsf {T}}}{\varvec{X}})^{-1}{\varvec{X}}^{{\mathsf {T}}}{\varvec{Y}}]\\&= ({\varvec{X}}^{{\mathsf {T}}}{\varvec{X}})^{-1}{\varvec{X}}^{{\mathsf {T}}} {\mathbb {E}}_{\epsilon } [{\varvec{Y}}]\\&= ({\varvec{X}}^{{\mathsf {T}}}{\varvec{X}})^{-1}{\varvec{X}}^{{\mathsf {T}}} {\mathbb {E}}_{\epsilon } [{\varvec{X}}\beta + \epsilon ]\\&= ({\varvec{X}}^{{\mathsf {T}}}{\varvec{X}})^{-1}{\varvec{X}}^{{\mathsf {T}}}{\varvec{X}} \beta \; \; \;(\mathrm{since \;the\;assumption\;} {\mathbb {E}}_{\epsilon } [\epsilon ] = 0)\\&= \beta , \end{aligned} \end{aligned}$$

(6)

where superscript ${\mathsf {T}}$ denotes the transpose of the matrix.

B Proof of Equation (5)

The $L_{1}$ loss between ${\hat{p}}_{\mathrm{n}} (x)$ and p(x) is given as follows:

$$\begin{aligned} f(x_{1}, \ldots , x_{\mathrm{n}}) = \int _{-\infty }^{\infty } | {\hat{p}}_{\mathrm{n}}(x) - p(x) | \, dx. \end{aligned}$$

(7)

Moreover, when $x_{i} \ne x'_{i}$, for any pair of n-tuples $(x_{1}, \ldots , x_{\mathrm{n}})$ and $(x_{1}, \ldots , x'_{i}, \ldots , x_{\mathrm{n}})$, we can write

$$\begin{aligned}&| f(x_{1}, \ldots , x_{\mathrm{n}}) - f(x_{1}, \ldots , x'_{i}, \ldots , x_{\mathrm{n}}) |\nonumber \\&\quad = \left| \int _{-\infty }^{\infty } \left| \frac{1}{\mathrm{nh}} \sum _{j=1}^{\mathrm{n}} K\left( \frac{x-x_{j}}{\mathrm{h}}\right) - p(x) \right| \, dx \right. \nonumber \\&\qquad \left. - \int _{-\infty }^{\infty } \left| \frac{1}{\mathrm{nh}} \sum _{j=1}^{\mathrm{n}} K\left( \frac{x-x_{j}}{\mathrm{h}}\right) - p(x) \right| \, dx \right| \nonumber \\&\quad \le \left| \int _{-\infty }^{\infty } \left| \frac{1}{\mathrm{nh}} \sum _{j=1}^{\mathrm{n}} K\left( \frac{x-x_{j}}{\mathrm{h}}\right) - \frac{1}{\mathrm{nh}} \sum _{j=1}^{\mathrm{n}} K\left( \frac{x-x_{j}}{\mathrm{h}}\right) \right| \, dx \right| \nonumber \\&\quad = \left| \int _{-\infty }^{\infty } \left| \frac{1}{\mathrm{nh}} \left( K\left( \frac{x-x_{i}}{\mathrm{h}}\right) - K\left( \frac{x-x'_{i}}{\mathrm{h}}\right) \right) \right| \, dx \right| \nonumber \\&\quad = \frac{1}{\mathrm{nh}} \int _{-\infty }^{\infty } \left| K\left( \frac{x-x_{i}}{\mathrm{h}}\right) - K\left( \frac{x-x'_{i}}{\mathrm{h}}\right) \right| \, dx\nonumber \\&\quad \le \frac{1}{\mathrm{nh}} \left( \int _{-\infty }^{\infty } K\left( \frac{x-x_{i}}{\mathrm{h}}\right) \,dx\right. \nonumber \\&\qquad \left. + \int _{-\infty }^{\infty } K\left( \frac{x-x'_{i}}{\mathrm{h}}\right) \, dx \right) . \end{aligned}$$

(8)

Since $\int _{-\infty }^{\infty } K(x) \, dx = 1$, we obtain

$$\begin{aligned} \begin{aligned}&\left| f(x_{1}, \ldots , x_{\mathrm{n}}) - f(x_{1}, \ldots , x'_{i}, \ldots , x_{\mathrm{n}}) \right| \\&\le \frac{1}{\mathrm{nh}} \left( \int _{-\infty }^{\infty } K\left( \frac{x-x_{i}}{\mathrm{h}}\right) \,dx + \int _{-\infty }^{\infty } K\left( \frac{x-x'_{i}}{\mathrm{h}}\right) \, dx \right) \\&= \frac{1}{\mathrm{nh}}( 2\mathrm{h} ) = \frac{2}{\mathrm{n}}. \end{aligned} \end{aligned}$$

(9)

Using McDiarmid’s inequality, for all $\delta > 0$,

$$\begin{aligned} \begin{aligned} P\left[ f - {\mathbb {E}}[f] \ge \delta \right]&\le \exp {\left( -\frac{2 \delta ^{2}}{\sum _{i=1}^{\mathrm{n}}(\frac{2}{\mathrm{n}})^{2}} \right) }\\&= \exp {\left( -\frac{\mathrm{n} \delta ^{2}}{2} \right) }. \end{aligned} \end{aligned}$$

(10)

Here, since with probability $1 - \eta$, it holds $|| {\hat{p}} - p ||_{1} < {\mathbb {E}}_{x \sim p} [ || {\hat{p}} - p ||_{1} ] + \delta$, for all n,

$$\begin{aligned} \begin{aligned} || {\hat{p}}_{\mathrm{n}} - p ||_{1}&< {\mathbb {E}}_{x \sim p} [ || {\hat{p}}^{1}_{\mathrm{n}} - p ||_{1} ] + \delta \\&= {\mathbb {E}}_{x \sim p} [ || {\hat{p}}^{1}_{\mathrm{n}} - p ||_{1} ] + \frac{1}{\sqrt{\mathrm{n}}}\sqrt{ 2 \ln \frac{1}{\eta } }. \end{aligned} \end{aligned}$$

(11)

Thus, Eq. 5 follows from Eq. 11.

C Attributes of input variables for datasets

Table 12 Attributes of input variables for benchmark datasets

Table 13 Attributes of input variables for materials datasets (for LUMI, e.g., MgO indicates film thickness of MgO in nm)

D Implementation note

The software used in this study was implemented in Python3, PyTorch, and Scikit-learn on Linux machine. The KDE in Scikit-learn was used. The GAN and real-NVP were constructed by referring to the Web sites.⁴ In addition, the main part of the TDA algorithm (lines 8–28 in Algorithm 1) is described as follows:⁵