Sequential support points

Abstract

By minimizing the energy distance, the support points (SP) method can efficiently compact big training sample into a representative point set with small size. However, when the training sample is deficient, the quality of SP will be greatly reduced. In this paper, a sequential version of SP, called sequential support point (SSP), is proposed. The new method has two appealing features. First, the construction algorithm of SSP can adaptively update the proposal density in importance sampling process based on the existing information. Second, a hyperparameter is introduced to balance the representativeness of sequentially added points with the representativeness of overall points, so that some special purpose experimental designs, such as augmented design and sliced designs, can be efficiently constructed by setting the hyperparameter.

Appendix

To prove Theorem 1, we require following Lemmas 1 and 2.

Lemma 1

Let $\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}$ and $q_1(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N)$ is defined as (5). Then function

$$\begin{aligned} \begin{aligned} u_1(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})=&\frac{1}{{n_k}N} \sum _{i=1}^{n_k} \sum _{m=1}^N w({\mathbf {y}}_m)\left( \frac{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i} \Vert _2^2}{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2}\right. \\&\left. +\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2\right) \end{aligned} \end{aligned}$$

is the surrogate function of $q_1$ at the current iteration $\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$, where ${\mathbf {y}}_m\ne {\mathbf {x}}_{n_c+i}^{(t)},\ \forall \ i=1,\ldots ,n_k,\ m=1,\ldots ,N$.

Proof of Lemma 1

On the one hand, according to Arithmetic-Geometric mean inequality,

$$\begin{aligned} \begin{aligned}&\frac{2}{n_kN} \sum _{i=1}^{n_k} \sum _{m=1}^N w({\mathbf {y}}_m)\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}\Vert _2\\&\quad =\frac{2}{n_kN} \sum _{i=1}^{n_k} \sum _{m=1}^N w({\mathbf {y}}_m)\left( \frac{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}\Vert _2^2}{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2}\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2\right) ^{\frac{1}{2}}\\&\quad \le \frac{2}{n_kN} \sum _{i=1}^{n_k} \sum _{m=1}^N w({\mathbf {y}}_m)\frac{1}{2}\left( \frac{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i} \Vert _2^2}{\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2}+\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2\right) \\&\quad = g_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}) \end{aligned} \end{aligned}$$

Therefore, $g_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})\ge q_1(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N)$, $\forall \ \{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}$. On the other hand,

$$\begin{aligned} \begin{aligned} g_2(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})&= \frac{2}{{n_k}N} \sum _{i=1}^{n_k} \sum _{m=1}^N w({\mathbf {y}}_m)\Vert {\mathbf {y}}_m - {\mathbf {x}}_{n_c+i}^{(t)} \Vert _2\\&= q_1(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N). \end{aligned} \end{aligned}$$

In the light of Definition 5, the proof is complete.$\square$

Lemma 2

Let $\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}$ and $q_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})$ is defined as (6). Then function

$$\begin{aligned} \begin{aligned}&u_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})\\&\quad =-q_2(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})-\frac{\rho }{n_k^2} \sum _{i=1}^{n_k}\sum _{\begin{array}{c} j=1\\ j\ne i \end{array}}^{n_k} \frac{2({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)})^T({\mathbf {x}}_{n_c+i} -{\mathbf {x}}_{n_c+i}^{(t)})}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)}\Vert _2}\\&\qquad - \frac{2(1-\rho )}{n_kn_c}\sum _{i=1}^{n_k} \sum _{j=1}^{n_c}\frac{({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j)^T({\mathbf {x}}_{n_c+i}-{\mathbf {x}}_{n_c+i}^{(t)})}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j\Vert _2} \end{aligned} \end{aligned}$$

(8)

is the surrogate function of $-q_2$ at the current iteration $\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$, where $\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$ is pairwise distinct and ${\mathbf {x}}_j\ne {\mathbf {x}}_{n_c+i}^{(t)},\ \forall \ i=1,\ldots ,n_k,\ j=1,\ldots ,n_c$.

Proof of Lemma 2

The tangent plane of $-q_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})$ at the current iteration $\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$ is

$$\begin{aligned} \begin{aligned}&t(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k})=-q_2(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})\\&\quad -\sum _{i=1}^{n_k}\left( \frac{\partial q_2 (\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})}{\partial {\mathbf {x}}_{n_c+i}}\Big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}}\right) ^T ({\mathbf {x}}_{n_c+i}-{\mathbf {x}}_{n_c+i}^{(t)}), \end{aligned} \end{aligned}$$

where

$$\begin{aligned}&\frac{\partial q_2 (\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})}{\partial {\mathbf {x}}_{n_c+i}}\Big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}} \\&\quad = \left( \frac{\partial \frac{\rho }{n_k^2} \sum _{j\ne i} 2\Vert {\mathbf {x}}_{n_c+i}-{\mathbf {x}}_{n_c+j}^{(t)}\Vert _2}{\partial {\mathbf {x}}_{n_c+i}}\right. +\left. \frac{\partial \frac{2(1-\rho )}{n_kn_c}\sum _{j=1}^{n_c} \Vert {\mathbf {x}}_{n_c+i}-{\mathbf {x}}_{j}\Vert _2 }{\partial {\mathbf {x}}_{n_c+i}} \right) \Big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}}\\&\quad = \frac{\rho }{n_k^2} \sum _{\begin{array}{c} j=1\\ j\ne i \end{array}}^{n_k} \frac{2({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)})^T({\mathbf {x}}_{n_c+i} -{\mathbf {x}}_{n_c+i}^{(t)})}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)}\Vert _2}\\&\qquad +\frac{2(1-\rho )}{n_kn_c} \sum _{j=1}^{n_c}\frac{({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j)^T({\mathbf {x}}_{n_c+i}-{\mathbf {x}}_{n_c+i}^{(t)})}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j\Vert _2}. \end{aligned}$$

Therefore, $t(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k})=u_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})$. As $q_2$ is convex about $\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}$, then $-q_2$ is concave. According to property of concave function,

$$\begin{aligned} u_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})\ge -q_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}),\ \forall \ \{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}. \end{aligned}$$

In addition, it is obvious that

$$\begin{aligned} u_2(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})=-q_2(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}) \end{aligned}$$

In the light of Definition 5, the proof is complete.$\square$

Proof of Theorem 1

Based on Lemmas 1 and 2, it is easy to show that $u^*$ is the the surrogate function of $q_1-q_2$ at the current iteration $\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$, we omit it here. Noting that $u^*$ is a separable and quadratic function for variables, then take the partial derivatives of the variables and set them equal to 0,

$$\begin{aligned} \begin{aligned}&\frac{\partial u^*(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N, \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})}{\partial {\mathbf {x}}_{n_c+i}}\Big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}}=0,\\&\frac{1}{n_kN}\sum _{m=1}^Nw_k({\mathbf {y}}_m) \frac{2({\mathbf {x}}_{n_c+i}-{\mathbf {y}}_m)}{\Vert {\mathbf {y}}_m-{\mathbf {x}}_{n_c+i}^{(t)}\Vert _2} -\frac{2\rho }{n_k^2}\sum _{\begin{array}{c} j=1\\ j\ne i \end{array}}^{n_k}\frac{{\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)}}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}- {\mathbf {x}}_{n_c+j}^{(t)}\Vert _2}\\&\quad -\frac{2(1-\rho )}{n_kn_c}\sum _{j=1}^{n_c}\frac{{\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_j\Vert _2}=0,\ i=1,\ldots ,n_k. \end{aligned} \end{aligned}$$

After a simple calculation, we can get the closed-form in (7).$\square$

Proof of Theorem 2

To prove this theorem, we require a lemma in Razaviyayn et al. (2013):

Lemma 3

Assume $u({\mathbf {x}}|{\mathbf {x}}_t)$ is a surrogate function of $f({\mathbf {x}})$ at the current iteration ${\mathbf {x}}_t$ on a closed convex set ${\mathcal {X}}\subseteq {\mathbb {R}}^p$. If $u({\mathbf {x}}|{\mathbf {x}}_t)$ is continuous and their directional derivatives with respect to ${\mathbf {x}}$ satisfy

$$\begin{aligned} u'({\mathbf {x}}|{\mathbf {x}}_t;{\mathbf {d}})\big |_{{\mathbf {x}}={\mathbf {x}}_t}= f'({\mathbf {x}};{\mathbf {d}})\big |_{{\mathbf {x}}={\mathbf {x}}_t},\quad \forall \ \text {direction}\ {\mathbf {d}},\ \text {and}\ {\mathbf {x}}_t+{\mathbf {d}}\in {\mathcal {X}}, \end{aligned}$$

Then every limit point of the iterates in MM algorithm is a stationary point of $f({\mathbf {x}})$.

Proof

(Theorem 2) According to Lemma 3 and Theorem 1, we only need to prove that $u^*$ is continuous and the condition of directional derivative holds. As $u^*$ is a quadratic function for variables, it is continuous. For any feasible direction set ${\mathbf {d}}=\{{\mathbf {d}}_i\}_{i=1}^{n_k} \in {\mathbb {R}}^{n_kp}$, and ${\mathbf {x}}_{n_c+i}+{\mathbf {d}}_i\in {\mathcal {X}},\ i=1,\ldots ,n_k$, then directional derivative (let ${\mathcal {M}}=\{{\mathbf {y}}_m\}_{m=1}^N\bigcup \{{\mathbf {x}}_j\}_{j=1}^{n_c}\bigcup \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}$ for brevity)

$$\begin{aligned} \begin{aligned}&[u^*]'(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ {\mathcal {M}};{\mathbf {d}})\big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}}\\&\quad =\underset{\lambda \downarrow 0}{\lim }\frac{u^*(\{{\mathbf {x}}_{n_c+i}^{(t)}+ \lambda {\mathbf {d}}_i\}_{i=1}^{n_k}\ |\ {\mathcal {M}})-u^*(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k} \ |\ {\mathcal {M}})}{\lambda }\quad \text {(L'Hospital's rule)}\\&\quad =\frac{1}{n_kN}\sum _{i=1}^{n_k}\sum _{m=1}^Nw({\mathbf {y}}_m)\frac{2({\mathbf {x}}_{n_c+i}^{(t)} -{\mathbf {y}}_m)^T{\mathbf {d}}_i}{\Vert {\mathbf {x}}_{n_c+i}^{(t)} -{\mathbf {y}}_m\Vert _2}-\left( \frac{\rho }{n_k^2}\sum _{i=1}^{n_k}\sum _{\begin{array}{c} j=1\\ j\ne i \end{array}}^{n_k} \frac{({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)})^T{\mathbf {d}}_i}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{n_c+j}^{(t)}\Vert _2} \right. \\&\qquad \left. +\frac{2(1-\rho )}{n_kn_c}\sum _{i=1}^{n_k} \sum _{j=1}^{n_c} \frac{({\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{j})^T{\mathbf {d}}_i}{\Vert {\mathbf {x}}_{n_c+i}^{(t)}-{\mathbf {x}}_{j}\Vert _2}\right) \\&\quad = q_1'(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N;{\mathbf {d}})-q_2'(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c};{\mathbf {d}})\big |_{\{{\mathbf {x}}_{n_c+i}={\mathbf {x}}_{n_c+i}^{(t)}\} _{i=1}^{n_k}}. \end{aligned} \end{aligned}$$

This proves the stationary convergence of Algorithm 1.$\square$