Generation of point sets by convex optimization for interpolation in reproducing kernel Hilbert spaces

Abstract

We propose algorithms to take point sets for kernel-based interpolation of functions in reproducing kernel Hilbert spaces (RKHSs) by convex optimization. We consider the case of kernels with the Mercer expansion and propose an algorithm by deriving a second-order cone programming (SOCP) problem that yields n points at one sitting for a given integer n. In addition, by modifying the SOCP problem slightly, we propose another sequential algorithm that adds an arbitrary number of new points in each step. Numerical experiments show that in several cases the proposed algorithms compete with the P-greedy algorithm, which is known to provide nearly optimal points.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Notes

  1. 1.

    If K is translation-invariant, we can choose a starting point x1 arbitrarily.

  2. 2.

    As shown in [3, Proposition 3.3.3], the function \(X \mapsto \log \det X\) is concave on the set of positive definite matrices. Then, the objective function of Problem (3.12) can be regarded as the restriction of this function to the affine subset of the set of positive definite matrices. Therefore, the objective function is log-concave.

  3. 3.

    The authors of [16] consider the more general case that aj are matrices with a common number of rows.

  4. 4.

    For example, we observed that Algorithm 1 failed for n = 100 in the case of Example 2 because of this phenomenon.

  5. 5.

    Therefore, the optimal value of SOCP (4.6) is \(2^{(p-1)2^{p-1}}\) (the optimal value of Problem (4.1))1/.

References

  1. 1.

    Alizadeh, F., Goldfarb, D.: Second-order cone programming. Math. Program., Ser. B 95, 3–51 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  2. 2.

    Atkinson, K., Han, W.: Spherical Harmonics and Approximations on the Unit Sphere: an Introduction. Springer, Berlin (2012)

    Google Scholar 

  3. 3.

    Borwein, J.M., Lewis, A.S.: Convex Analysis and Nonlinear Optimization: Theory and Examples, 2nd edn. Springer, New York (2006)

    Google Scholar 

  4. 4.

    Briani, M., Sommariva, A., Vianello, M.: Computing Fekete and Lebesgue points: simplex, square, disk. J. Comput. Appl. Math. 236, 2477–2486 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  5. 5.

    Buhmann, M.: Radial basis functions. Acta Numer. 10, 1–38 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  6. 6.

    Dai, F., Xu, Y.: Approximation Theory and Harmonic Analysis on Spheres and Balls. Springer, New York (2013)

    Google Scholar 

  7. 7.

    Fasshauer, G.: Meshfree Approximation Methods with MATLAB. World Scientific, Singapore (2007)

    Google Scholar 

  8. 8.

    Fasshauer, G., McCourt, M.: Kernel-Based Approximation Methods Using MATLAB. World Scientific, Singapore (2015)

    Google Scholar 

  9. 9.

    Lobo, M.S., Vandenberghe, L., Boyd, S., Lebret, H.: Applications of second-order cone programming. Linear Algebra Appl. 284, 193–228 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  10. 10.

    De Marchi, S.: On optimal center locations for radial basis function interpolation: computational aspects. Rend. Sem. Mat. Univ. Pol. Torino 61, 343–358 (2003)

    MATH  MathSciNet  Google Scholar 

  11. 11.

    De Marchi, S.: Geometric greedy and greedy points for RBF interpolation. In: Proceedings of the International Conference on Computational and Mathematical Methods in Science and Engineering, CMMSE 2009 (2009)

  12. 12.

    De Marchi, S., Schaback, R., Wendland, H.: Near-optimal data-independent point locations for radial basis function interpolation. Adv. Comput. Math. 23, 317–330 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. 13.

    Nguyen, V.P., Rabczuk, T., Bordas, S., Duflot, M.: Meshless methods: a review and computer implementation aspects. Math. Comput. Simul. 79, 763–813 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  14. 14.

    Müller, S.: Complexity and stability of kernel-based reconstructions (in German). dissertation, Georg-August-Universität Göttingen, Institut für Numerische und Angewandte Mathematik, Lotzestrasse 16-18, D-37083 Göttingen, Jan 2009. Göttinger Online Klassifikation: EDDF 050 (2009)

  15. 15.

    Sangol, G.: Computing optimal designs of multiresponse experiments reduces to second order cone programming. J. Statist. Plann. Inference 141, 1684–1708 (2011)

    Article  MathSciNet  Google Scholar 

  16. 16.

    Sangol, G., Harman, R.: Computing exact D-optimal designs by mixed integer second-order cone programming. Ann. Statist. 43, 2198–2224 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  17. 17.

    Santin, G., Haasdonk, B.: Convergence rate of the data-independent P-greedy algorithm in kernel-based approximation. Dolomites Research Notes on Approximation 10, 68–78 (2017)

    MATH  MathSciNet  Google Scholar 

  18. 18.

    Schaback, R., Wendland, H.: Adaptive greedy techniques for approximate solution of large RBF systems. Numer. Algor. 24, 239–254 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  19. 19.

    Schaback, R., Wendland, H.: Kernel techniques: from machine learning to meshless methods. Acta Numer. 15, 543–639 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  20. 20.

    Tanaka, K.: Matlab programs for generating point sets for interpolation in reproducing kernel Hilbert spaces. https://github.com/KeTanakaN/mat_points_interp_rkhs (last accessed on October 19, 2018)

  21. 21.

    Wendland, H.: Scattered Data Approximation. Cambridge University Press, Cambridge (2005)

    Google Scholar 

Download references

Acknowledgments

The author thanks Takanori Maehara for his valuable comment about the MATLAB programs used in the numerical experiments. Thanks to the comment, much faster execution of the proposed algorithms has been realized than that in the initially submitted version of this article. Furthermore, the author gives thanks to the anonymous reviewers for their valuable suggestions about this article.

Funding

The author is supported by the grant-in-aid of Japan Society of the Promotion of Science with KAKENHI Grant Number 17K14241.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Ken’ichiro Tanaka.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix. Remarks on implementation of SOCP (4.6) by MOSEK

Appendix. Remarks on implementation of SOCP (4.6) by MOSEK

MOSEK

As a software to solve SOCP problems, we use the MOSEK Optimization Toolbox for MATLAB 8.1.0.56, which is provided by MOSEK ApS in Denmark (https://www.mosek.com/, last accessed on 19 October 2018).

Expression of rotated second-order cones

Recall that \(p = \lceil \log _{2} \ell \rceil \). In SOCP (4.6) in which is even, we need to consider the rotated second-order cone constraints

$$ \begin{array}{@{}rcl@{}} && z_{ij}^{2} \leq t_{ij} w_{i} \qquad (i=1,\ldots,m, j=1,\ldots, \ell), \end{array} $$
(5.6)
$$ \begin{array}{@{}rcl@{}} && {u_{i}^{2}} \leq u_{2i} u_{2i+1} \qquad (i=1,\ldots, 2^{p-1}-1), \end{array} $$
(5.7)
$$ \begin{array}{@{}rcl@{}} && {u_{i}^{2}} \leq \tilde{g}_{2i-2^{p}+1} \tilde{g}_{2i-2^{p}+2} \qquad (i = 2^{p-1}, \ldots, 2^{p-1} + \ell/2 - 1). \end{array} $$
(5.8)

Furthermore, in the counterpart in which is odd, we need to consider the constraints

$$ \begin{array}{@{}rcl@{}} && z_{ij}^{2} \leq t_{ij} w_{i} \qquad (i=1,\ldots,m, j=1,\ldots, \ell), \end{array} $$
(5.9)
$$ \begin{array}{@{}rcl@{}} && {u_{i}^{2}} \leq u_{2i} u_{2i+1} \qquad (i=1,\ldots, 2^{p-1}-1), \end{array} $$
(5.10)
$$ \begin{array}{@{}rcl@{}} && {u_{i}^{2}} \leq \tilde{g}_{2i-2^{p}+1} \tilde{g}_{2i-2^{p}+2} \qquad (i = 2^{p-1}, \ldots, 2^{p-1} + (\ell-3)/2), \end{array} $$
(5.11)
$$ \begin{array}{@{}rcl@{}} && {u_{i}^{2}} \leq \tilde{g}_{2i-2^{p}+1} u_{1} \qquad (i = 2^{p-1} + (\ell-1)/2). \end{array} $$
(5.12)

However, MOSEK permits only the expression \({\eta _{1}^{2}} + {\cdots } + {\eta _{N}^{2}} \leq 2 \xi \zeta \) for a rotated second-order cone, where we just need the case that N = 1. Therefore, we employ the following variable transformations:

$$ \begin{array}{@{}rcl@{}} && w_{i} = 2 \hat{w}_{i} \quad (i = 1,\ldots, m), \end{array} $$
(5.13)
$$ \begin{array}{@{}rcl@{}} && \tilde{g}_{i} = \sqrt{2} \hat{g}_{i} \quad (i = 1, \ldots, \ell-1), \end{array} $$
(5.14)
$$ \begin{array}{@{}rcl@{}} && \tilde{g}_{\ell} = \left\{\begin{array}{ll} \sqrt{2} \hat{g}_{\ell} & (\ell \text{ is even}), \\ 2 \hat{g}_{\ell} & (\ell \text{ is odd}). \end{array}\right. \end{array} $$
(5.15)

We do not change ui because of the following reason. If we change the constraints \({u_{i}^{2}} \leq u_{2i} u_{2i+1}\) to \({u_{i}^{2}} \leq 2 u_{2i} u_{2i+1}\), they just change the optimal value by \(2^{(p-1)2^{p-1}}\) times.Footnote 5 Consequently, we deal with the SOCP problem

(5.16)

in the case that is even, and

(5.17)

in the case that is odd.

Restriction about cone constraints

MOSEK does not permit that a variable belongs to plural cones. However, we need to consider the cone constraints in (5.16) and (5.17) in which each ui appears 2 times and each \(\hat {w}_{i}\) appears times. Therefore, we prepare the copies of these variables:

$$ \begin{array}{@{}rcl@{}} u_{i} \ \leftrightarrow \ u_{i1}, u_{i2}, \qquad \hat{w}_{i} \ \leftrightarrow \ \hat{w}_{i1}, \hat{w}_{i2}, \ldots, \hat{w}_{i \ell}, \end{array} $$

and set the linear constraints between these variables:

$$ \begin{array}{@{}rcl@{}} && u_{i1} = u_{i2}, \end{array} $$
(5.18)
$$ \begin{array}{@{}rcl@{}} && \hat{w}_{i1} = \hat{w}_{i2} = {\cdots} = \hat{w}_{i \ell}. \end{array} $$
(5.19)

Then, we introduce a vector

$$ \begin{array}{@{}rcl@{}} v = (&& u_{11}, u_{12}, \ u_{21}, u_{22}, \ \ldots, \ u_{(2^{p}-1), 1}, u_{(2^{p}-1), 2}; \\ && \hat{g}_{1}, \hat{g}_{2}, \ldots, \hat{g}_{\ell}; \\ && z_{11}, \ldots, z_{m1}, \ z_{12}, \ldots, z_{m2}, \ \ldots, \ z_{1\ell}, \ldots, z_{m\ell}; \\ && t_{11}, \ldots, t_{m1}, \ t_{12}, \ldots, t_{m2}, \ \ldots, \ t_{1\ell}, \ldots, t_{m\ell}; \\ && \hat{w}_{11}, \ldots, \hat{w}_{m1}, \ \hat{w}_{12}, \ldots, \hat{w}_{m2}, \ \ldots, \ \hat{w}_{1\ell}, \ldots, \hat{w}_{m\ell})^{T}. \end{array} $$
(5.20)

Linear constraints

To describe the linear constraints in (5.16) and (5.17) in terms of the vector v, we provide a matrix A and vectors bl and bu that express the constraints as blAvbu.

We begin with the case that is even. First, we express the constraints in (5.18) and (5.19) by

$$ \begin{array}{@{}rcl@{}} \underbrace{ \left[\begin{array}{llllllll} 1 & -1 & 0 & 0 & {\cdots} & 0 & 0 & 0_{\ell + 3m\ell}^{T} \\ 0 & 0 & 1 & -1 & {\cdots} & 0 & 0 & 0_{\ell + 3m\ell}^{T} \\ {\vdots} & & & & {\ddots} & & {\vdots} & {\vdots} \\ 0 & 0 & 0 & 0 & {\cdots} & 1 & -1 & 0_{\ell + 3m\ell}^{T} \end{array}\right]}_{A_{1}} v = 0_{2^{p} - 1}, \end{array} $$
(5.21)

and

$$ \begin{array}{@{}rcl@{}} \underbrace{ \left[\begin{array}{lllllll} O_{m, 2(2^{p}-1) + \ell + 2m\ell} & I_{m} & -I_{m} & O_{m,m} & {\cdots} & O_{m,m} & \\ O_{m, 2(2^{p}-1) + \ell + 2m\ell} & O_{m,m} & I_{m} & -I_{m} & & O_{m,m} & \\ {\vdots} & & & {\ddots} & & & \\ O_{m, 2(2^{p}-1) + \ell + 2m\ell} & O_{m,m} & {\cdots} & O_{m,m} & I_{m} & -I_{m} & \end{array}\right]}_{A_{2}} v = 0_{m (\ell-1)}, \end{array} $$
(5.22)

respectively. Next, we consider the constraint

$$ \begin{array}{@{}rcl@{}} (a_{1}, \ldots, a_{m}) Z = \left[\begin{array}{llll} \sqrt{2} \hat{g}_{1} & 0 & {\cdots} & 0 \\ \ast & \sqrt{2} \hat{g}_{2} & {\ddots} & {\vdots} \\ \ast & \ast & {\ddots} & 0 \\ \ast & \ast & \ast & \sqrt{2} \hat{g}_{\ell} \end{array}\right]. \end{array} $$

By using v and the expression

$$ \begin{array}{@{}rcl@{}} (a_{1}, \ldots, a_{m}) = \left[\begin{array}{lll} \boldsymbol{\alpha}_{1} & {\cdots} & \boldsymbol{\alpha}_{\ell} \end{array}\right]^{T} \qquad (\boldsymbol{\alpha}_{j} \in \mathbf{R}^{m}), \end{array} $$

we express the linear constraint as follows:

(5.23)

Next, we express the constraint \(\hat {w}_{1\ell } + {\cdots } + \hat {w}_{m\ell } = n/2\) by

$$ \begin{array}{@{}rcl@{}} \underbrace{ \left[\begin{array}{ll} 0_{2(2^{p}-1) + \ell + 3m(\ell - 1)}^{T} & {1_{m}^{T}} \end{array}\right]}_{A_{4}} v = \frac{n}{2}. \end{array} $$
(5.24)

Next, we express the inequality constraints

$$ \begin{array}{@{}rcl@{}} \sum\limits_{i=1}^{m} t_{ij} \leq \sqrt{2} \hat{g}_{j} \quad (j = 1,{\ldots} , \ell) \end{array} $$

by

(5.25)

Finally, we consider the constraints

$$ \begin{array}{@{}rcl@{}} {u_{i}^{2}} \leq {u_{1}^{2}} \qquad (i = 2^{p-1} + \ell/2, \ldots, 2^{p}-1), \end{array} $$
(5.26)

which are equivalent to the linear constraints uiu1 owing to the non-negative constraints ui ≥ 0. These constraints are expressed by

(5.27)

Consequently, by setting

$$ \begin{array}{@{}rcl@{}} A & =& \left[\begin{array}{llllll} {A_{1}^{T}} & {A_{2}^{T}} & {A_{3}^{T}} & {A_{4}^{T}} & {A_{5}^{T}} & {A_{6}^{T}} \end{array}\right]^{T}, \end{array} $$
(5.28)
$$ \begin{array}{@{}rcl@{}} b_{\mathrm{l}} & =& \left( 0_{2^{p} - 1}^{T}, \ 0_{m (\ell-1)}^{T}, \ 0_{\ell (\ell+1)/2}^{T}, \ n/2, \ -\infty \cdot 1_{\ell}^{T} , \ -\infty \cdot 1_{2^{p-1}-\ell/2} \right)^{T}, \end{array} $$
(5.29)
$$ \begin{array}{@{}rcl@{}} b_{\mathrm{u}} & =& \left( 0_{2^{p} - 1}^{T}, \ 0_{m (\ell-1)}^{T}, \ 0_{\ell (\ell+1)/2}^{T}, \ n/2, \ 0_{\ell}^{T} , \ 0_{2^{p-1}-\ell/2}^{T} \right)^{T}, \end{array} $$
(5.30)

we can express the linear constraints as blAvbu.

In the case that is odd, we need to modify A3, A5, A6, bl, and bu. We replace the component “\(-~\sqrt {2}\)” in the last rows of A3 and A5 by “− 2”. Furthermore, we modify A6, bl, and bu as

figureh

respectively.

Constraints for the ranges of the variables

Besides the constraints ui ≥ 0 and \(0 \leq \hat {w}_{j} \leq 1/2\) in (5.16) and (5.17), we can add \(\hat {g}_{i} \geq 0\) and tij ≥ 0 because any solution that does not satisfy these constraints is useless. Therefore, by using the vectors given by

$$ \begin{array}{@{}rcl@{}} \tilde{b}_{\mathrm{l}} & =& \left( 0_{2(2^{p}-1) + \ell}^{T}, \ -\infty \cdot 1_{m\ell}^{T}, \ 0_{2m\ell}^{T} \right)^{T}, \end{array} $$
(3.4)
$$ \begin{array}{@{}rcl@{}} \tilde{b}_{\mathrm{u}} & =& \left( \infty \cdot 1_{2(2^{p}-1) + \ell + 2m\ell}^{T}, \ (1/2) \cdot 1_{m\ell}^{T} \right)^{T}, \end{array} $$
(3.5)

we can express the constraints for the ranges of the variables as \(\tilde {b}_{\mathrm {l}} \leq v \leq \tilde {b}_{\mathrm {u}}\).

Cone constraints

To express the cone constraints in (5.16) and (5.17) by MOSEK, we need to specify the components of the vector v that correspond to the constraints. For the constraints

$$ \begin{array}{@{}rcl@{}} {u_{i}^{2}} \leq 2 u_{2i} u_{2i+1} \qquad (i = 1, \ldots, 2^{p-1} - 1), \end{array} $$

we take the components v2i, v2(2i)− 1, and v2(2i+ 1)− 1 for ui, u2i, and u2i+ 1, respectively. For the constraints

$$ \begin{array}{@{}rcl@{}} {u_{i}^{2}} \leq 2 \hat{g}_{2i - 2^{p} + 1} \hat{g}_{2i - 2^{p} + 2} \qquad \left\{\begin{array}{ll} (i = 2^{p-1} , \ldots, 2^{p-1} + \ell/2 + 1) & (\ell \text{ is even}), \\ (i = 2^{p-1} , \ldots, 2^{p-1} + (\ell-3)/2) & (\ell \text{ is odd}), \end{array}\right. \end{array} $$

we take the components v2i, \(v_{2^{p} + 2i - 1}\), and \(v_{2^{p} + 2i}\) for ui, \(\hat {g}_{2i - 2^{p} + 1}\), and \(\hat {g}_{2i - 2^{p} + 2}\), respectively. In the case that is odd, we take the components \(v_{2^{p} + \ell - 1}\), \(v_{2(2^{p} -1) + \ell }\), and v1 for the constraint \(u_{2^{p-1} + (\ell -1)/2}^{2} \leq 2 \hat {g}_{\ell } u_{1}\). Finally, for the constraints

$$ \begin{array}{@{}rcl@{}} z_{ij}^{2} \leq 2 t_{ij} \hat{w}_{i} \qquad (i=1,\ldots, m, \ j=1,\ldots, \ell), \end{array} $$

we take the components \(v_{2(2^{p}-1) + \ell + ij}\), \(v_{2(2^{p}-1) + \ell + ij + m\ell }\), and \(v_{2(2^{p}-1) + \ell + ij + 2m\ell }\).

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tanaka, K. Generation of point sets by convex optimization for interpolation in reproducing kernel Hilbert spaces. Numer Algor 84, 1049–1079 (2020). https://doi.org/10.1007/s11075-019-00792-w

Download citation

Keywords

  • Reproducing kernel Hilbert space
  • Kernel interpolation
  • Point set
  • Power function
  • Optimal design
  • Second-order cone programming

Mathematics Subject Classification (2010)

  • 65D05
  • 65D15
  • 41A05
  • 46E22