ITSO: a novel inverse transform sampling-based optimization algorithm for stochastic search

Abstract

Optimization algorithms appear in the core calculations of numerous Artificial Intelligence (AI) and Machine Learning methods and Engineering and Business applications. Following recent works on AI’s theoretical deficiencies, a rigour context for the optimization problem of a black-box objective function is developed. The algorithm stems directly from the theory of probability, instead of presumed inspiration. Thus the convergence properties of the proposed methodology are inherently stable. In particular, the proposed optimizer utilizes an algorithmic implementation of the n-dimensional inverse transform sampling as a search strategy. No control parameters are required to be tuned, and the trade-off among exploration and exploitation is, by definition, satisfied. A theoretical proof is provided, concluding that when falling into the proposed framework, either directly or incidentally, any optimization algorithm converges. The numerical experiments verify the theoretical results on the efficacy of the algorithm apropos reaching the sought optimum.

Appendices

Appendix 1: Programming code

The corresponding computer code is available at GitHub https://github.com/nbakas/ITSO.jl. The examples of Figure 3 may be reproduced by running __run.jl. The short version of the Algorithm 1 is available in Julia (Bezanson et al. 2017) Language, and particularly in the file ITSO-short.jl, along with Octave (Contributors 2020), in the file ITSOshort.m, and Python (Contributors 2020), in the file ITSO-short.py. The implementation of the framework is integrated in a few lines of computer code, which can be easily adapted for case specific applications with high efficiency.

Appendix 2: Black-Box functions

The following functions were used for the numerical experiments. Equations 20, 21 (Elliptic, Cigar), were utilized from Liang et al. 2013, Cigtab (Eq. 22), Griewank (Eq. 23) from Au and Leung 2012, Quartic (Eq. 24) from Jamil and Yang 2013, Schwefel (Eq. 25), Rastrigin (Eq. 26), Sphere (Eq. 27), and Ellipsoid (Eq. 28) from Finck et al. 2010; Feldt 2013-2018, and Alpine (Eq. 29) from Hussain et al. 2017. Equations 30, 31, 32, were developed by the authors. The code implementation for the selected equations appears in file functions_opti.jl in the supplementary computer code.

The exact variation used in this work is as follows. We have adopted the notation presented in the Nomenclature section, where i denotes the optimization history step, and j the dimension of the design variable $x_{ij}$.

$$\begin{aligned}&\begin{aligned} f_{elliptic}(\mathbf {x}_i)=\sum _{j=1}^n{c_j ( x_{ij} +\frac{3}{2})^2}, \text {where} \\ {\mathbf {c}}=10^3 \{0,\frac{1}{n-1},\dots ,1\}. \end{aligned}\end{aligned}$$

(20)

$$\begin{aligned}&f_{cigar}(\mathbf {x}_i)=x_1^2+\sum _{j=2}^n{| x_{ij} |}.\end{aligned}$$

(21)

$$\begin{aligned}&f_{cigtab}(\mathbf {x}_i)=x_1^2+\sum _{j=2}^{n-1}{| x_{ij} |}+x_n^2. \end{aligned}$$

(22)

$$\begin{aligned}&f_{griewank}(\mathbf {x}_i)=1+{\frac{1}{4000}}\sum _{{j=1}}^{n}x_{ij}^{2}-\prod _{{j=1}}^{n}\cos \left( {\frac{x_{ij}}{{\sqrt{j}}}}\right) . \end{aligned}$$

(23)

$$\begin{aligned}&f_{quartic}(\mathbf {x}_i)=\sum _{j=1}^n{j (x_{ij}-2)^4}.\end{aligned}$$

(24)

$$\begin{aligned}&\begin{aligned} f_{schwefel}(\mathbf {x}_i)=\sum _{j=1}^n{ c_j^2}, \text {where} \\ {c_j}=\sum _{k=1}^j{(x_{ik}-9)}. \end{aligned}\end{aligned}$$

(25)

$$\begin{aligned}&\begin{aligned} f_{rastrigin}(\mathbf {x}_i)=10 n + \sum _{j=1}^n{(x_{ij}+\frac{7}{10})^2}\\ -10 \sum _{j=1}^n{\cos (2 \pi (x_{ij}+\frac{7}{10})^2 )}. \end{aligned}\end{aligned}$$

(26)

$$\begin{aligned} f_{sphere}(\mathbf {x}_i)=\sum _{j=1}^n{(x_{ij}-\frac{13}{10})^2}.\end{aligned}$$

(27)

$$\begin{aligned}&\begin{aligned} f_{ellipsoid}(\mathbf {x}_i)=\sum _{j=1}^n{(x_{ij}-\sqrt{2})^2}. \end{aligned}\end{aligned}$$

(28)

$$\begin{aligned}&f_{alpine}(\mathbf {x}_i)=\sum _{j=1}^n{|x_{ij} \sin {x_{ij}} + \frac{1}{10} x_{ij}|}.\end{aligned}$$

(29)

$$\begin{aligned}&f_{x\_j}(\mathbf {x}_i)=\sum _{j=1}^n{(x_{ij}-j-\frac{21}{10})^2}.\end{aligned}$$

(30)

$$\begin{aligned}&f_{x\_5}(\mathbf {x}_i)=\sum _{j=1}^n{(x_{ij}-5)^2} - 5.\end{aligned}$$

(31)

$$\begin{aligned}&f_{sin\_x}(\mathbf {x}_i)=\sum _{j=1}^n{( \sin (x_{ij}+\frac{7}{10}) + \frac{(x_{ij}+\frac{7}{10})^2}{100}} ). \end{aligned}$$

(32)