Approximations of semicontinuous functions with applications to stochastic optimization and statistical estimation

Abstract

Upper semicontinuous (usc) functions arise in the analysis of maximization problems, distributionally robust optimization, and function identification, which includes many problems of nonparametric statistics. We establish that every usc function is the limit of a hypo-converging sequence of piecewise affine functions of the difference-of-max type and illustrate resulting algorithmic possibilities in the context of approximate solution of infinite-dimensional optimization problems. In an effort to quantify the ease with which classes of usc functions can be approximated by finite collections, we provide upper and lower bounds on covering numbers for bounded sets of usc functions under the Attouch-Wets distance. The result is applied in the context of stochastic optimization problems defined over spaces of usc functions. We establish confidence regions for optimal solutions based on sample average approximations and examine the accompanying rates of convergence. Examples from nonparametric statistics illustrate the results.

Appendix

Proof of Theorem 4.4

Let $\rho >0$ and $F = \{f\in {\text {usc-fcns}}({\mathbb {R}}^d)~|~f(x)\ge -\rho \text{ for } \text{ at } \text{ least } \text{ one } x\in [0,\rho ]^n\}$. We show that F cannot be covered with a lower number of balls than stipulated. Clearly, $\mathop {\mathrm{dist}}\nolimits _\infty (0,\mathrm{hypo} \;f) \le \rho$ for all $f\in F$. Thus, in view of (3), ${\mathbb {d}}(0,f) \le \rho + 1$ for all $f\in F$, where 0 is the zero function on ${\mathbb {R}}^n$, and F is therefore bounded.

Next, let $\varepsilon \in (0,\rho e^{-\rho }/6]$. We discretize $[0,\rho ]^n$ by defining $x_i^k = k \rho /\nu _\varepsilon$, $k = 1, \ldots , \nu _\varepsilon -1$ and $i=1, \ldots , n$, where

$$\begin{aligned} \nu _\varepsilon = \left\lfloor \frac{\rho e^{-\rho }}{3\varepsilon }\right\rfloor \ge 2, \end{aligned}$$

with $\lfloor a \rfloor$ being the largest integer not exceeding a. The discretization of $[0,\rho ]^n$ then contains the points $(x_1^{k_1}, x_2^{k_2}, \ldots , x_n^{k_n})$, with $k_i \in \{1, 2, \ldots , \nu _\varepsilon -1\}$ and $i=1, \ldots , n$. Clearly, the distance between any two such points in the sup-norm is at least $\rho /\nu _\varepsilon \ge 3\varepsilon e^\rho$. We carry out a similar discretization of $[-\rho ,0]$ and define $y^l = l \rho / \nu _\varepsilon$, $l=1, \ldots , \nu _\varepsilon$. The functions that are finite on the discretization points of $[0,\rho ]^n$, with values at each such point equal to $y^l$ for some l, and have value minus infinity elsewhere are given by $F_{\varepsilon }$, i.e.,

$$\begin{aligned} F_{\varepsilon } =&\{f\in {\text {usc-fcns}}({\mathbb {R}}^n)~|~ \text{ for } \text{ each } x=(x_1^{k_1}, \ldots , x_n^{k_n}), \\&\text{ with } k_i \in \{1, 2, \ldots , \nu _\varepsilon -1\}, f(x) = y^l\\&\text{ for } \text{ some } l=1, \ldots , \nu _\varepsilon ; f(x) = -\infty \text{ otherwise } \}. \end{aligned}$$

Certainly, $F_{\varepsilon } \subset F$. We next define

$$\begin{aligned} G_\varepsilon (f) = \{g\in {\text {usc-fcns}}({\mathbb {R}}^n)~|~ \hat{\mathbb {d}}_\rho (f,g) \le \varepsilon e^\rho \}, ~ \text{ for } f\in {\text {usc-fcns}}({\mathbb {R}}^n). \end{aligned}$$

We establish that $G_\varepsilon (f) \cap G_\varepsilon (f') = \emptyset$ for $f,f'\in F_{\varepsilon }, f\ne f'$. Suppose for the sake of a contradiction that there is a g with $g\in G_\varepsilon (f)$ and $g\in G_\varepsilon (f')$ for $f,f'\in F_\varepsilon$, $f\ne f'$. Then, $\hat{\mathbb {d}}_\rho (f,g) \le \varepsilon e^\rho$ and $\hat{\mathbb {d}}_\rho (f',g) \le \varepsilon e^\rho$. However, since $f\ne f'$, there exists a point $x\in [0,\rho ]^n$ with $|f(x) - f'(x)| \ge 3 \varepsilon e^\rho$. Without loss of generality, suppose that $f(x) \ge f'(x) + 3\varepsilon e^\rho$. Since $f(z), f'(z) = -\infty$ for all $z\ne x$ with $\Vert z-x\Vert _\infty < 3\varepsilon e^\rho$, we have that $\hat{\mathbb {d}}_\rho (f,g) \le \varepsilon e^\rho$ implies that $g(z) \ge f(x) - \varepsilon e^\rho$ for some $z\in {\mathbb {B}}(x,\varepsilon e^\rho )$. Moreover, $\hat{\mathbb {d}}_\rho (f',g) \le \varepsilon e^\rho$ implies that $g(z) \le f'(x) + \varepsilon e^\rho \le f(x) - 3\varepsilon e^\rho + \varepsilon e^\rho = f(x) - 2\varepsilon e^\rho$ for all $z\in {\mathbb {B}}(x,\varepsilon e^\rho )$. Since this is not possible for g, we have reached a contradiction. Thus, $G_\varepsilon (f) \cap G_\varepsilon (f') = \emptyset$ for $f,f'\in F_{\varepsilon }, f\ne f'$.

By Lemma 4.1, for any $f\in {\text {usc-fcns}}({\mathbb {R}}^n)$,

$$\begin{aligned} {\mathbb {d}}(f,g) \ge e^{-\rho } \hat{\mathbb {d}}_\rho (f,g) > e^{-\rho } \varepsilon e^\rho = \varepsilon \text{ for } \text{ all } g\not \in G_\varepsilon (f). \end{aligned}$$

Hence, for $f\in F_{\varepsilon }$, an ${\mathbb {d}}$-ball with radius $\varepsilon$ that contains f needs to be centered at some $g\in G_\varepsilon (f)$. Since the sets $G_\varepsilon (f)$, $f\in F_{\varepsilon }$, are nonoverlapping, a cover of $F_{\varepsilon }$ by ${\mathbb {d}}$-balls with radius $\varepsilon$ must involve a number of balls that is at least as great as the number of functions in $F_{\varepsilon }$, which is $\nu _\varepsilon ^{m_\varepsilon }$, where $m_\varepsilon = (\nu _\varepsilon -1)^n$. Thus,

$$\begin{aligned} \log N(F,\varepsilon ) \ge \nu _\varepsilon ^n \log \nu _\varepsilon \ge \left( \frac{\rho e^{-\rho }}{3\varepsilon }-2\right) ^n \log \left( \frac{\rho e^{-\rho }}{3\varepsilon }-1\right) . \end{aligned}$$

(11)

Let $c_1 = |\log (\rho e^{-\rho }/4)|$ and ${{\bar{\varepsilon }}} = \min \{\rho e^{-\rho }/12, e^{-2c_1}\}$. Continuing from (11), we then find that

$$\begin{aligned} \log N(F,\varepsilon ) \ge \left( \frac{\rho e^{-\rho }}{6}\right) ^n \left[ 1+ \frac{\log (\rho e^{-\rho }/4)}{\log \varepsilon ^{-1}} \right] \frac{1}{\varepsilon ^n}\log \frac{1}{\varepsilon }. \end{aligned}$$

Since $\log \varepsilon ^{-1} \ge 2|\log (\rho e^{-\rho }/4)|$ for $\varepsilon \in (0, {{\bar{\varepsilon }}}]$, we have that

$$\begin{aligned} \log N(F,\varepsilon ) \ge \left( \frac{\rho e^{-\rho }}{6}\right) ^n \frac{1}{2}\frac{1}{\varepsilon ^n}\log \frac{1}{\varepsilon }~ \text{ for } \varepsilon \in (0, {{\bar{\varepsilon }}}], \end{aligned}$$

and the conclusion is reached. $\square$