Starshaped sets

Abstract

This is an expository paper about the fundamental mathematical notion of starshapedness, emphasizing the geometric, analytical, combinatorial, and topological properties of starshaped sets and their broad applicability in many mathematical fields. The authors decided to approach the topic in a very broad way since they are not aware of any related survey-like publications dealing with this natural notion. The concept of starshapedness is very close to that of convexity, and it is needed in fields like classical convexity, convex analysis, functional analysis, discrete, combinatorial and computational geometry, differential geometry, approximation theory, PDE, and optimization; it is strongly related to notions like radial functions, section functions, visibility, (support) cones, kernels, duality, and many others. We present in a detailed way many definitions of and theorems on the basic properties of starshaped sets, followed by survey-like discussions of related results. At the end of the article, we additionally survey a broad spectrum of applications in some of the above mentioned disciplines.

1 Introduction

While convex geometry has a long history (see, for instance, the bibliographies in [453] as well as in [185, 232, 234, 292]), going back even to ancient times (e.g., Archimedes) and to later contributors like Kepler, Euler, Cauchy, and Steiner, the geometry of starshaped sets is a younger field, and no historical overview exists. The notion of starshapedness is a natural generalization of that of convexity. Its various versions appear later, starting with the beginning of the 20th century (see, e.g., [6, 117]). An intensive development started in the 60’s, notably with the pioneering work of Léopold Bragard (see [68,69,70,71,72,73,74,75,76]); many papers appeared also in the 70’s, 80’s, and later. This branch of geometry is still actively developing, and it has numerous applications. Starshaped sets come up naturally in many fields, including functional analysis, classical geometry (e.g., as star polytopes), computational geometry, integral geometry, mixed integer programming, optimization, operations research, etc. (see, e.g., [136, 143, 151, 165, 418], but also our Sects. 14, 17, 19).

This survey is a first exposition on starshapedness in the broadest sense, following Victor Klee’s suggestion to one of the authors, of writing such a survey. It reflects (to our best knowledge) a very large part of all existing literature. However, we were not able to cover all existing topics and viewpoints; the missing aspects might yield enough material for a second part.

For the sake of convenience, we give here a list of our Sections:

• 1. Introduction

• 2. Basic notions and definitions

• 3. Cones

• 4. Starshaped sets and visibility

• 5. Star generators. Representations of the kernel

• 6. Krasnosel’skii-type theorems

• 7. Asymptotic structure of starshaped sets

• 8. Support cones

• 9. Separation of starshaped sets

• 10. Extremal structure of starshaped sets

• 11. Dimension of the kernel of a starshaped set

• 12. Admissible kernels of starshaped sets

• 13. Radial functions of starshaped sets

• 14. Sums, unions, and intersections of starshaped sets

• 15. Spaces of starshaped sets

• 16. Selectors for star bodies

• 17. Star duality, intersection bodies, and related topics

• 18. Extensions and generalizations

• 19. Applications and further topics

  • 19.1 Discrete and computational geometry

  • 19.2 Inequalities

  • 19.3 Starshapedness in differential geometry

  • 19.4 Starshaped sets and PDE

  • 19.5 Starshapedness in fixed point theory

  • 19.6 Starshaped sets in approximation theory

  • 19.7 Applications of starshapedness in optimization

  • 19.8 Further topics.

2 Basic notation and definitions

The geometry of starshaped sets is developed mainly in Euclidean spaces, but many results have generalizations in topological vector spaces. Starshaped sets are also considered in more general settings (see Sect. 18). Unless otherwise stated, the framework in which we shall work is the d-dimensional Euclidean space ${\mathbb {R}}^{d}$ (with origin 0), and we shall use standard concepts from set theory, topology, linear algebra, and convexity. If $A\subset {\mathbb {R}}^{d}$ is any set, its complement will be denoted by $A^{\prime }$. If a, b are different points, by [a : b] , [a : b), and (a : b) we shall denote the segment with endpoints a and b, the half-line or ray with origin a through b, and the line through a and b, respectively. The replacement of [ by ] in the definition of [a : b] or [a : b) simply means that the endpoint or origin a does not belong to the segment or ray; analogously for the replacement of ] by [ in [a : b]. In the case of segments we extend this definition agreeing that $[a:a]=\{a\}$, and we say that a segment is non-degenerate if its endpoints are different. Similarly, open and closed intervals in ${\mathbb {R}}$ will be denoted by $]\alpha :\beta [$, $[\alpha :\beta ]$, $[\alpha :+\infty [$, etc. We shall denote half-lines from the origin and lines through the origin by $\Delta$ and $\Gamma ,$ respectively, and sometimes we shall refer to such a $\Delta$ as a direction. When necessary, we shall write $\Delta _{u}$ instead of $\Delta$ for the half-line [0 : u), where $u\ne 0$. Some authors use different notations, in particular $\Delta \left( a,b\right)$ for the non-degenerate segment with endpoints a and b, which is a special case of the notation for the k -dimensional simplex with vertices $x_{0},\ldots ,x_{k}$; that is the set

$$\begin{aligned} \Delta (x_{0},\ldots ,x_{k}):=\{t_{0}x_{0}+\cdots t_{k}x_{k}\ |\ t_{0},\ldots ,t_{k}\ge 0,\quad \Sigma _{i=0}^{k}t_{i}=1\}, \end{aligned}$$

where $x_{0},\ldots ,x_{k}\in {\mathbb {R}} ^{d}$, $1\le k\le d,$ are affinely independent points. But we shall reserve the greek letter $\Delta$ for half-lines.

If $A\subset {\mathbb {R}}^{d}$, $B\subset {\mathbb {R}}^{d}$, and $\Lambda \subset {\mathbb {R}}$, then $A+B=\left\{ a+b:a\in A,b\in B\right\}$ (Minkowski sum) and $\Lambda A=\left\{ \lambda a:\lambda \in \Lambda ,a\in A\right\}$. When A or $\Lambda$ are singletons, we shall simply write $a+B$ and $\lambda A$. If x is a point and X is a set, we shall write $x\cup X$ instead of the more clumsy $\{x\}\cup X$.

We refer to standard texts for the basic concepts of convexity (e.g., [235, 325, 431, 453, 515, 527, 538]). If $A\subset {\mathbb {R}}^{d}$ is any set, its affine hull will be denoted by $\mathrm {aff}\,A$, its convex hull by $\mathrm {conv\,}A,$ and its relative interior (with respect to aff$\,A)$ and interior will be denoted by $\mathrm {relint\,}A$ and $\mathrm {int\,}A$, respectively. For the closure and the boundary of A we will write $\mathrm {cl\,}A$ and $\mathrm {bd\,}A$, respectively. By $U\left( x:\varepsilon \right)$ and $B\left( x:\varepsilon \right)$ we shall denote the open and closed balls with center x and radius $\varepsilon$. The closed unit ball $B\left( 0:1\right)$ will be denoted by B, and the unit sphere $\mathrm {bd}B$ by $S^{d-1}$. A nonempty compact subset A of ${\mathbb {R}}^{d}$ such that $\mathrm {int}A$ is connected is a body if $\mathrm {cl}\left( \mathrm { int}A\right) =A$. More generally, if we drop the boundedness condition, we say that A is a hunk or regular domain. The family of all compact convex sets in ${\mathbb {R}}^{d}$ is denoted by ${\mathcal {K}}^{d}$, and the family of convex bodies is denoted by ${\mathcal {K}}_{0}^{d}$. If A is a set and $p\notin A$, the join of p and A, denoted by [p : A], is the union of all segments [p : a] for $a\in A$. More generally, the join of two disjoint sets A and B is the union [A : B] of all segments [a : b] with $a\in A$, $b\in B$. If A and B are convex, [A : B] coincides with $\mathrm {conv}(A\cup B),$ which we shall also denote by $\mathrm {conv} (A,B)$ (see [515]). If A is a closed convex set, we denote by $\mathrm {ext\,}A$ the set of extreme points of A. As usual, flats are the translates of subspaces of any dimension, and we say that a set S is line-free if there is no line contained in S.

A point set centered at the origin will simply be called a centered set. By $M^{d}$ we denote a d-dimensional normed or Minkowski space, i.e., a d-dimensional real Banach space whose unit ball is a d-dimensional centered, compact and convex set (see the monograph [492], introducing this field).

3 Cones

Cones play an important role in the geometry of starshaped sets. A general reference for cones is [283]. A set $C\subset {\mathbb {R}}^{d}$ is a cone if there exists a point $a\in {\mathbb {R}}^{d}$ such that

$$\begin{aligned} ]0,\infty [\cdot (C-a)\subset C-a. \end{aligned}$$

The point a is then called an apex of the cone, and it does not necessarily belong to the cone. A cone may not have only one apex; but if one of its apices belongs to the cone, then all of them belong to it. The set of all apices of C is a flat called the summit of C, and it is denoted by $^{\gamma }C$¹. Therefore

$$\begin{aligned} ^{\gamma }C=\{y:\,]0,\infty [\cdot (C-y)\subset C-y\}. \end{aligned}$$

If $^{\gamma }C\subset C$, the cone is called sharp; otherwise it is dull. A sharp cone C is salient if no line through any of its apices is included in C, and in that case it has a unique apex. If C is a salient cone with apex a, the opposite cone of C is the cone $\text {opp }C=2a-C$. In particular, if $C=\left[ a:p\right)$, then $\mathrm {opp}\left[ a:p\right) =2a-\left[ a:p\right)$ is the half-line with origin a in the opposite direction of $\left[ a:p\right)$.

For any cone C with apex a, its translate to the origin $C_{0}=C-a$ is called the centralized cone of C.

If V is a compact convex set and K is a convex cone with apex 0, then $V+K$ is a convex conic tail.

If $A\subset {\mathbb {R}}^{d}$ and $a\in {\mathbb {R}}^{d}$, the conic hull of A from a is the set

$$\begin{aligned}{}[a:A)=\{a\}+[0,\infty [\cdot (A-a). \end{aligned}$$

Thus, $\left[ a:A\right)$ is the smallest (with respect to inclusion) sharp cone with apex a including A. If $a=0$, it will be simply called the conic hull of A and denoted by C(A). If A is a convex set, then [a : A) is a convex cone for any a. The witness cone $[a:A)_{0}$ of A is the centralized conic hull of A from a:

$$\begin{aligned}{}[a:A)_{0}=[a:A)-a. \end{aligned}$$

If $a\notin A$ and A is convex, the convex cone $[a:2a-A)=\bigcup \nolimits _{x\in A}~[a:2a-x)$ is said to be opposite to [a : A).

It is well known (see [24]) that if A is convex and compact, then [a : A) is a closed convex cone, but this may not be the case if A is merely closed.

4 Starshaped sets and visibility

The notion of starshaped set is a natural generalization of that of a convex set. While a set $C\subset {\mathbb {R}}^{d}$ is convex if $\forall x\in C$ and $\forall y\in C$ the segment $\left[ x:y\right] \subset C$, a set $S\subset {\mathbb {R}}^{d}$ is starshaped if $\exists x\in S$ such that $\forall y\in S$ the segment $\left[ x:y\right] \subset S.$ A closely related notion is that of visibility: given two points x, y of a set S, we say that x sees y via S if $\left[ x:y\right] \subset S$. If S is a set and $x\in S$, the star of x in S is the set $\mathrm {st}(x:S)=\left\{ y\in S:[x:y]\subset S\right\}$, that is, the set of all points of S which are seen from x via S. If $A\subset S$, the star of A in S is the set $\mathrm {st}\left( A:S\right) =\bigcap \nolimits _{x\in A}\mathrm {st}\left( x:S\right)$. A set S is said to be starshaped if there exists some $x\in S$ such that $\mathrm {st}(x:S)=S$. The kernel of S, denoted by $\mathrm {\ker }S$, is the set of all $x\in S$ such that $\mathrm {st}(x:S)=S$, and its elements are called star centers of S. This concept was defined by Brunn [117]. It is easy to see that $\ker S$ is a convex set. Obviously, S is convex if and only if $\ker S=S$. Some authors speak of convex kernel, or mirador, instead of kernel. Sometimes, if $p\in \ker S$, we shall say that S is starshaped at p. A starshaped set in ${\mathbb {R}}^{d}$ is called a starshaped body (or star body) if it has nonempty interior.

Without explicit mention we shall assume that all starshaped sets are non-empty.

Remark 1

Bragard [69] proved that

$$\begin{aligned} \ker \sum \nolimits _{i=1}^{n}\left( \alpha _{i}S_{i}\right) \supset \sum \nolimits _{i=1}^{n}\alpha _{i}\ker S_{i}. \end{aligned}$$

If S is a (starshaped) set, a convex component of S is a maximal (with respect to inclusion) convex subset of S. The following problem is due to Valentine (Problem 9.3 in [527]): characterize starshaped sets in a Minkowski space in terms of convex components of these sets. A solution of this problem was given independently by Toranzos [506] and Smith [467]:

Theorem 2

A subset of a linear space is starshaped if and only if the intersection of all its convex components is nonempty.

In the following section we will present a more precise result.

Let $w,z\in [x:y).$ We shall say that z is subsequent to w in [x : y) if $z\notin [x:w]$. If S is a starshaped set and $m\in \mathrm {ker}\,S$, we shall say that a point $l\in S$ is the last point of a ray [m : x) in S if $l\in [m:x)$ and there is no point $y\in [m:x)\cap S$ subsequent to l in [m : x). Obviously, such points may not exist if S is not closed.

An easy consequence of the definitions is the extension (to starshaped sets) of the linear accessibility theorem for convex sets, which states that if C is a convex set, $m\in \mathrm {int}C$, and $x\in \mathrm {cl}C$, then $[m:x[~\subset \mathrm {int}\,C$. The corresponding result for starshaped sets, which is a basic tool for many problems (see [255, 441, 527]), is given by

Theorem 3

Let S be a starshaped set such that $\mathrm {int}\ker S\ne \varnothing$. If $m\in \mathrm {int\ker }\,S$ and $x\in \mathrm {cl}\,S$, then $[m:x[~\subset \mathrm {int}\,S$.

Corollary 4

If S is a closed starshaped set such that $\mathrm {int\ker } \,S\ne \varnothing$, then S is a hunk.

If C is a closed convex set and $x\in \mathrm {bd}C$, then it is obvious that $C\cap V$ is convex for every convex neighborhood of x. Apparently, the boundary of a closed starshaped set S, which is not convex, must contain points x such that $S\cap V$ is not convex whatever the convex neighborhood V of x is. Such points are called points of local nonconvexity of S. More precisely: a point $x\in \mathrm {bd}S$ is a point of local nonconvexity of S if for every neighborhood V of x the set $S\cap V$ is not convex. The set of points of local nonconvexity of a set S is denoted by $\mathrm {lnc}\,S$. Points of S that are not of local nonconvexity of S are said to be points of local convexity of S, and the set of all such points is denoted by $\mathrm {lc}S$. A classical theorem of Tietze (see [494] and also [450, 476]) states, roughly, that a compact and connected set S (a compact starshaped set in particular) is nonconvex if and only if in its boundary there are points of local nonconvexity. This was generalized by Klee (see [298, 299]):

Theorem 5

Let S be a closed connected set in a locally convex linear topological space. Then the following conditions are equivalent: (i) $\mathrm {lnc} \,S=\varnothing$, (ii) S is convex.

If S is a closed connected nonconvex set, then for every convex component C of S there are local nonconvexity points of S arbitrarily close to $\mathrm {bd}C$. More precisely, Toranzos [509] proved

Theorem 6

Let S be a closed connected nonconvex set, and let K be a convex component of S. Then for every neighborhood V of the origin

$$\begin{aligned} \left( K+V\right) \cap \mathrm {lnc}\,S\ne \varnothing . \end{aligned}$$

If $\mathrm {lnc}\,S$ is compact, then in every convex component of S there exist points of local nonconvexity:

Theorem 7

Let S be a closed connected set such that $\mathrm {lnc}\,S$ is compact or empty, and let K be a convex component of S. Then the following conditions are equivalent: (i) $K=S$, (ii) $K\cap \mathrm {lnc} \,S=\varnothing$.

In [4], a sufficient condition for a compact set S in 3-space to be locally starshaped is proved; this condition is given in terms of local nonconvexity points of S. In [482] the following is shown: Let S be a compact, connected subset of a Banach space, and for any $x\in S$ let S(x) be the set of all points from S which can be seen from x via S. Then S is starshaped iff the intersection of all closures of convex hulls of the sets S(x) is nonempty, where x runs through every suitable neighborhood of points of local nonconvexity of S. Using Valentine’s definition of mild convexity points of sets in topological vector spaces, the following theorem is proved in [187]: Let S be an open, connected set in a locally convex Hausdorff space over the reals. If the boundary of S contains exactly one point which is not a mild convexity point of S and this point is not isolated in this boundary, then the dimension of the space is 2 and S is starshaped.

McMullen [376] proved the equivalence of the following (and more) properties of a compact set S in ${\mathbb {R}}^{d}$: (1) each homothetic image $\lambda S$ with $0<\lambda <1$ is the intersection of a family of translates of S; (2) the set S is starshaped and each maximal convex subset of it is a cap-body of the kernel (i.e., the convex hull of $\ker S$ and a countable set of points outside of $\ker S$ whose pairwise joining segments meet $\ker S$). Based on such equivalences, he proved the following nice characterization of starshaped sets among compact sets: S is starshaped if and only if for each $0<\lambda <1$ there exists a point $z_{\lambda }$ such that $(1-\lambda )z_{\lambda }+\lambda S\subset S$.

A set S is said to have the finite visibility property, or to be a finitely starshaped set, or also finitely starlike set, if for every $A\subset S$ with $\mathrm {card}A<+\infty$, there is $x_{A}\in S$ which sees every point of A via S, see [7, 90, 104, 413, 435, 436]. Edelstein, Keener and O’Brien [176] confirmed that S is starshaped if and only if the set of regular points of S has the finite visibility property. In [20] it is proved that the closure of a bounded finitely starlike set is starshaped, and the author gives an example to illustrate that the boundedness condition cannot be relaxed. Based on this, related versions of Helly’s and Krasnosel’skii’s theorem are derived. In [481] it is proved that in a uniformly smooth and uniformly rotund Banach space B, the closure of a bounded $S\subset B$ is starshaped if and only if S has the finite visibility property, where B is said to be uniformly rotund if and only if for any $\epsilon >0$ there exists $\delta (\epsilon )>0$ such that $1-\left\| \frac{1}{2}(x+y)\right\| \ge \delta$ whenever $\left\| x-y\right\| \ge \epsilon$ and $\left\| x\right\| =\left\| y\right\| =1$, and it is said to be uniformly smooth if and only if for any $\epsilon >0$ there exists $\delta (\epsilon )>0$ such that $1-\left\| \frac{1}{2}(x+y)\right\| \le \epsilon \left\| x-y\right\|$ whenever $\left\| x-y\right\| \le \delta$ and $\left\| x\right\| =\left\| y\right\| =1$.

In [175] it is proved that a normed linear space is infinite-dimensional [reflexive] iff it contains a weakly closed, linearly bounded [bounded] set which has the finite visibility property but fails to be starshaped. In the definition of the finite visibility property replacing the word “finite” with the word “countable”, Borwein et al. [65] derived results of the following type: In metrizable locally convex spaces every relatively weakly compact subset with the finite visibility property has the countable visibility property. The authors present examples of a non-starshaped set with the countable visibility property, and also of a set having the finite visibility property, but not the countable visibility property. Further related results can be found in [66].

A notion somewhat more restrictive than starshapedness was defined by Demianov and Rubinov (cf. [157, 441]): let S be a closed proper subset of ${\mathbb {R}}^{d}$ with non-empty interior. Then S is said to be strongly starshaped whenever there exists an $a\in \mathrm {int}S$ such that for every $u\in S^{d-1}$ the half-line $a+\Delta _{u}$ does not intersect the boundary $\mathrm {bd}S$ more than once. We also say that S is strongly starshaped at a, and some authors also say that S is radiative at a. The set of all $a\in S$ with these properties is denoted by $\ker _{*}S$, and we call it the strong kernel of S. Bragard [69] and also Shveidel [461] proved that if $S\subset {\mathbb {R}}^{d}$ is strongly starshaped, then $\ker _{*}S$ is convex. A strongly starshaped set is starshaped, but the converse is not true. If S is strongly starshaped, the point a mentioned in the definition must belong to $\ker S$; that is, $\ker _{*}S\subset \ker S$, because if $a\in \mathrm {int}S\backslash \ker S$, then there exists $x\in S$ such that $\left[ a:x\right] \nsubseteqq S$. Whence, $\left[ a:x\right] \cap \mathrm {bd}S$ must include at least two different points because S is closed. As a direct consequence of the linear accessibility theorem (Theorem 3) we obtain the following: if S is a closed starshaped set, then

$$\begin{aligned} \mathrm {int}\ker S\subset \ker _{*}S\subset \ker S \end{aligned}$$

(see also [69, 157, Chapter 6]). Therefore, if a closed starshaped set S is not strongly starshaped, then $\mathrm {int}\ker S=\varnothing$ (see also [442]).

A set S in ${\mathbb {R}}^{d}$ is an $L_{n}$ set (see [116, 268, 479, 480, 528]) if every pair of points of S can be joined by a polygonal arc in S consisting of at most n line segments. In [483] short proofs of the following results of Valentine are given: Let S be closed and connected in ${\mathbb {R}}^{d}$. If S has at most n points of local nonconvexity, then S is an $L_{n}$ set. If the set of points of local nonconvexity of S can be decomposed into n convex sets, then S is an $L_{2n+1}$ set.

Another extension of the notion of starshapedness, called $\Delta$-starshapedness, was introduced in [211], see also Sect. 0.7 of Gardner$^{\prime }$s book [204]. A set $A\subset {\mathbb {R}}^{d}$ is $\Delta$- starshaped at a point a (which may be outside of A), whenever for every line $a+\Gamma$ through a the intersection $\left( a+\Gamma \right) \cap A$ is connected. Clearly, if A is $\Delta$-starshaped at a point $a\in A$, then A is starshaped at a. Also, for every $a\in {\mathbb {R}}^{d}$, if A is $\Delta$-starshaped at a, then $\left[ a:A\right]$ is starshaped at a.

Melzak [378] investigated properties of starshaped sets in 3-space (extending Blaschke’s selection theorem suitably and representing the kernel as the intersection of certain systems of half-spaces), and he showed that for any convex set C there is an almost ball-shaped body with kernel C.

In [526] minimal (with respect to inclusion) sets of visibility are used to characterize planar compact sets which are either convex or starshaped with respect to one point. The analogous problems for closed sets or higher dimensional analogues are posed as open questions.

Stanek [474] characterized starshapedness as follows: A set S is p- arcwise convex if each pair $x\in S$, $y\in S$ can be joined by a convex arc C(x, y) lying in the intersection of S and the triangle (p, x, y). A set is locally p-arcwise convex if each point of S has a neighborhood whose intersection with S is p-arcwise convex. The author shows that a closed connected set in a linear topological space is p-arcwise convex if and only if it is locally p-arcwise convex, which is equivalent to being starshaped with respect to p. Another topological characterization of starshapedness is given in [316].

In [58, 61], and Chapter II of [59] the notion of starshapedness is also extended to normed spaces. More precisely, extensions of the notion of starshapedness, defined with the help of generalized convexity notions and mainly useful for metrical problems in normed spaces, are discussed there. The first one is the so-called d- starshapedness defined via metric segments. This is done as follows: In the definitions of segments and starshapedness, usual segments are replaced by (metric or) d-segments, which are the sets of all points “between” the two segment endpoints which satisfy the triangle inequality with equality, with respect to the metric d of the space. Thus, the d-segment $\left[ x:z\right] _{d}$ with endpoints x and z in a normed space is the set $\left\{ y:d\left( x,z\right) =d\left( x,y\right) +d\left( y,z\right) \right\}$, where d(x, z) denotes the usual distance in this normed space. This useful concept was introduced by Menger [379]. It is clear that in strictly convex normed spaces all d-segments are usual line segments, but if the norm is not strictly convex, then d-segments can be of any dimension between 2 and the full dimension of the space. Thus, the shape of d-segments strongly depends on the shape of the unit ball. And since in the definition of d-starshaped sets (see, e.g., Sect. 15 of [59] for a more recent representation) the connecting d-segments creating the corresponding d-visibility have to be contained in the respective sets, the geometric properties of the unit ball are very important. Basic properties of d-starshaped sets were already derived in [60, 472, 496, 497].

Closely related to d-starshapedness is a second generalization of usual starshapedness, the so-called c-starshapedness introduced in [472]. A third type of starshapedness, H-starshapedness (based on the so-called H-convexity), is less related to normed spaces and is introduced via restricted intersections of halfspaces (see [60] and Chapter III from [59]).

The authors of [176] extend basic properties of finite visibility of starshaped sets to infinite-dimensional normed spaces if these spaces have a uniformly convex and uniformly smooth norm. E.g., Krasnosel’skii’s theorem is extended this way.

5 Star generators: representations of the kernel

If A is any set, then it is clear that $\ker A=\mathrm {st}\left( A:A\right) =\bigcap \nolimits _{x\in A}\mathrm {st}\left( x:A\right)$. A natural question arises: are there subsets $D\subset A$, $D\ne A$, such that $\ker A=\bigcap \nolimits _{x\in D}\mathrm {st}\left( x:A\right)$? If such a set D exists, we say that D is a star generator of $\ker A$, or that D star generates $\ker A$ (see [493]). A considerable amount of research is dedicated to finding star generators for the kernel of a starshaped set S (cf. [515]). The following early result is due to Bragard (see [75]).

Theorem 8

If S is a closed set in ${\mathbb {R}}^{d}$, then $\mathrm {bd }S$ is a star generator of $\ker S$.

Certain subsets of a star generator are also star generators (see [493]):

Theorem 9

Let S be a closed set in ${\mathbb {R}}^{d}$, D a star generator of $\ker S$, and M a dense subset of D. Then M is a star generator of $\ker S$.

Theorem 10

Let S be a compact starshaped set in ${\mathbb {R}}^{d}$ and D a star generator of $\ker S$. Then there exists a countable dense subset M of D which is a star generator of $\ker S$.

Theorem 8 was generalized in several directions, pointing to other subsets of $\mathrm {bd}S$ that are star generators of $\ker S$. We shall return to the subject repeatedly. A related question arises: is it true that if D is a star generator of $\ker A$, then the set A can be recovered from D, i.e.,

$$\begin{aligned} A=\bigcup \limits _{x\in D}\mathrm {st}\left( x:A\right) ? \end{aligned}$$

For $D=\mathrm {bd}A$ the answer is positive.

More generally: is there a family ${\mathcal {F}}$ of subsets of a starshaped set S such that

$$\begin{aligned} S=\bigcup \limits _{F\in {\mathcal {F}}}F \end{aligned}$$

(1)

and

$$\begin{aligned} \ker S=\bigcap \limits _{F\in {\mathcal {F}}}F~? \end{aligned}$$

(2)

In [511], Toranzos called any collection ${\mathcal {F}}$ of subsets of a starshaped set S a crown of S if it satisfies (2), and he called the collection ${\mathcal {F}}$ (not necessarily a crown) a covering collection if it satisfies (1). The paper [511] contains a quite complete and detailed summary, up to 1996, of many related results, proposes a general approach to these matters and suggests several lines for further study. In [506] (see also [515]) Toranzos proved that the set of convex components of a starshaped set constitutes a crown.

Theorem 11

If ${\mathcal {F}}$ is a covering family of convex components of a starshaped set S, then ${\mathcal {F}}$ is a crown, that is,

$$\begin{aligned} S=\bigcap \limits _{F\in {\mathcal {F}}}F. \end{aligned}$$

A point $y\in S$ clearly sees $x\in S$ via S if and only if there exists a neighborhood V of x such that $V\cap S\subset \mathrm {st}(y:S)$. The nova, or clear star, of x in S, denoted by $\mathrm {nova}(x:S)$, is the set of all points of $\mathrm {st}(x:S)$ that clearly see x via S. If x and y are points in S, y is a point of critical visibility of x in S, denoted by $y\in \mathrm {cv} (x:S)$, if $y\in \mathrm {int}S\cap \mathrm {bd}\left( \mathrm {st}\left( x:S\right) \right)$. Stavrakas [475] proved that the family of novae of points of local nonconvexity of a compact set $S\subset {\mathbb {R}}^{d}$ is a crown of S. In [512] Toranzos and Forte Cunto gave a generalized version of this result, namely

Theorem 12

Let S be a closed connected subset of ${\mathbb {R}}^{d}$ (or of a real locally convex topological vector space E) such that $\mathrm {lnc}\,S$ is nonempty and compact. Then the family of novae of points of local nonconvexity of S is a crown of S.

Further levels of low visibility are studied in [193].

A point $x\in S$ is a k-extreme point of S (see [13, 493]) if there is no k-simplex $\Delta (x_{0},\ldots ,x_{k})\subset S$ such that $x\in \mathrm {relint}\Delta (x_{0},\ldots ,x_{k})$. The set of all k-exteme points of S is denoted by $\mathrm {ext}_{k}S$. Independently, Kenelly et al. [289] and Tidmore [493] proved the following statement:

Theorem 13

Let S be a compact starshaped subset of $R^{d}$. The family ${\mathcal {F}} =\left\{ \mathrm {st}\left( x:S\right) :x\in \mathrm {ext}_{d-1}S\right\}$ is a crown of S.

If x and y are points of a starshaped set S, x has higher visibility via S than y if $\mathrm {st}\left( y:S\right) \subset \mathrm {st}\left( x:S\right)$. A point $x\in S$ is called a peak or point of maximal visibility in S if there exists a neighborhood V of x in S such that x has higher visibility in S than any other point of $S\cap V$. The following theorem was proved by Cel [131], and previously for compact sets by Toranzos and Forte Cunto [513]; see also the earlier papers [127,128,129] by Cel.

Theorem 14

If A is an open connected subset of ${\mathbb {R}}^{d}$ or $\mathrm {int}A$ is connected and $A=\mathrm {cl}\left( \mathrm {int}A\right)$, then $\ker A$ is the set of all points of maximal visibility in A.

The visibility cell of x in S is the set $\mathrm {vis} \left( x:S\right)$ of all the points of S having higher visibility via S than x. Toranzos proved in [509] the following statement.

Theorem 15

Let S be a closed connected set with $\mathrm {lnc}\,S$ being compact. Then the family of visibility cells of all points of local nonconvexity of S is a convex crown of S.

Let $y\in \mathrm {bd}S$ and $x\in \mathrm {st}\left( y:S\right)$. In [510] Toranzos² defined the notion of inward ray as follows: $\left[ x:y\right)$ is said to be an inward ray through y if there exists $t\in \left[ y:2y-x\right) =\mathrm {opp}\left[ y:x\right)$, $t\ne y$, such that $\left] y:t\right[ \subset \mathrm {int}S$. Otherwise $\left[ x:y\right)$ is an outward ray through y. The inner stem of y in S is the set $\mathrm {ins}\left( y:S\right)$ formed by y and all the points of $\mathrm {st}\left( y:S\right)$ that issue outward rays through y.

For the following theorem we refer to [432, 510].

Theorem 16

If S is a nonconvex hunk, then the family

$$\begin{aligned} {\mathcal {F}}=\left\{ \mathrm {ins}\left( x:S\right) :x\in \mathrm {lnc} \,S\right\} \end{aligned}$$

is a crown of S.

Toranzos proved in [510] the following relations.

Theorem 17

Let $S\subset {\mathbb {R}}^{d}$ be a closed connected set and $y\in \mathrm {bd} S$. Then

$$\begin{aligned} \mathrm {vis}\left( y:S\right) \subset \mathrm {ins}\left( y:S\right) \subset \mathrm {st}\left( y:S\right) , \\ y\in \mathrm {lc}S\Longrightarrow \mathrm {ins}\left( y:S\right) =\mathrm {st} \left( y:S\right) , \\ x\in \mathrm {ins}\left( y:S\right) \Longrightarrow [y:x)\cap \mathrm { st}\left( y:S\right) \subset \mathrm {ins}\left( y:S\right) . \end{aligned}$$

In [192] the following was shown for a smooth planar Jordan domain whose boundary has only finitely many inflection points: the kernel of such a starshaped domain S is the intersection of the stars of the inflection points in the boundary of S. Also related Krasnosel’skii-type theorems are derived, yielding conditions for the kernel of S to be 1- or 2-dimensional and given in terms of the mentioned inflection points.

Based on the PhD Thesis [430] of Robkin and properties of points of spherical support, characterizations of the kernel of starshaped sets are presented in [460].

For a closed starshaped subset S of the plane whose boundary is a continuously differentiable simple closed curve, Halpern [252] proved the following: The boundary of the kernel of S is contained in the union of the boundary of S and the tangent lines to the boundary at its inflection points. In particular, if only a finite number m of inflection points exists in the boundary of S, and if the intersection of both boundaries (also that of the kernel) is empty, then the kernel is a polygon with at most m sides. Further intersection theorems based on inflection points and yielding the kernel of S are given.

In [47] the notion of affinely starshaped sets was introduced, and a characterization of affine kernels is given there.

6 Krasnosel’skii-type theorems

Krasnosel’skii’s classical theorem (see [318]) states that a nonempty compact subset S of ${\mathbb {R}}^{d}$ is starshaped if and only if any $d+1$ points of S are visible via S from a common point. Its proof relies on Helly’s classical theorem (cf. [261]) which states that if ${\mathcal {F}}$ is a family of convex sets in ${\mathbb {R}}^{d}$, then

$$\begin{aligned} \bigcap \limits _{F\in {\mathcal {F}}}F\ne \varnothing \end{aligned}$$

if and only if any $d+1$ members of ${\mathcal {F}}$ have a nonempty intersection. A Krasnosel’skii-type theorem for a set S results as the conjunction of a theorem about the representation of $\ker S$ as the intersection of a certain family of subsets of S and a Helly-type theorem applicable to that family. For Helly’s theorem and its consequences we refer to the classical paper of Danzer, Grünbaum, and Klee [150] and also the more recent, excellent survey [172]. The difficulty of this conjunction resides in the fact that the sets of this family are not necessarily convex, as a Helly-type theorem usually requires. Normally, this difficulty is overcome by proving a new representation theorem for $\ker S$ as the intersection of the closed convex hulls of the family members. It is common to call this new representation theorem Krasnosel’skii-type Lemma; see [512], where we find the next three theorems.

Theorem 18

Let S be a compact hunk in ${\mathbb {R}}^{d}$. S is starshaped if and only if for every subset $P\subset \mathrm {lnc}\,S$ with cardinality $\mathrm {card }(P)\le d+1$ there exists a point $x\in S$ that clearly sees every point of P.

Theorem 19

Let S be a hunk in ${\mathbb {R}}^{d}$ such that $\mathrm {lnc}\,S$ is compact and there exists $y_{0}\in \mathrm {lnc}\,S$ with $\mathrm {st}\left( y_{0}:S\right)$ bounded. Then S is starshaped if and only if for every subset $P\subset \mathrm {lnc}\,S$ with $\mathrm {card}\left( P\right) \le d+1$ there exists a point $x\in S$ that clearly sees every point of P.

Theorem 20

Let S be a compact hunk in ${\mathbb {R}}^{d}$ such that for every $M\subset \mathrm {lnc}\,S$ with $card\left( M\right) \le d+1$ there exists a ball B of radius $\delta >0$ included in S and such that each point of B clearly sees every point of M. Then S is starshaped, and $\ker S$ includes a ball of radius $\delta$.

On the same line Rodríguez [434] proved the following results.

Theorem 21

Let $S\subset {\mathbb {R}}^{d}$ be a nonconvex hunk such that for every set $M\subset \mathrm {lnc}\,S$ with k elements $(k\le d+1)$ there exists a point $p\in S$ that sees each of the points of M and issues outward rays through these points. Then S is starshaped.

Theorem 22

Let $S\subset {\mathbb {R}}^{d}$ be a nonconvex hunk, and $\delta >0$ be such that for every set $M\subset \mathrm {lnc}\,S$ with k elements $(k\le d+1)$ there exists a disk D of radius $\delta$ included in the star of each point of M, and such that every point of D issues an outward ray through each point of M. Then $\ker S$ includes a disk of radius $\delta$.

The theorems of Helly and Krasnosel’skii are in fact equivalent, see [62], and also the compactness property cannot be dropped (see [96, 97]). Direct and natural refinements and extensions of Krasnosel’skii’s original theorem can be found in [57, 98, 101, 133, 180, 190, 311, 385, 528, 529], see also the monograph [538].

Stavrakas (see [478]) proved a Krasnosel’skii-type theorem for $(d-2)$-extreme points which combines and generalizes known results.

Krasnosels’skii’s theorem is also discussed in the monographs [59] (section 15 and p. 161), [61, 148] (see section E2 there), and [337, 545]. The following nice variant for polygons was proved in [386]: A side of some polygon is called a side of inflection if one of its two angles is concave and the other one is convex. Let P be a planar polygon with at least three sides of inflection. If to each triple of sides of inflection there is a point of P from which at least one point of each of the three sides can be seen, then P is starshaped.

In [312, 314, 315] combinatorial characteristics of convexity spaces related to convex sets and starshaped sets (and, in particular, with Krasnosel’skii’s theorem) are compared, yielding results on convex sums and product spaces.

Breen wrote many interesting papers on Krasnosel’ski-type results for starshaped sets. They discuss the following issues: getting conditions for the size and/or dimension of the kernel of a starshaped set (see [79, 80, 82, 83, 85, 88, 97], and further references therein), (clear) visibility of point sets of local nonconvexity (cf. [80, 82, 85, 87, 88, 92, 93, 97]) or of $(d-2)$-extreme points of a starshaped set (see [81]), and conditions for planar bounded sets which are almost starshaped if the “Krasnosel’skii number” 3 is replaced by 4 (see [89, 91]). The property of being “almost starshaped” is defined in Sect. 18.

One might interpret Krasnosel’skii’s theorem as the “starshapedness analogue” of Helly’s theorem, which originally refers to convex sets. However, there also exist direct analogues of Helly-type theorems referring to families of starshaped sets. We start with [105], where it was shown that, in a family ${\mathcal {F}}$ of compact convex sets in ${\mathbb {R}} ^{d}$, any $d+1$ or fewer members have a starshaped union if and only if the intersection of all members of ${\mathcal {F}}$ is nonempty. In [53, 54] the following extension of Helly’s theorem was derived: If ${\mathcal {F}}$ is a family of compact sets in ${\mathbb {R}}^{d}$ and any $d+1$ (not necessarily distinct) sets from ${\mathcal {F}}$ have an intersection which is nonempty and starshaped, then the intersection of all the sets from ${\mathcal {F}}$ is nonempty and starshaped. And here “intersection” can also be replaced by “union”. Concerning “union” we also refer to [270, 313]. Vesely [534] reproved the following result on $(d+1)$-families of convex sets any d-tuple of which has a common point and whose union is starshaped: If either all members of such a family are closed, or all of them are open, then their intersection is non-empty. Closely related, Breen [106] showed that if any $d+1$ (not necessarily distinct) sets from ${\mathcal {F}}$ intersect in a starshaped set whose kernel contains a translate of a set A, then the intersection of all sets from ${\mathcal {F}}$, also a starshaped one, has a kernel containing a translate of A. This paper also contains sharper results for the planar case. Continuing this, the following was derived in [107]: if every countable subfamily of ${\mathcal {F}}$ has a starshaped intersection whose kernel is at least k-dimensional, then the intersection of all members of ${\mathcal {F}}$ is also starshaped with kernel at least k-dimensional ($0\le k\le d$); and if every countable subfamily of ${\mathcal {F}}$ has a starshaped union, then the union of all members of ${\mathcal {F}}$ is also starshaped, with kernel at least k-dimensional. And if every countable subfamily of ${\mathcal {F}}$ has an intersection expressible as a union of k starshaped sets, each having a d-dimensional kernel, then the intersection of all is nonempty and expressible as a union of k such starshaped sets; if on the other hand members of ${\mathcal {F}}$ are compact and every finite subfamily of $\mathcal { F}$ has a union of k starshaped sets as its intersection, then this intersection is again a union of k starshaped sets, see [110].

In [108] the following Helly-type theorems are proved: for a nonempty finite family ${\mathcal {F}}$ of closed sets in ${\mathbb {R}}^{d}$, let $k\le d+1$, L be a $(d-k+1)$-dimensional flat in ${\mathbb {R}}^{d}$, and T be the union of all members of ${\mathcal {F}}$. If every union of at most k members of ${\mathcal {F}}$ is starshaped and its kernel contains some translate of L, then T is starshaped, and its kernel also contains a translate of L. And if every union of at most k members of ${\mathcal {F}}$ is starshaped and the kernel of each set in the union meets some translate of L, then there exists a translate $L_{0}$ of L such that every point of T sees some point of $L_{0}$ via T.

The paper [21] deals with finite unions of starshaped sets. The authors solve a problem posed by F. A. Valentine on page 178 of [527]. Namely, if the ordered pair of sets (S, K) in a linear topological space is of Helly type $(n+1,n)$ (i.e., for every $n+1$ points in S there is a point of K that sees at least n of these points via S), then, for S closed, K compact, and $n>2$, the nontrivial visibility sets in K are pairwise disjoint, also yielding sufficient conditions for the starshapedness of S. In [22] these investigations are continued, also concerning starshapedness when the equality $K=S$ holds.

In [58, 60, 61, 501, 504], and Sect. 15 of [59] Krasnosel’skii’s theorem and related statements are extended to normed spaces, using again the concept of d-starshapedness (see our explanations of d-segments, d-visibility etc. in Sect. 4). Here respective analogues of points of non-convexity play an important role (see again [501, 504]). In an analogous way, H-convexity is used in [60] to give an extension of Krasnosel’skiĭ’s theorem for H-starshaped sets.

For further studies on intersection formulae and Krasnosels$^{\prime }$ skii-type theorems in more general frameworks we refer to [132] and the references therein.

7 Asymptotic structure of starshaped sets

The following results describe “asymptotic properties” that closed starshaped sets may have. Recall that for convex sets unboundedness implies the existence of half-lines included in these sets. For properties of unbounded convex sets we refer, for example, to Rockafellar [431]. The following theorems from [255] collect related statements for starshaped sets.

Theorem 23

Let S be an unbounded closed starshaped set. Then there exists a half-line $\Delta$ such that $\ker S+\Delta \subset S$. Moreover, if $\ker S$ is unbounded, then $S+\Delta \subset S$, where $\Delta$ is now a half-line such that $m+\Delta \subset \ker S$ for some $m\in \ker S$.

Corollary 24

Let S be a closed starshaped set and $\Delta$ a direction. If $m+\Delta \subset S$ for some $m\in \ker S$, then $\ker S+\Delta \subset S$. If $m+\Delta \subset \ker S$ for some $m\in \ker S$, then $\ker S+\Delta \subset \ker S$ and $S+\Delta \subset S$. Conversely, if $S+\Delta \subset S$ for some direction $\Delta$, then $\ker S+\Delta \subset \ker S$.

Corollary 25

Let S be a closed starshaped set. If there exists a flat $F\subset S$, then $\ker S+(F-F)\subset S$.

Definition 26

Let S be a closed starshaped set and let $\Delta$ be a direction. We say that $\Delta$ is a recession direction of S if $\ker S+\Delta \subset \ker S$, and it is called an infinity direction of S if $\ker S+\Delta \subset S$. The set of all recession directions of S will be called the recession cone of S (see, e.g., [462]), denoted by $\mathrm {rc}\,S$, and the set of all infinity directions of S will be called the infinity cone of S and denoted by $\mathrm {ic}\,S$.

Note that a recession direction of a starshaped set S is a recession direction of the convex set $\ker S$.

The following figures, in which the absence of borders means that the set continues in the “natural” way, illustrate these concepts.

For the set S in Fig. 1, we have

$$\begin{aligned}&\displaystyle \mathrm {ic}S=\left\{ \lambda (0,1):\lambda \ge 0\right\} \cup \left\{ \mu (1,0):\mu \ge 0\right\} , \\&\displaystyle \mathrm {rc}S=\left\{ (0,0)\right\} , \end{aligned}$$

and for the set S in Fig. 2 we have

$$\begin{aligned} \mathrm {ic}S={\mathbb {R}}^{2}\backslash \left\{ \lambda (1,0)+\mu (0,1):\lambda>0,\mu >0\right\} , \\ \mathrm {rc}S=\left\{ \lambda (-1,0)+\mu (0,-1):\lambda \ge 0,\mu \ge 0\right\} . \end{aligned}$$

The kernel of a starshaped set S may also include lines or, more generally, flats, but in those cases the structure of S is relatively simple.

Fig. 1

Infinity cones

Fig. 2

Infinity and recession cones

Theorem 27

Let S be a closed starshaped set and assume that there exists a flat $F\subset \ker S$. Then $S\cap \left( F-F\right) ^{\bot }$ is starshaped and

$$\begin{aligned} S=S\cap \left( F-F\right) ^{\bot }+\left( F-F\right) . \end{aligned}$$

Moreover, if F is a flat included in $\ker S$ and there is no flat $F^{\prime }$ such that $F\subsetneqq F^{\prime }\subset \ker S$, then $\ker \left[ S\cap \left( F-F\right) ^{\bot }\right]$ is line-free.

Corollary 28

Let S be a closed starshaped set. If a set $H\subset \ker S$ is a hyperplane, then S is a convex set of one of the following types: (a) the whole space; (b) a closed half-space; (c) a layer between two parallel hyperplanes. In cases (b) and (c) the bounding hyperplanes of S are parallel to H.

This generalizes known results on convex sets; see [238, p.26 (Exercise 12), 486].

Here we also mention the related paper [225] in which useful properties of the kernels of usual starshaped sets and their analogues for sets which are starshaped “at infinity” are studied and important links between them are shown.

8 Support cones

Support cones of starshaped sets can be seen as analogues of support half-spaces for convex sets. According to Hansen and Martini [255] we have that if $S\subset {\mathbb {R}}^{d}$ is any set, a convex cone C with apex a and non-empty interior is a support cone of S at a if $a\in S$, $S\subset \left( \mathrm {int}C\right) ^{\prime }$ and C is a maximal (with respect to inclusion) convex cone with these properties.

Theorem 29

Let $S\subset {\mathbb {R}}^{d}$ be a closed starshaped set with $\mathrm {int} \ker S\ne \varnothing$. Then for every $x\in \mathrm {bd}S$ there exists a support cone of S at C.

Theorem 30

Let $S\subset {\mathbb {R}}^{d}$ be a closed starshaped set with $\mathrm {int} \ker S\ne \varnothing$. Then

$$\begin{aligned} S=\bigcap \nolimits _{x\in \mathrm {bd}S}\left( \mathrm {int}C_{x}\right) ^{\prime }, \end{aligned}$$

where $C_{x}$ is any support cone of S at x.

A well known lemma of Krasnosel’skiĭ states that for a closed subset S of a finite-dimensional space and a point $s\in S$, if $\left[ 0:s\right] \not \subset S$, then there exists a cone point z of S (i.e., a point z such that there is a closed half-space that has z in its bounding hyperplane and contains the set $\left\{ x\in S:\left[ x:z\right] \subset S\right\}$ so that 0 does not belong to the closed convex hull of the star of z). Edelstein, Keener, and O$^{\prime }$ Brien extended this result in [176] to infinite-dimensional normed spaces with uniformly convex and uniformly smooth norms. Using this result they proved that a set S is starshaped if it satisfies the above assumptions, is bounded and has a finite visibility property.

Let K be a closed subset of a Banach space. For $x\in K$, the pseudotangent cone P(K, x) to K at x is the closed convex hull of the set

$$\begin{aligned} \left\{ y:y=\lim t_{n}(x_{n}-x),where\ x_{n}\rightarrow x\text { }and\ t_{n}\ge 0\right\} . \end{aligned}$$

If K is starshaped, then its convex kernel is a subset of the union of all sets $P(K,x)+x$ for all $x\in K$. The central result of [63] is that this intersection actually equals the convex kernel when K is boundedly relatively weakly compact. Also, the author applies this to an extension of Krasnosel’skii’s theorem.

9 Separation of starshaped sets

It is clear, and can be shown with elementary examples, that there are pairs of disjoint closed starshaped sets $S_{1}$ and $S_{2}$ such that there is no hyperplane separating them. They cannot even be separated by cones, despite the fact that cones play a role for starshaped sets analogous to that of half-spaces for convex sets, see Fig. 3.

Fig. 3

No cone separation is possible

However, it is possible to separate them by means of complementary starshaped sets, in analogy with the Stone–Kakutani separation lemma for disjoint convex sets. This was proved by Drešević [170].

Theorem 31

Let S and T be two disjoint sets, in a real linear space L, starshaped at points p and q, respectively. Then there exist sets C and D, starshaped at p and q, respectively, such that:

$$\begin{aligned} S\subset C,\qquad T\subset D,\qquad C\cap D=\varnothing ,~\qquad and\qquad C\cup D=L. \end{aligned}$$

A different kind of separation was studied by Shveidel [461]. Given two sets A and B in ${\mathbb {R}}^{d}$, he showed them to be separable if there exists a finite set of linear functionals $f_{1},\ldots ,f_{p}$ such that $\bigcup \nolimits _{i=1}^{p}\left\{ x\in {\mathbb {R}}^{d}:f_{i}(x)\le 0\right\} \ne {\mathbb {R}}^{d}$ and for any $a\in A$, $b\in B$ the inequality $f_{i}\left( a\right) \le f_{i}\left( b\right)$ holds for some i, $1\le i\le p$. Also, he said that a vector g is a feasible direction for a set A at a point $x_{0}$ if there exist sequences $\left( g_i\right)$ and $\left( \alpha _{i}\right)$ such that $g_i\in {\mathbb {R}}^{d}$, $\alpha _{i}\in {\mathbb {R}}$, $g_{i}\rightarrow g$, $\alpha _{i}\downarrow 0$, $x_{0}+\alpha _{i}g_{i}\in A$. He proved that given two sets starshaped at a point, one has to shift them in order to get separable sets. More precisely, this is given in

Theorem 32

Assume that A and B are starshaped at the origin, $B-A$ is closed, $a,b\in {\mathbb {R}}^{d}$, and either $a-b\notin B-A$ or $a-b\in bd\left( B-A\right)$, but the vector $a-b$ is not a feasible direction of $B-A$ at the point $a-b$. Then $a+A$ and $b+B$ can be separated by d linearly independent linear functionals.

Another kind of separation of two sets by means of cones, where one of the sets is starshaped and the other one is convex and compact, was studied by Hansen and Martini [255]. They proved

Theorem 33

Let S and V be two disjoint sets, where S is closed and starshaped with $\mathrm {int}\ker S\ne \varnothing$, and V is compact. Then there exists a finite family of convex cones $K_{i}$, $i=1,\ldots ,n$, such that

$$\begin{aligned} V\subset \bigcup \nolimits _{i=1}^{n}K_{i}, \\ \bigcup \nolimits _{i=1}^{n}K_{i}\cap S=\varnothing . \end{aligned}$$

Theorem 34

Let S and V be two disjoint sets, where S is closed and starshaped with $\mathrm {int}\ker S\ne \varnothing$ and V is compact and convex. Then

• (a) there exists a convex cone K with apex 0 such that $V\subset V+K$ and $\left( V+K\right) \cap S=\varnothing$,

• (b) there exist a compact convex set $W\supsetneqq V$ and a convex cone K with apex 0 such that $V\subset W+K$, $\left( W+K\right) \cap S=\varnothing$, $V\subset \mathrm {int}\left( W+K\right)$, and $S\subset \mathrm {int}\left( W+K\right) ^{\prime }$.

Rubinov and Shveidel [444] considered closed subsets of a Euclidean space that are strongly starshaped with respect to infinity. They gave geometric characterizations and separation properties for the members of such classes.

In [443] the useful notion of conical support collection was introduced, namely referring to the separability of starshaped sets. The authors use this concept for generalizing distances between convex sets to a best approximation-like starshaped distance. It is shown that even some problems involving the distance function to arbitrary (not even starshaped) sets can be studied by means of starshaped analysis.

Let A be a subset of a (real) Banach space and $x\notin A$. The authors of [382] discuss cone-separability in terms of separation by a collection of linear functionals and give, based on this, necessary and sufficient conditions for the cone-separability of A and x. Within this framework, they characterize starshaped separability. They apply such separability to approximation problems for starshaped sets. Also in [442] the separability of two starshaped sets by a finite collection of linear functions is discussed, and the same is done in [444] with respect to infinity, but again by finite collections of linear functions. In [461] it is shown that under natural assumptions two star-shaped subsets of the d-dimensional Euclidean space can be separated by a finite number of (and even only d linearly independent) linear functionals. In these assumptions geometric properties of recession cones of the considered sets and their complements play a role, and the obtained separation statements are then applied to an optimization problem.

10 Extremal structure of starshaped sets

Extreme points play an important role in the theory of convex sets and in convex analysis; see, for example, Chapter 2 in [453] and Sect. 18 in [431], respectively. Further related monographs, also showing important relations to optimization theory, are [67, 269, 441, 464, 522].

A more general concept was defined by Asplund [13] for sets not necessarily convex: if S is a subset of a linear space L, a point $x\in S$ is a k-extreme point of S if no k-simplex $\nabla$ exists such that $x\in \mathrm {relint}\nabla \subset S$. Kenelly, Hare, Evans, and Ludescher [289], Tidmore [493], and Stavrakas [477] studied properties of the set of k-extreme points of a starshaped set. They proved the following statements.

Theorem 35

Let S be a compact starshaped subset of ${\mathbb {R}}^{d}$ $(d\ge 2)$. Then the set $S(d-2)$ of $(d-2)$-extreme points of S is a star generator of $\ker S$.

Theorem 36

Let S be a compact starshaped subset of ${\mathbb {R}}^{d}$ such that $\dim S\ge 3$. Then $S(d-2)$ is an uncountable set.

Stavrakas (see [477]) proved a somewhat more general result. For a set $S\subset {\mathbb {R}}^{d}$ he said that S has the half-ray property³ if for every $x\in S^{\prime }$ there exists a half-line $\Delta$ such that $\left( x+\Delta \right) \cap S=\varnothing$.

Theorem 37

Let $S\subset {\mathbb {R}}^{d}$ $(d\ge 2)$ be a compact set such that

$$\begin{aligned} \bigcap \nolimits _{x\in S(d-2)}\mathrm {st}\left( x:S\right) \ne \varnothing . \end{aligned}$$

Then the following statements are equivalent:

• (a) S has the half-ray property;

• (b) $\ker S=$ $\bigcap \nolimits _{x\in S(d-2)}\mathrm {st}\left( x:S\right)$.

Also the following statement is from [477].

Theorem 38

Let $S\subset R^{d}$ $(d\ge 2)$ be a compact set. Then the following properties are equivalent:

• (a) S is starshaped;

• (b) $\bigcap \nolimits _{x\in S(d-2)}\mathrm {st}\left( x:S\right) \ne \varnothing$ and S has the half-ray property.

Akin to these results, Rodríguez studied in [433] the external visibility of a closed set S (see also [119, 530]), which is practically the study of visibility in the complement of S, and connects the external visibility of a certain set S with properties that involve points of S instead of points in its complement. She defined the algebraic hull of a set $A\subset {\mathbb {R}}^{d}$ as the set $\mathrm {alg}A$ of all $y\in {\mathbb {R}}^{d}$ such that there exists $x\in A$ with $[x:y[\subset A$. This is the set of all points in ${\mathbb {R}} ^{d}$ linearly accessible in A from points of A, and, if $S\subset {\mathbb {R}}^{d}$ is a closed set such that $\mathrm {alg}S=S$ and $\mathrm {alg} S^{\prime }=\mathrm {cl}S^{\prime }$, she says that S has the shining boundary property if $S^{\prime }$ has no bounded connected components and for each boundary point of S there exists a ray issuing from it which is disjoint with $\mathrm {int}S$. She proved the following two theorems.

Theorem 39

Let $S\subset {\mathbb {R}}^{2}$ be a body such that $\mathrm {alg}S=S$ and $\mathrm {alg}S^{\prime }=\mathrm {cl}S^{\prime }$. Then S has the half-ray property if and only if S has the shining boundary property.

Theorem 40

Let $S\subset {\mathbb {R}}^{2}$ be a compact set such that $\mathrm {alg}S=S$ and $\mathrm {alg}S^{\prime }=\mathrm {cl}S^{\prime }$. Then S is starshaped if and only if S has the shining boundary property and the intersection of the stars of 0-extreme points is nonempty.

The extension of both results to higher dimensions remains an open problem. Further results on external visibility and illumination properties were studied in [194, 220, 372, 437].

Continuing [289], Goodey [221] proved that a compact subset S of ${\mathbb {R}}^{d}$ is starshaped if and only if it is nonseparating and the intersection of the stars of $(d-2)$-extreme points of S is nonempty.

Another useful concept was defined by Martini and Wenzel [371]: let S be a compact starshaped set, and K be a nonempty convex and compact subset of $\ker S$. Then a point $q_{0}\in S\backslash K$ is an extreme point of S modulo K if $q_{0}\notin \mathrm {conv}\left( K\cup \left\{ p\right\} \right)$ for all $p\in S\backslash \left( K\cup \left\{ q_{0}\right\} \right)$. They studied the operator $\sigma =\sigma _{K}:{\mathcal {P}}\left( {\mathbb {R}}^{d}\backslash K\right) \rightarrow {\mathcal {P}}\left( {\mathbb {R}}^{d}\backslash K\right)$ defined as follows: for a set $A\subset {\mathbb {R}}^{d}\backslash K$, $\sigma \left( A\right)$ consists of A as well as all points $x\in {\mathbb {R}}^{d}\backslash \left( A\cup K\right)$ such that there exists some $z\in A$ with $\left[ z:x\right] \cap K=\varnothing$, but $\left[ z:x\right) \cap K\ne \varnothing$. In other words, if $A\subset S\backslash K$, then $\sigma \left( A\right) \backslash A$ consists of those points of $S\backslash \left( A\cup K\right)$ which lie in some segment $\left[ z:x\right]$ with $z\in A$ and $x\in K$. They proved the following

Theorem 41

Let $K\ne \varnothing$ be a convex subset of ${\mathbb {R}}^{d}$. Then for every $A\subset {\mathbb {R}}^{d}\backslash K$ the set $\tau _{K}\left( A\right) =K\cup \sigma _{K}\left( A\right)$ is starshaped, and every point $x\in K$ is a star center of $\tau _{K}\left( A\right)$.

Formica and Rodríguez studied in [194] relations between visibility and illumination operators.

A basic theorem of classical convexity, usually referred to as Minkowski’s theorem, states that a compact, convex subset of ${\mathbb {R}} ^{d}$ may be “recovered” from the set of its extreme points, in the sense that the set itself is the convex hull of the set of its extreme points. This theorem was generalized in several directions, e.g. regarding unbounded sets by Klee (see [12, 296, 297, 410]), subsets of spaces of infinite dimension by Krein and Milman (see [319]), etc.; today it is common to call it in general the Krein–Milman theorem for convex sets; see, e.g., Sect. 4 in [325], Sect. 1.4 in [453], and Sect. 2.6 in [538]. Klee in [299], Martini and Wenzel in [370, 371], and Hansen and Martini in [255] extended these results to starshaped sets (see also [44, 256]). In [371] the following statements are proved.

Theorem 42

Let S be a compact starshaped set, and K be a nonempty compact convex subset of $\ker S$. Then the set $S_{0}$ of extreme points of S modulo K satisfies the condition

$$\begin{aligned} S=\tau _{K}\left( S_{0}\right) =K\cup \sigma _{K}\left( S_{0}\right) . \end{aligned}$$

Moreover, if $S_{1}\subset S\backslash K$ satisfies $\tau _{K}\left( S_{1}\right) =S$, then $S_{0}\subset S_{1}$. In other words, $S_{1}=S_{0}$ is the uniquely determined minimal subset of $S\backslash K$ satisfying $\tau _{K}\left( S_{1}\right) =S$.

For closed convex sets, the 0-extreme points are just the extreme points. Recall that if $C\subset \mathbb { {\mathbb {R}} }^{d}$ is a closed convex set, a point $p\in C$ is an extreme point of C if and only if there are no points $x,y\in C$ such that $p\in \left] x:y\right[$. This is equivalent to the statement that $C\backslash \left\{ p\right\}$ is convex. In this sense we may say that extreme points are dispensable. This view was extended to starshaped sets by Klee [300] and, independently with a different approach and without limiting to the case of compact sets, by Hansen and Martini [255]. In the latter paper the authors say that a point p of a closed starshaped set S is dispensable if $\ker \left( S\backslash \left\{ p\right\} \right) =\left( \ker S\right) \backslash \left\{ p\right\}$. Roughly speaking, this means that dropping the point p from the set does not change its kernel. The set of dispensable points of S is denoted by $\mathrm {disp}S.$ Obviously, $\mathrm {disp}S\subset \mathrm {bd}S$. Points of S that are not dispensable are called indispensable. It is clear that $p\in S$ is indispensable if and only if there exist points $x\in \ker S$ and $y\in S$ such that $p\in \left] x:y\right[$.

In turn, the following theorems were proved in [255].

Theorem 43

Let S be a closed starshaped set with $\ker S$ compact. Then

$$\begin{aligned} S=\left( \ker S+\mathrm {re}S\right) \cup \left[ \ker S:\mathrm {disp}S\right] . \end{aligned}$$

Theorem 44

Let S be a closed starshaped set with $\ker S$ compact. Then

$$\begin{aligned} S=\left( \ker S+\mathrm {rc}S\right) \cup \left[ \mathrm {conv}\left( \mathrm { ext}\ker S\right) :\mathrm {disp}S\right] . \end{aligned}$$

For compact starshaped sets they proved

Theorem 45

Let S be a compact starshaped set. Then

• (a) $S=\left[ \mathrm {conv}(\mathrm {ext}\ker S):\mathrm {disp}~S\right]$;

• (b) if f is a real linear functional, then there exists a point $x\in \mathrm {ext}\ker S\cup \mathrm {disp}~S$ such that

  $$\begin{aligned} f(x)=\underset{y\in S}{\sup }f(y)\text {.} \end{aligned}$$

If S is a compact starshaped set, its set of dispensable points is a covering star generator of $\ker S$. This can be expressed as follows.

Theorem 46

Let S be a compact starshaped set. Then

$$\begin{aligned} S= & {} \bigcup \limits _{x\in \mathrm {disp}~S}\mathrm {st}\left( x:S\right) \text {,} \\ \ker S= & {} \bigcap \limits _{x\in \mathrm {disp}~S}\mathrm {st}\left( x:S\right) \text {.} \end{aligned}$$

11 Dimension of the kernel of a starshaped set

If $C\subset {\mathbb {R}}^{d}$ is a set, the dimension of C, denoted $\dim C$, is by definition the dimension of $\mathrm {aff}C$ [527]. It is obvious that if C is convex, this dimension coincides with the dimension of the (unique) convex component of C. For starshaped sets the situation is completely different. Consider the following example in ${\mathbb {R}}^{3}$:

$$\begin{aligned} S= & {} \left\{ \left( \xi ,\nu ,\zeta \right) :\zeta \ge \left( \xi ^{2}+\nu ^{2}\right) ^{1/2}\right\} \cup \left\{ \left( \xi ,\nu ,\zeta \right) :\zeta =0\right\} \\&\cup \left\{ \left( \xi ,\nu ,\zeta \right) :\xi =\nu =0,\zeta \le 0\right\} . \end{aligned}$$

Then $\dim (\mathrm {aff}S)=3$ and $\ker S=\left\{ \left( 0,0,0\right) \right\}$, whence $\dim \ker S=0$, but S has convex components of dimensions 1, 2 or 3. It is obvious that if S has more than one point, then every convex component of S has dimension greater than 0.

Answering a question in Valentine [527], Larman [322] found a condition which ensures that the kernel of a compact set S consists of exactly one point. For a compact set S and subsets A, B, and C of ${\mathbb {R}}^{d}$ (or of a topological vector space E) of dimension at least two, he denoted by $\left( A,B,C\right)$ the set of those points of S which can be seen, via S, from a triad of points a, b, and c, with $a\in A$, $b\in B$, and $c\in C$. He said that S has property ${\mathcal {P}}$ if, whenever A is a line segment and B, C are points of S which are not collinear with any point of A,  then the set $\left( A.B,C\right)$ has linear dimension at most one, and degenerates to a single point whenever A is a point. He proved

Theorem 47

Let S be a compact subset, of dimension at least two, of a topological linear space ${\mathcal {L}}$. If S has property ${\mathcal {P}}$, then $\ker S$ consists of exactly one point.

Several authors studied bounds for the dimension of the kernel. A general result was found by Toranzos (see [507]). He said that a starshaped set S has the property $\left( \alpha _{k}\right)$ if every affinely independent $\left( k+2\right)$-tuple $P\subset S$ satisfies the condition $\dim \mathrm {st}\left( P:S\right) \le k-1$; see also [189, 258]. Toranzos [507] proved

Theorem 48

If S has the property $\left( \alpha _{k}\right)$ and $\dim S>k$, then $\dim \left( \ker S\right) \le k-1$.

Kenelly et al. [289] found an upper bound for the dimension of the kernel in terms of convex components. They called a collection of intersecting flats intersectionally independent if none of the flats contains the intersection of the remaining flats. By definition, a single flat is taken to be an intersectionally independent collection. A collection of sets is called intersectionally independent if the collection of containing flats is intersectionally independent.

Theorem 49

Let $S\subset {\mathbb {R}}^{d}$ be a set. If S contains k intersectionally independent convex components $S_{i}$, $1\le i\le k$, then $\dim \ker S\le \min \left( \dim S_{i}\right) -k+1$.

Stavrakas [475] studied the question by means of the set $\mathrm {lnc} \,S$ of points of local nonconvexity of S. He proved

Theorem 50

Let S be a compact connected subset of ${\mathbb {R}}^{d}$. Then

$$\begin{aligned} \dim \ker S\ge k\qquad for\qquad 0\le k\le d \end{aligned}$$

if and only if there exists a flat H, with $\dim H=k$, and a point $x\in relint\left( H\cap S\right)$ such that given $y\in \mathrm {lnc}\,S$, there exist open sets $N_{y}$ and $N_{x}^{y}$ such that $N_{x}^{y}\cap S\cap H$ sees $N_{y}\cap S$.

On the same lines Toranzos and Forte Cunto [512] proved

Theorem 51

Let S be a closed connected subset of a real, locally convex, linear topological space E such that $\mathrm {lnc}\,S$ is compact and nonempty, and let $\alpha$ be a cardinal, finite or infinite. Then $\dim \ker S\ge \alpha$ if and only if there exist an $\alpha$-dimensional flat H and a point $x\in \mathrm {relint}\left( H\cap S\right)$ such that, for every $y\in \mathrm {lnc}\,S$, there exists a neighborhood ${\mathcal {U}}_{y}$ of the origin satisfying the condition

$$\begin{aligned} \left( x+{\mathcal {U}}_{y}\right) \cap H\cap S\subset \mathrm {nova}\left( y:S\right) . \end{aligned}$$

In [78] the results of [77] on $(d-2)$-dimensional kernels were extended to subsets S of linear topological spaces, where additional properties like the following ones are taken into consideration: the maximal contained convex set has dimension $d-1$, and the intersection of the affine hull of the kernel of S with S is the kernel itself.

Using a modified type of visibility (replacing contained segments by contained rays), the author of [126] proved certain theorems combining Krasnosel’skiĭ’s theorem and statements on the dimension of the respective kernel.

12 Admissible kernels of starshaped sets

A natural question, posed by L. Fejes Toth, is whether any convex set is the kernel of some non-convex starshaped set. de Bruijn and Post [417] answered this question for the planar case, and Klee [301] gave a general answer with

Theorem 52

A closed convex subset K of ${\mathbb {R}}^{d}$ is the kernel of a non-convex starshaped set whenever K contains no hyperplane.

Independently, Breen solved the problem for compact sets in [84].

13 Radial functions of starshaped sets

The notion of radial function of starshaped sets is widely discussed in Sects. 0.7, 0.8 and 0.9 of Gardner’s monograph [204]; various properties (such as the known polar relation to support functions, or differentiability conditions) and related notions (like radial linear combination and radial metric) are given there in a comprehensive way, also concerning the more general definition of starshapedness given in Sect. 0.7 of this monograph.

If S is a closed starshaped set, $m\in \ker S$ and $u\in S^{d-1}$, then there are two possibilities: (i) there exists a last point p of $m+\Delta _{u}$ in S, (ii) $m+\Delta _{u}\subset S$. In the first case, let

$$\begin{aligned} \rho _{m,S}(u)=\sup \left\{ \lambda \in {\mathbb {R}} :m+\lambda u\in S\right\} . \end{aligned}$$

The radial function of S at m is the function $r_{m,S}:S^{d-1}\rightarrow {\mathbb {R}} _{+}$ defined by

$$\begin{aligned} r_{m,S}(u)=\left\{ \begin{array}{l@{\quad }l} \rho _{m,S}(u) &{} if~m+\Delta _{u}\nsubseteq S \\ +\infty &{} if~m+\Delta _{u}\subseteq S \end{array} \right. . \end{aligned}$$

As an immediate consequence of this definition we have

$$\begin{aligned} r_{m,S}>0\qquad if\ and\ only\ if\qquad m\in \mathrm {int}S. \end{aligned}$$

Also, by the linear accesssibility theorem, if S is compact and $m\in \mathrm {int}\ker S$, then $r_{m,S}$ is a Lipschitz function, whence it is continuous. Other relations between the structure of S and the continuity of the radial function are:

• If $r_{m,S}$ is continuous, then S is a compact hunk (star body).

• If S is strongly starshaped at 0, then $r_{0,S}$ is continuous. For the case that, in addition, S is compact, Klain [294] proved that the volume V(S) can be expressed by the formula

  $$\begin{aligned} V(S)=\frac{1}{d}\int \limits _{S^{d-1}}\left( r_{0,S}\right) ^{d}d\sigma , \end{aligned}$$

  where $\sigma$ is the $(d-1)$-dimensional spherical Lebesgue measure.

The role of the radial function at 0 of a set S starshaped at 0 is analogous to that of the support function for a convex set. In particular, $r_{0,S}$ determines S uniquely.

Properties of the radial function are different, depending on whether $m\in \mathrm {int}\ker S$ or $m\notin \mathrm {int}\ker S$ holds (see [257, 506]). For the non-compact case see also [392, 429, 431, 453]. If S is compact and $m\in \mathrm {int}\ker S$, then $r_{m,S}$ is a Lipschitz function, as noted before. More precisely: let $r<R$ be two positive real numbers, $k=R\left[ \left( R/r\right) ^{2}-1\right] ^{1/2}$ and, for $u_{1},u_{2}\in S^{d-1}$, let $d\left( u_{1},u_{2}\right)$ be the angle between $\Delta _{u_{1}}$ and $\Delta _{u_{2}}$. Toranzos [506] proved the following statement.

Theorem 53

If S is a compact starshaped set such that $0\in \mathrm {int}\ker S$ and $rB\subset S\subset RB$, then for every $u_{1},u_{2}\in S^{d-1}$

$$\begin{aligned} \left| r_{0,S}(u_{1})-r_{0,S}(u_{2})\right| \le kd(u_{1},u_{2}) \end{aligned}$$

holds, and k is the best Lipschitz constant valid for all such sets.

This result applies with obvious changes to $r_{m,S}$ if S is a compact starshaped set with $\mathrm {int}\ker S\ne \varnothing$ and $m\in \mathrm { int}\ker S$. If $m\notin \mathrm {int}\ker S$, the properties of $r_{m,S}$ are related to the structure of $\mathrm {bd}S$. Hansen and Martini [257] proved

Theorem 54

Let $S\ne {\mathbb {R}} ^{d}$ be a closed starshaped set and $m\in \mathrm {bd}\ker S$. Then the radial function $r_{m,S}$ of S at m is upper semicontinuous at every $u\in S^{d-1}$ and fails to be continuous at such a point u if and only if the half-line $m+\Delta _{u}$ contains at least two indispensable boundary points of S.

For the definition of dispensable and indispensable points of a starshaped set see Sect. 10.

Corollary 55

Let $S\ne {\mathbb {R}} ^{d}$ be a closed starshaped set. If $\mathrm {disp}S=\mathrm {bd}S$ (that is, S has no indispensable boundary points), then all radial functions of S are continuous.

Theorem 56

Let $S\ne {\mathbb {R}} ^{d}$ be a closed starshaped set such that $\mathrm {int}\ker S\ne \varnothing$. Then for each $m\in \mathrm {bd}\ker S$ there exists an open hemisphere $S_{m}^{d-1}$ of $S^{d-1}$ such that $r_{m,S}\left( S_{m}^{d-1}\right) =0$ if and only if S is convex.

Corollary 57

Let S be a closed starshaped set such that $\mathrm {int}\ker S\ne \varnothing$. Then S is non-convex if and only if there exists a point $m\in \mathrm {bd}\ker S$ such that the set $\left\{ u\in S^{d-1}:r_{m}(u)\ne 0\right\}$ contains a closed hemisphere of $S^{d-1}$.

For the case of $\Delta$-starshaped sets, a more general notion of radial function was introduced by Gardner [204]: for a nonempty compact set A which is $\Delta$-starshaped at 0 and $u\in S^{d-1}$, he introduced

$$\begin{aligned} \rho _{A}(u)=\sup \{\lambda \ :\ \lambda u\in A\}. \end{aligned}$$

This notion coincides with the radial function at 0 of the starshaped set $\left[ 0:A\right]$.

Definition 58

The reciprocal of the radial function at 0 is called the gauge function $g_{A}:{\mathbb {R}}^{d}\backslash \{0\}\rightarrow {\mathbb {R}}_{+}$, defined for $x\in {\mathbb {R}}^{d}\backslash \{0\}$ by

$$\begin{aligned} g_{A}(x)=\inf \left\{ \lambda :x\in \lambda A\right\} . \end{aligned}$$

Using their gauge functions, in [361] d-dimensional starshaped sets are investigated by means of their images in the space of all positively homogeneous continuous functionals, considered over the d-dimensional Euclidean space.

We now mention results of a more applied nature, namely in the sense that radial functions are taken as a tool to solve (mainly geometric) problems.

We start with questions from the field of geometric tomography, dealing with the retrieval of information on geometric objects from data concerning their projections onto and sections by lower dimensional flats. Further results from geometric tomography (not described in this section), e.g. related to the Busemann–Petty problem and the notions of intersection body, cross-section body, as well as centroid body , are presented in Sect. 17 below. Since we discuss such problems concerning shapes and also measures of sections of starshaped sets by flats, the reader should consult, as starting points for our respective discussions here (and also in Sect. 17 below), Chapters 6, 7, 8, and 9 of Gardner’s monograph [204]. An old result of Funk states that any starshaped body S having continuous radial function with respect to the origin 0 and centered at 0 is uniquely determined by its i-th section function (i.e., by the i-dimensional volumes of its sections with all i-dimensional subspaces, for some i from 1 to $d-1$). This is related to Lutwak’s dual Brunn–Minkowski theory (see [336, 337]), and the authors of [210] continued these investigations. They showed that, up to reflection at 0, no starshaped body S with continuous radial function, but without centeredness at 0, is determined in such a way. In addition, they proved also that the set of starshaped bodies that are determined (up to reflection at the origin) in this way is nowhere dense in their own family. In particular, also for $i=d-1$ the verified non-uniqueness holds if S is not centered at o. Inspired by this, in [55] it was proved that a compact body starshaped with respect to the origin is uniquely determined by the $(d-1)$-volumes and the centroids of its hyperplane sections through the origin. This result is mainly based on the proof that if the volumes and centroid functions of two convex bodies are close, then their radial functions are also close, where closeness is defined in terms of the $L_{2}$ norm. Funk’s section theorem can also be read like this: if K and L are origin-centered star bodies in Euclidean d-space whose i-dimensional central sections (for some i between 1 and $d-1$) all have equal volume, then $K=L$. Extending this, the authors of [343] verified the same result under the assumption that K is origin-centered and L is only starshaped with respect to the origin. The related paper [227] also considers origin-centered star bodies in ${\mathbb {R}} ^{d}$ and their radial functions. Let S be such a body, and let $\mathcal { H}$ denote an open connected set of hyperplanes through the origin covering ${\mathbb {R}}^{d}$. If for some $H_{0}$ from ${\mathcal {H}}$, S osculates a ball centered at the origin to infinite order along $H_{0}$, and S has constant cross-sectional areas when intersected by hyperplanes from ${\mathcal {H}}$, then S is that ball. These results are then extended to give conditions under which two origin-centered star bodies in d-space are equal (up to a set of measure zero). Also counterexamples are constructed to show that the infinite order osculation condition cannot be replaced by finite order conditions. Without the osculation hypothesis there are counterexamples, and similar results were proved for areas of projections (instead of section areas). The proofs are mainly based on properties of the radial function of S, Radon transforms, as well as cosine transforms. Using spherical harmonics, Groemer (see [229, 231] and [230, Sect. 5.6]) studied similar properties obtained with sections of star bodies with continuous radial function, and he also derived related stability results; see also [224]. A definition of star body which is more general than the usual one was given in [211], see also Sect. 0.7 of [204] and our Sect. 4: the body should intersect all lines through the star center in a line segment. With additional conditions on the radial function, also more general than the usually considered ones, a star body can even be disconnected. Numerous problems in the spirit of geometric tomography, e.g. about central sections of such bodies S, are studied in this setting. For example it is proved that if S is centrally symmetric and all i-dimensional sections through the center have equal i-dimensional volume, then S has to be a ball. On the other hand, the authors construct non-spherical examples with concurrent sections of equal i-volumes. If S is such a body (not centrally symmetric) with concurrent sections of constant i-volume for two different values of i, then S is a centered ball. Further results in this direction, again for star bodies with continuous radial function, were obtained in [210]. In [532] the radial function of two-dimensional starshaped sets having a uniform density function are used to answer tomographic questions on these sets. In particular, it is shown that such an object is uniquely determined by its parallel projections sampled over an angular range of 180 ${{}^\circ }$ with a detector that only covers an interior field-of-view, even if the density of the object is not known.

There are various results on pairs of convex bodies with homothetic and similar sections; the reader is referred to Sect. 7.1 of Gardner’s book [204]. A recent contribution in this direction regarding starshaped sets is [8]. In this paper the authors show that starshaped sets of dimensions $d\ge 4$ with directly congruent sections coincide up to translation and some special orthogonal transformation.

The paper [144] contains a characterization of balls among smooth bounded starshaped domains by the constancy of a function involving principal curvatures and the cut value of boundary points. This geometric result yields several applications to various symmetry questions for PDE’s. The author of [375] studied radial functions of compact sets in ${\mathbb {R}}^{d}$ starshaped with respect to the origin, which are used to describe the starshaped attainability of a differential inclusion; in this framework directional derivatives are investigated.

Petty [414] defined the boundary of a centroid body with respect to bounded measurable sets S of positive measure via the loci of centroids of mass-distributions in half-spaces, after symmetrizing S yielding a center z. It turns out that if S is strictly starshaped relative to z, then it is uniquely determined by its centroid surface. In [245], the authors brought centroid bodies into connection with the floating body problem. They studied planar bodies S starshaped at the origin which float in equilibrium at every position and have non-uniform density. Their boundary is parametrized by polar coordinates, and the centroids of their boundary arcs and respective polar regions, generated in half-planes bounded by lines through the origin, are also considered. It was shown that if the loci of these centroids are, in both cases, circles centered at the origin, then S floats in equilibrium at every position. Furthermore, two characterizations of the Euclidean disc were obtained, namely for the case when S is starshaped with respect to each of these centroids. Campi [123] investigated the reconstruction of sets S in ${\mathbb {R}}^{d}$, which are starshaped with respect to the origin, given the volumes of the intersections of S with half-spaces determined by $(d-1)$-dimensional subspaces. Rubin [440] derived an explicit inversion formula for the Radon-like transform that assigns to a function on the unit sphere the integrals of that function over hemispheres lying in lower-dimensional central cross-sections. The results are applied to determining star bodies from the volumes of their central half-sections.

Now we will discuss some geometric inequalities related to starshapedness. The Chernoff inequality, concerning the area and width function of planar convex curves and characterizing the circle as extremal curve, was somewhat dualized in [554]: Replacing some k-th order support function occuring in the more general Chernoff–Ou–Pan inequality by the dual k-order radial function, a somewhat “dual” inequality for starshaped curves in the plane is obtained which analogously contains the area of the enclosed region. Let $D_{1}$, $D_{2}$ be two concentric disks, and let S be a compact, planar set included in $D_{1}$ that is starshaped with respect to all points of $D_{2}$. Using radial functions, the authors of [516] found a sharp upper bound on the perimeter of S.

In [30] a very general form of the isodiametric inequality for measurable sets was obtained; the unit sphere of the linear space under consideration is the boundary of a compact body which is starshaped with respect to the origin.

For a compact starshaped set S in ${\mathbb {R}}^{d}$ having the origin as interior point of its kernel, the author of [536] derived the smallest Lipschitz constant for the radial projection of the unit sphere of $\mathbb {R }^{d}$ on the boundary of S. (This naturally continues the work of Toranzos [506].) He also found the least upper bound of the upper outer surface area (in the sense of Minkowski) of the boundary of a compact starshaped set contained in the unit ball and containing a concentric ball in its kernel.

14 Sums, unions, and intersections of starshaped sets

We will start with theorems on several types of “addition of starshaped sets”.

As a consequence of Remark 1 (see [69]), we infer that if $S_{1}$ and $S_{2}$ are starshaped, then so is $S_{1}+S_{2}$ (see also [391]). The analogue of this result for sets strongly starshaped at a and with closed Minkowski sum was proved in [461]. Generally, the Minkowski sum of closed starshaped sets need not be closed. For strongly starshaped sets Shveidel [461] proved the following

Theorem 59

Let $S_{1}$ and $S_{2}$ be closed sets strongly starshaped at 0. If there are no $a_{1},a_{2}$ and $x\ne 0$ such that $\left\{ a_{1}+tx:t\ge 0\right\} \subset S_{1}$ and $\left\{ a_{2}+tx:t\le 0\right\} \subset S_{2}$, then $S_{1}+S_{2}$ is closed.

The assumption of strong starshapedness is essential in the above theorem.

Remark 60

The Minkowski sum of sets $\Delta$-starshaped at a point a need not be $\Delta$-starshaped at a.

While the Minkowski addition has especially nice properties for compact convex sets (e.g., the cancellation law is satisfied), for star bodies the so called radial addition $\tilde{+}$ is more natural and is commonly used. It is defined by means of radial functions as follows:

$$\begin{aligned} \rho _{S_{1}\tilde{+}S_{2}}(u):=\rho _{S_{1}}(u)+\rho _{S_{2}}(u)\quad \mathrm {for\ \ every}\quad u\in S^{n-1}. \end{aligned}$$

It is easy to see that if $S_{1}$ and $S_{2}$ are star bodies, then $S_{1} \tilde{+}S_{2}$ is a star body. Moreover,

$$\begin{aligned} S_{1}\tilde{+}S_{2}\subset S_{2}+S_{2}. \end{aligned}$$

For the radial sum of star bodies, Lutwak proved (in [337]) the following theorem, called the dual Brunn–Minkowski inequality, which is an analogue of the Brunn–Minkowski inequality for convex bodies (compare with Sect. 6.1 in [453]).

Theorem 61

Let $S_{1}$ and $S_{2}$ be star bodies in ${\mathbb {R}}^{d}$. Let $0\in \ker S_{i}$ and assume that $\rho _{S_{i}}$ is continuous on its support for $i=1,2$. Then

$$\begin{aligned} V_{d}(S_{1}\tilde{+}S_{2})^{1/d}\le V_{d}(S_{1})^{1/d}+V_{d}(S_{2})^{1/d}. \end{aligned}$$

Equality holds if and only if either $d=1$ or $d\ge 2$ and $S_{2}=\lambda S_{1}$ for some $\lambda >0$.

We mention that this section is also related to Sect. 17 below. For a set S and a hyperplane H, the Minkowski symmetral of S at H is the set $S+T$, where T denotes the reflection of S at H. Clearly, $S+T$ is symmetric with respect to the hyperplane H. It is known that successive Minkowski symmetrizations of compact, convex sets in different suitable directions yield a sequence of sets that converges, in some sense, to the Euclidean ball. In [188] the authors derived upper bounds for the number of Minkowski symmetrizations which are necessary to transform a starshaped set in ${\mathbb {R}}^{d}$ into another one which, in the Hausdorff metric, is arbitrarily close to the Euclidean ball. Further related results given in [188] treat starshaped sets of given mean width. The paper [531] contains a result on the interior of the Minkowski sum of two starshaped sets compared with the Minkowski sum of their interiors.

Given two sets starshaped with respect to the origin, one can consider a natural addition of points of both sets lying on the same ray emanating from the origin. Based on this type of addition, Gardner [205] proved a dual Brunn–Minkowski inequality, with equality iff these sets are homothets modulo a set of measure zero.

Similarly, a “dual” of the Orlicz–Brunn–Minkowski theory (an important extension of the classical Brunn–Minkowski theory) was developed. It is well known that Orlicz spaces are Banach spaces of measurable functions which generalize $L_{p}$ spaces (for $L_{p}$ spaces see our Sect. 17). In recent years, certain types of results on convexity and related fields were extended to Orlicz spaces. In particular, the Orlicz–Brunn–Minkowski theory represents a generalization of the $L_{p}$-Brunn–Minkowski theory analogously to the way how Orlicz spaces generalize $L_{p}$ spaces, see our discussion of $L_{p}$ intersection bodies in Sect. 17 below. The Orlicz–Brunn–Minkowski theory was introduced in [206, 341] established the Orlicz–Brunn–Minkowski theory by defining the useful notion of Orlicz addition; this paper contains many results on operations between compact convex and compact starshaped sets. Independently, [541] introduced the Orlicz addition of convex bodies containing the origin in their interiors and obtained the Orlicz–Brunn–Minkowski inequality. In [281, 561] Orlicz dual mixed volumes and the harmonic Orlicz sum of star bodies were studied, yielding interesting properties of the Orlicz harmonic combination and the harmonic Orlicz addition version of the Brunn–Minkowski inequality. The dual theory of the Orlicz–Brunn–Minkowski theory, concerning star bodies, was introduced in [208, 562], together with the key notion of Orlicz radial sum of two star bodies. This is based on their radial functions and has, for example, the $L_{p}$ harmonic radial sum as subcase. With this tool, the authors established the dual Orlicz–Minkowski inequality and the dual Orlicz–Brunn–Minkowski inequality for star bodies. Moreover, the equivalence between these two important inequalities was also shown. The authors of [207] introduced more general notions of volume and curvature of star bodies that include many previously considered types of dual mixed volumes and dual curvature types, in particular also a new general dual Orlicz-type curvature measure. They established general variational formulas for such general volumes of two Orlicz-type linear combinations. One of them yields a new dual Orlicz–Brunn–Minkowski inequality, dual Orlicz–Minkowski-type inequalities and uniqueness results for star bodies.

We shall now deal with unions and intersections of starshaped sets. In general, neither the union nor the intersection of starshaped sets is starshaped. However, the following statements (which are Helly-type theorems) are true; see [54].

Theorem 62

Let $S_{t}$ be a starshaped compact set in ${\mathbb {R}}^{d}$, for $t\in T$, where the cardinality card $T\ge d+1$. Assume that, for every subfamily $\left\{ S_{t}:t\in T_{0}\right\}$ with card $T_{0}\le d+1$, the set $\bigcap \nolimits _{t\in T_{0}}S_{t}$ is starshaped. Then $\bigcap \nolimits _{t \in T}S_{t}$ is starshaped.

The dual theorem holds, too.

Theorem 63

Let $S_{t}$ be a starshaped compact set in ${\mathbb {R}}^{d}$, for $t\in T$, where card $T\ge d+1$. Assume that, for every subfamily $\left\{ S_{t}:t\in T_{0}\right\}$ with card $T_{0}\le d+1$, the set $\bigcup \nolimits _{t\in T_{0}}S_{t}$ is starshaped. Then $\bigcup \nolimits _{t \in T}S_{t}$ is starshaped.

Remark 64

The intersection of a decreasing sequence of compact starshaped subsets of ${\mathbb {R}}^{d}$ is starshaped (see [35, 125]). But the union of an increasing sequence of starshaped subsets of ${\mathbb {R}}^{d}$ need not be starshaped (see [259]).

In [29] the authors considered nested (decreasing and increasing) sequences of starshaped sets in Banach spaces. The intersection, if decreasing, and the closure of the union, if increasing, were studied in view of the preservation of these properties, and related results for starshaped sets in reflexive spaces were obtained.

Let us now concentrate on families of sets starshaped at a point. The following statement is known and obvious: if $S_{t}$ is starshaped at a for every $t\in T$, then $\bigcup \nolimits _{t\in T}S_{t}$ and $\bigcap \nolimits _{t\in T}S_{t}$ are starshaped at a.

Analogous results are valid for finite intersections and finite unions of strongly starshaped sets and for arbitrary intersections and finite unions of $\Delta$-starshaped sets (compare [442, Proposition 2.3], [69, 2.9], and [390, Proposition 14.1.7]). Moreover, let $S_{i}$ be bodies starshaped at 0, for $i=1,\ldots ,k$, and let $S=\bigcup \nolimits _{i=1}^{k}S_{i}$ and $S_{0}=\bigcap \nolimits _{i=1}^{k}S_{i}$. Then S and $S_{0}$ are bodies starshaped at 0, and for every $u\in S^{d-1}$

$$\begin{aligned} \rho _{S}(u)=\max _{i=1,\ldots ,k}\rho _{S_{i}}(u),\ \ \rho _{S_{0}}(u)=\min _{i=1,\ldots ,k}\rho _{S_{i}}(u) \end{aligned}$$

(see [390]).

The following problem, closely related to the classical ”Art Gallery Problem” stated by Victor Klee many years ago (cf. the discussion in our Sect. 19, the part on discrete and computational geometry), was open for a long time (see [514]).

Problem 65

What geometric conditions are sufficient, for a set $X\subset {\mathbb {R}}^{d}$, to be a finite union of k starshaped subsets?

Partial solutions, for various k and d, were given by several authors, in particular for $k=2$ and arbitrary d (see [102, 303]). Some results in the plane can be found in [94, 95, 103, 111], and solutions for arbitrary k and d were given in [514].

The paper [100] contains characterizations of a compact set S in ${\mathbb {R}}^{d}$ that is a finite union of starshaped sets, using sequences of certain compact sets converging to S.

Hare and Kenelly [259] showed that the intersection of the maximal (with respect to inclusion) starshaped subsets of a compact, simply connected set in the plane is starshaped or empty. In [499] it was shown that if the intersection I of all maximally inclusive starshaped subsets of a compact set M is nonempty, then there exists a maximally inclusive star-like subset S of M whose kernel is contained in the kernel of I. Continuing this, the same author proved in [503] the following (we use the same notation): Let T be a triangle with the property that if at least two of its vertices are contained in I, then all its edges are contained in the given compact set M. Then I is starshaped or empty iff any such triangle T lies in M. From this, the result of Hare and Kenelly [259] follows. Analogous results for planar $L_{n}$-starshaped sets M (having a point x such that any point can be joined with x by a broken line within M, which consists of at most n segments), and c- and d-starshaped sets (see our Sect. 6) are proved in [498, 500], respectively. In the latter paper, also sufficient conditions are given under which a closed set is the union of two of its c-starshaped or d-starshaped subsets. And in [502] sufficient conditions for finite unions of d-convex, d-starshaped, and $L_{n}$-starshaped sets are given.

A compact set S has property P if there is a line l such that each triple of points x, y, z in S determines a point p on l for which at least two of the segments [x : p], [y : p], and [z : p] are from S. If S is the union of two starshaped sets, then it has property P. Valentine [527, p. 178, problem 6.6] suggested that property P might characterize unions of two starshaped sets, but Larman [323] found a counterexample and gave an additional condition yielding the wanted characterization (see also Koch and Marr [303]).

In [86, 111] unions of two starshaped sets in the plane are characterized with the help of the notion of clear visibility; some results fail when “clear visibility” is replaced by usual “visibility”. Also the paper [505], dealing with d-starshapedness, is related to this type of results.

A compact set S in the plane is called staircase connected if every two points from S can be connected by an x-monotone and y-monotone polygonal path whose sides are all parallel to the coordinate axes. In [344], staircase analogues of stars and kernels were introduced and investigated, and in [345] it was proved that if the so-called staircase k-kernel (this k describes minimal edge numbers of connecting staircase paths) is not empty, then it can be expressed as the intersection of a covering family of maximal subsets of staircase diameter k of S.

Starshaped and so-called co-starshaped sets can be represented as intersections of finite unions of closed half-spaces and are essential in optimization theory. Shveidel [463] derived conditions under which sets are starshaped or (strongly) co-starshaped: e.g., a finite union of closed half-spaces is strongly co-starshaped if and only if it differs from ${\mathbb {R}}^{d}$. He also obtained properties of associated kernels or co-kernels, conditions for finite unions of starshaped sets to be starshaped, and convex sets to be co-starshaped, as well as a necessary and sufficient condition for a polyhedral set to be co-starshaped.

Breen [99] examines how intersections of a given set S with various flats will yield conclusions about the starshapedness of S. A sufficient condition for a compact set S to be a union of m starshaped sets is derived, and this yields the following characterization of compact starshaped sets: Let S be a compact set in Euclidean d-space, $p\in S$ be fixed, and k be a fixed integer between 1 and d. Then S is starshaped with p as a point from its kernel if and only if, for every k-dimensional flat F passing through p, the intersection of S and F is starshaped. A little bit different is the following result which, however, is also based on intersections with flats. Namely, Tamássy [489] derived (generalizing a result of P. Funk) the following characterization of balls in 3-space among all starshaped bodies C centered at the origin and having smooth boundary. Let $\epsilon >0$, Q be a fixed plane, and $Q^{*}$ be the set of planes passing through the origin whose normals make an angle less than $\epsilon$ with Q. If the intersection of C with each plane from $Q^{*}$ has area $\pi$, then C is the unit ball.

15 Spaces of starshaped sets

Several metrics have been studied on the family ${\mathcal {X}}$ of starshaped compact subsets of ${\mathbb {R}}^{d}$. Of course, since ${\mathcal {X}}$ consists of nonempty compact subsets of ${\mathbb {R}}^{d}$, one can use the Hausdorff metric $\rho _{H}$ to measure distances in this family. For $({\mathcal {X}},\rho _{H})$, Hirose proved a counterpart of the Blaschke Selection Theorem (see [265]). The same result (in terms of gauge functions) was obtained by Beer (see [33]), and [161] contains another approach. In [508] it was proved that compact starshaped sets can be uniformly approximated (in the sense of the Hausdorff metric) by starshaped polytopes and by starshaped smooth sets. A set S in ${\mathbb {R}}^{d}$ is said to be m-starshaped if there is a subset M of S with non-empty interior such that each pair of points $x\in M$, $y\in S$ can be joined by a polygonal line in S having at most m segments. In [171] a convergence theorem of Blaschke type is proved for m-starshaped sets. Inspired by these results, Spiegel [473] showed the completeness of the space of certain more general compact sets, in the sense of the Hausdorff metric.

Another well known and frequently used metric is the radial metric $\delta$ (see, e.g., [470], Sect. 0.7 in [204], and Sect. 14.3 in [388]): if $A_{1},A_{2}$ are starshaped with respect to 0, then

$$\begin{aligned} \delta (A_{1},A_{2})=\sup _{u\in \mathrm {S}^{n-1}}|\rho _{A_{1}}(u)-\rho _{A_{2}}(u)|. \end{aligned}$$

For subsets of ${\mathbb {R}}^{d}$ starshaped at 0, the radial metric is topologically stronger than the Hausdorff metric, i.e., any sequence convergent with respect to the radial metric $\delta$ is convergent with respect to the Hausdorff metric $\rho _{H}$, but the converse implication does not generally hold. However, the following result was proved in [469], where, for any $r>0$,

$$\begin{aligned} {\mathcal {S}}^{d}(r):=\{A\in {\mathcal {S}}^{d}\ |\ rB^{d}\subset \ker A\}. \end{aligned}$$

Theorem 66

For any $r>0$, the radial metric and the Hausdorff metric are topologically equivalent in ${\mathcal {S}}^{d}(r)$.

Since a disadvantage of the radial metric is its (direct or indirect) dependence on 0, some other metrics for the family ${\mathcal {S}}^{d}$ were introduced in [390, 469]. The following metric $\delta _{st}$ was defined in [390]: for $A_{1},A_{2}\in {\mathcal {S}}^{d}$, let

$$\begin{aligned} \mathbf {\delta }(A_{1},A_{2}):=\sup _{x_{1}\in \ker A_{1}}\inf _{x_{2}\in \ker A_{2}}\delta (A_{1}-x_{1},A_{2}-x_{2}) \end{aligned}$$

and

$$\begin{aligned} \delta _{st}(A_{1},A_{2}):=\max \{\mathbf {\delta }(A_{1},A_{2}),\mathbf {\delta } (A_{2},A_{1})\}+\rho _{H}(\ker A_{1},\ker A_{2}). \end{aligned}$$

The function $\delta _{st}$ is a metric on ${\mathcal {S}}^{d}$ (see [388, Theorem 14.4.2]). It is called the star metric.

Herburt [262] showed that the operation of taking convex hulls is, for some classes of compact starshaped sets in ${\mathbb {R}}^{d}$, not (Lipschitz) continuous with respect to the radial metric and the star metric.

Closely related to these metrics and interesting for applications in nonsmooth optimization are the investigations on spaces of starshaped sets presented in [445, 446]. Based on the isomorphism between the space of starshaped sets considered there (see again [445]) and the space of continuous, positively homogeneous real-valued functions, in [404] the starshaped differential of a directionally differentiable function was defined. The authors obtained formulae for starshaped differentials of pointwise maxima and minima of a finite number of directionally differentiable functions, investigated compositions of them, and derived the respective mean-value theorem.

Quasidifferentials are the topic of the monograph [157]. In the paper [558] notions from quasidifferential analysis are combined with starshapedness. A quasidifferential of a quasidifferential function over $x\in {\mathbb {R}}^{d}$ consists of two compact sets, and the authors successfully introduced the notion of star-differential represented by a pair of starshaped sets.

Sójka introduced and studied other metrics for families of star bodies in [470]. He used selectors for kernels to define these new metrics. Let us recall that a selector for the family ${\mathcal {K}}^{n}$ is a function $\phi :{\mathcal {K}}^{n}\rightarrow {\mathbb {R}}^{n}$ such that $\phi (A)\in A$ for every $A\in {\mathcal {K}}^{n}$. Let $\phi$ be a selector for ${\mathcal {K}}^{n}$. Then, for $A_{1},A_{2}\in {\mathcal {S}}^{n}$, the functions $\delta _{\phi }$ and $\delta _{\phi \ker }$ are defined as follows:

$$\begin{aligned} \delta _{\phi }(A_{1},A_{2}):=\delta (A_{1}-\phi (\ker A_{1}),A_{2}-\phi (\ker A_{2}))+||\phi (\ker A_{1})-\phi (\ker A_{2})||, \\ \delta _{\phi \ker }(A_{1},A_{2}):=\delta (A_{1}-\phi (\ker A_{1}),A_{2}-\phi (\ker A_{2}))+\rho _{H}(\ker A_{1},\ker A_{2}). \end{aligned}$$

These two functions are metrics (see [470, Proposition 4.3]). Moreover, Sójka introduced one more metric, $\delta _{\phi }^{L}$, defined in terms of selectors and the Lipschitz constant of the difference of radial functions of two star bodies.

Only for the metric $\rho _{H}$ is the correspoding hyperspace separable. However, the subclass of star bodies whose kernels have nonempty interior is separable for the metrics $\rho _{H}$, $\delta$, $\delta _{\phi }$, and $\delta _{\phi \ker }$.

Several results concerning operations on starshaped sets are known, in particular on their continuity we have:

• (i) The function $\ker :{\mathcal {S}}^{d}\rightarrow {\mathcal {K}}^{d}$ is continuous with respect to $\delta _{st}$ and $\delta _{\phi \ker }$, but it is not continuous with respect to $\rho _{H},\delta ,\delta _{\phi },\delta _{\phi }^{L}$. But for the subclass of star bodies whose kernel has nonempty interior the function $\ker$ is continuous also with respect to the metric $\delta _{\phi }^{L}$ (see [470]).

• (ii) As is well known, the function $\mathrm {conv}$ is continuous on the class ${\mathcal {S}}^{d}$ with the Hausdorff metric. However, as it was proved in [263], it is not continuous for the radial metric and for the star metric. But for the subclass consisting of star bodies with kernels contained in the interior, the function $\mathrm {conv}$ is continuous with respect to the radial metric (see [263]).

Sójka [469] characterized homeomorphic embeddings of ${\mathbb {R}} ^{d}$ into itself preserving the class of bodies starshaped at 0.

To finish this section, let us mention some papers concerning generic properties of a hyperspace of compact starshaped sets. In [237, 550, 551] some generic properties of compact starshaped sets with respect to the Hausdorff metric were derived. For instance, most compact starshaped sets X are nowhere dense, have a one-point kernel a, and have a dense set of directions determined by a segment $\left[ a:x \right]$, for $x\in X$. Similar questions for the metrics $\rho _{H}$ and $\delta$ are considered in [393].

16 Selectors for star bodies

As mentioned in the preceding section, selectors for ${\mathcal {K}}_{0}^{d}$ may be used to define a new metric for the hyperspace ${\mathcal {S}}^{d}$ of star bodies in ${\mathbb {R}}^{d}$ (see [470]). On the other hand, it is natural to extend any selector $s:{\mathcal {K}}_{0}^{d}\rightarrow {\mathbb {R}} ^{d}$ to a selector for ${\mathcal {S}}^{d}$, that is, to a function ${\bar{s}}: {\mathcal {S}}^{d}\rightarrow {\mathbb {R}}^{d}$ satisfying the condition ${\bar{s}} (A)\in \ker A$ for every $A\in {\mathcal {S}}^{d}$. Moszyńska (see [390]) defined, for any selector $s:{\mathcal {K}}^{d}\rightarrow {\mathbb {R}}{ ^{d}}$, the extension ${\bar{s}}:{\mathcal {S}}^{d}\rightarrow {\mathbb {R}}^{d}$ as follows: for any $A\in {\mathcal {S}}^{d}$

$$\begin{aligned} {\bar{s}}(A):=\xi _{\ker A}(s(\mathrm {conv}A)), \end{aligned}$$

where $\xi _{\ker A}$ is the metric projection on $\ker A$ (i.e., it is ”the nearest point map”; compare with Theorem 5.1 in [390]).

Another idea is to extend selectors for ${\mathcal {K}}_{0}^{d}$ over some class of star bodies, namely, over the class ${\mathcal {T}}^{d}\subset {\mathcal {S}}^{d}$ whose members satisfy the following condition: there exists a subset $S_{0}$ of the unit sphere $S^{d-1}$ such that

• $S_{0}$ has spherical measure zero,

• the function $\ker A\ni x\mapsto \rho _{A-x}(u)$ is continuous for $u\in S^{d-1}\backslash S_{0}$.

For properties of ${\mathcal {T}}^{d}$ see Proposition 2.5 in [390]. Another family of selectors for ${\mathcal {T}}^{d}$, called the radial center map, is discussed in the same paper.

17 Star duality, intersection bodies, and related topics

This part of our paper is also related to some topics discussed in Sects. 13 and 14. As the classical Brunn–Minkowski theory (see [453]) is also connected with projections of convex bodies, the dual Brunn–Minkowski theory (cf. the early works [336, 337]) mainly concerns intersections with subspaces, replacing convex bodies with starshaped sets, support functions with radial functions, mixed volumes with dual mixed volumes, and also showing the connections between projection bodies and intersection bodies. Especially we underline that the notion of dual mixed volume (see again [336, 337] as well as “Appendix A” in Gardner’s book [204]) presents an important tool for the development of the dual Brunn–Minkowski theory and the theory of intersection bodies. Thus, in the dual Brunn–Minkowski theory one usually works with starshaped bodies in ${\mathbb {R}}^{d}$ having (interior points and) continuous radial functions with respect to the origin; these sets are endowed with the Hausdorff topology induced by the uniform convergence for radial functions. As already mentioned (see Sect. 13), in [211] this theory was successfully extended to a larger class of sets, but is still based on starshapedness. Since all these, and many more recent, developments reflect a deeper type of duality, we put duality and intersection bodies in the same section. Thus, let us now concentrate on ”star counterparts” of two notions that are well known for convex bodies: the polar dual, or polar, of a convex body (or of an arbitrary nonempty set), and the projection body of a convex body.

First let us consider polar duality, or polarity. For the case of convex bodies we refer to Sect. 1.6 of [453] and Chapter 13 of [388]. For the general case, if $A\subset {\mathbb {R}}^{d}$ is a nonempty set, the polar to A is the set $A^{*}$ defined by the formula

$$\begin{aligned} A^{*}:=\{x\in {\mathbb {R}}^{d}\ :\ \forall a\in A\ \ x\circ a\le 1\}, \end{aligned}$$

where $\circ$ is the scalar product in ${\mathbb {R}}^{d}$ (see [388, Definition 13.2.1]). The function $A\mapsto A^{*}$ is called polar duality or polarity.

Let us mention basic properties of polarity for convex bodies with 0 in their interior.

• If $A\in {\mathcal {K}}_{0}^{d}$ and $0\in \mathrm {int}A$, then $A^{**}=A$, i.e., polarity is an involution.

• Polarity restricted to convex bodies with 0 in their interior is continuous with respect to $\rho _{H}$ and satisfies the condition

  $$\begin{aligned} h_{A}={\frac{1}{\rho _{A}}}, \end{aligned}$$

  where $h_{A}$ is the support function of the convex body A and $\rho _{A}$ is its radial function.

The “star counterpart” of polarity is the following.

Let ${\mathcal {S}}_{+}^{n}$ be the class of star bodies in ${\mathbb {R}}{^{d}}$ with 0 in the kernel, and let $i:{\mathbb {R}}{^{d}\backslash } \{0\}\rightarrow {\mathbb {R}}{^{d}\backslash }\{0\}$ be the inversion with respect to the unit sphere $S^{d-1}$:

$$\begin{aligned} i(x):={\frac{x}{\Vert x\Vert ^{2}}}. \end{aligned}$$

Then, for every $A\in {\mathcal {S}}_{+}^{d}$, the star dual of A is defined by the formula

$$\begin{aligned} A^{o}:=\mathrm {cl}({\mathbb {R}}{^{d}\backslash }i(A)) \end{aligned}$$

(compare Definition 3.2 in [389]).

According to Proposition 3.3 in [389], for any star body A with 0 in its interior

$$\begin{aligned} \rho _{A^{o}}(u)={\frac{1}{\rho _{A}(u)},\ }\mathrm {for\ every}\ u\in S^{d-1}. \end{aligned}$$

The star duality $A\mapsto A^{o}$ is an involution that reverses inclusion (see [388], 15.4.3, or Theorem 3.4 in [389]). Its relations to polar duality are the following: for any $A\in {\mathcal {K}}^{n}$ with $0\in \mathrm {int}A$,

• $A^{*}\subset A^{o}$,

• $A^{o}=A^{*}$ if and only if $A=\alpha B^{d}$ for some $\alpha >0$ (where $B^{d}$ is the unit ball).

Remark 67

Let us note that in [389] the approach is categorical, while here (as it was done also in [388]) we restrict our considerations to objects only.

As it was said at the beginning of this section, the second “star counterpart” that we are going to recall is the notion of projection body of a convex body (see Chapter 4 of [204], Sects. 5.3 and 7.4 of [453], and Sect. 15.2 of [388]).

For any convex body A in ${\mathbb {R}}^{d}$, its projection body $\Pi A$ is the convex body defined by the condition

$$\begin{aligned} \forall u\in S^{d-1},\ h(\Pi A,u)=V_{d-1}(\pi _{u^{\perp }}(A)), \end{aligned}$$

where $V_{d-1}(\pi _{u^{\perp }}(A))$ denotes the $(d-1)$-volume of the orthogonal projection of A onto a hyperplane with u as normal vector. In 1964, Shephard asked the following question, usually referred to as the Shephard problem (compare [337]). Is it true for any two convex bodies A, B in ${\mathbb {R}}^{d}$, both symmetric with respect to 0, that if for every hyperplane H

$$\begin{aligned} V_{d-1}(\pi _{H}(A))<V_{d-1}(\pi _{H}(B)), \end{aligned}$$

then also $V_{d}(A)<V_{d}(B)$?

In 1967, Petty and Schneider (see, e.g., [454]) independently proved that the implication is not generally true, but that it is true under the assumption that B is a projection body.

We are interested in the notion of intersection body of a star body, the ”star counterpart” of the notion of projection body. It was introduced by Lutwak in [337] when studying the class ${\mathcal {S}}_{1}^{d}$ of star bodies with 0 in the kernel and the radial function being continuous.

For any $A\in {\mathcal {S}}_{1}^{d}$, its intersection body, IA, is the star body with 0 in the kernel and radial function defined by the formula

$$\begin{aligned} \rho _{IA}(u):=V_{d-1}(A\cap u^{\perp })\ \ \mathrm {for\ every\ }u\in S^{d-1} \end{aligned}$$

(see 15.3.1 in [388]).

In view of 15.3.2 in [388],

$$\begin{aligned} A\in {\mathcal {S}}_{1}^{d}\Longleftrightarrow IA\in {\mathcal {S}}_{1}^{d}. \end{aligned}$$

In 1956, Busemann and Petty asked the following question, usually referred to as the Busemann–Petty problem (see [120], where one can also find nine further related problems, all of them still unsettled).

Is it true for any two convex bodies A and B, both centered at 0,  that if for every hyperplane $H\ni 0$

$$\begin{aligned} V_{d-1}(A\cap H)\le V_{d-1}(B\cap H), \end{aligned}$$

then also $V_{d}(A)\le V_{d}(B)$?

Lutwak proved that the answer to this question is generally negative (see [337]), but that it is positive under the assumption that A is the intersection body of a star body (see also [388, Theorem 15.3.5]). More precisely, Lutwak [337] observed that, in ${\mathbb {R}}^{d}$, the Busemann–Petty problem has a positive answer iff every centrally symmetric convex body is an intersection body, meaning that its radial function is the spherical Radon transform of a nonnegative measure on the unit sphere. The general answers to the Busemann–Petty problem (negative for all $d>3$, and otherwise positive) were first given by different authors in several steps, with different methods and fascinating ideas (see [203, 556], and the references there). Furthermore, the Busemann–Petty problem could be reduced to that of the non-existence of certain intersection bodies, see, e.g., [202, 556]. Koldobsky [304] proved that a symmetric star body is an intersection body iff its radial function is a positive definite distribution on ${\mathbb {R}}^{d}$. Based on this, the authors of [209] derived a formula connecting the derivatives of parallel section functions of a symmetric star body S in ${\mathbb {R}} ^{d}$ with the Fourier transform of powers of the radial function of S. This made it possible to get a unified analytical approach to the Busemann–Petty problem, also clearly explaining the reasons for special situations in certain dimensions. Another unified approach to the (affirmative part of the) Busemann–Petty problem, namely via spherical harmonics, was given in [305], and meanwhile this problem is also studied in spherical and hyperbolic spaces (see [264, 547]). There are many further extensions and generalizations of the Busemann–Petty problem which also go beyond the scope of our survey. However, for tools and methods which are extremely useful for solving problems of the type discussed here the reader should consult the monographs [307, 310] as well as the exposition [306], and important further papers cited therein. These publications also show the power of Fourier transforms and harmonic analysis for studying sections of star bodies; in [308] related stability problems are presented.

The study of intersection bodies (of convex bodies and starshaped bodies) is based on radial functions and the notion of starshapedness, and some of their properties are nicely discussed and presented in Chapter 8 of Gardner’s monograph [204], see also Chapter 10 of Schneider’s monograph [453]. They are closely related to Busemann’s theorem (establishing their convexity for given convex bodies centered at the origin, see Sect. 8.1 from [204]) and the Busemann–Petty problem (see again Chapter 8 of [204]). Intersection bodies also play an essential role in Minkowski geometry (i.e., in the geometry of finite dimensional real Banach spaces): the isoperimetrix of a Minkowski space is, when the Busemann definition of area is used, the polar of the intersection body of its unit ball, clearly centered at the origin (and for the Holmes–Thompson definition of area, “dual” projection bodies play a similar role, since the isoperimetrix then is the projection body of the polar of the unit ball). A broad representation of this application is given in Chapter 5 of [492]. Further geometric properties and applications of intersection bodies, which also help to understand the “dual” role that they play with respect to projection bodies (i.e., the class of zonoids centered at the origin), can be found in the following papers and the references given therein. Inspired by Koldobsky’s result that the cross polytope is an intersection body, Zhang [557] showed that no origin-symmetric convex polytope in ${\mathbb {R}}^{d}$ ($d>3$) is the intersection body of a star body. The same author gave interesting charactzerizations of intersection bodies via dual mixed volumes, see [555]. Interesting relationships between the notion of star duality and that of intersection body can be found in [389]. In [222] the authors describe intersection bodies via functional analysis. They characterize a convex cone of a locally convex Hausdorff topological vector space by a dense subset of the dual space. The class of intersection bodies is identified with a closed convex cone of the dual space of even signed measures on the unit sphere. By using the dense set of differences of continuous positive functions, intersection bodies are then characterized in terms of inequalities involving dual mixed volumes. And in [223] the following is shown: A star body having continuous radial function with respect to its center, the origin, is an intersection body iff it is the limit of radial sums of ellipsoidal bodies in the topology of uniform convergence of radial functions over the unit sphere; for this characterization an extended definition of intersection bodies is needed. In [228] also star bodies in ${\mathbb {R}}^{d}$ are studied using Radon transforms on Grassmann manifolds, cosine transforms on the unit sphere, and convolutions on the rotation group of ${\mathbb {R}}^{d}$. Dual mixed volumes are used to characterize generalized intersection bodies. Further extensions of intersection bodies concern the concepts of mixed intersection bodies (created by finite families of star bodies, see [333, 560]) and complex intersection bodies (i.e., their analogues constructed in complex vector spaces, cf. [309]). Finally for this part, we repeat that the important tool of dual mixed volumes is nicely discussed in “Appendix A” of Gardner’s book [204].

The notion of intersection bodies was extended in many directions, e.g., concerning so-called mixed intersection bodies and (non-symmetric) $L_{p}$ intersection bodies. (It is well known that $L_{p}$ spaces are function spaces defined by using a natural generalization of the $l_{p}$-norm for finite-dimensional vector spaces; they are important in the local theory of Banach spaces, but also for theoretical discussions of problems in physics, engineering, statistics and so on.) E.g., for $L_{p}$ intersection bodies we refer to [49, 248,249,250, 339, 547, 548]. For example, in the paper [49] it was proved that the $L_{p}$ intersection body of a centered convex body is a convex body, whereas [248, 547] deal with the corresponding extension of the Busemann–Petty problem. There are many deep recent results in this direction and also concerning Orlicz-type extensions. The authors find it impossible to reflect all this stormy development here.

Another notion, strongly related to intersection bodies, is that of the cross-section body CK of a given convex body K in ${\mathbb {R}}^{d}$ (see [366] and Sect. 8.3 in [204]). Namely, the starshaped set CK is the symmetric body whose continuous radial function in direction u is the maximal volume of a hyperplane section of K orthogonal to u. (Thus, CK can also be defined as the union of all intersection bodies of all translates of K.) In ${\mathbb {R}}^{2}$, CK is clearly a convex set, and the same holds in ${\mathbb {R}}^{3}$, as shown by Meyer [380]. Brehm [112] found couterexamples for any $d>3$; thus there are non-convex cross-section bodies in any dimension larger than 3. Interesting problems related to cross-section and intersection bodies are collected in [366] and Sect. 8.3 of [204], and further results in this direction are given in [356,357,358].

Intersection bodies, projection bodies, and cross-section bodies play an essential role in geometric tomography (see the monograph [204] and our Sect. 13, where further results, mainly related to the tool of radial functions, are discussed). This is the field which also the following (types of) results refer to. In [359], it was shown by Makai, Martini and Ódor that a convex body K in d-dimensional Euclidean space is centered if all $(d-1)$-volume functions of parallel hyperplane sections have a critical value at the origin. Now let K be a starshaped body, and let the hyperplanes be replaced by circular hypercones with centers at the origin. In [449] it was shown that K is centered if all the $(d-1)$-volume functions of intersections with K have critical values when the cone degenerates to a hyperplane. The proof uses Fourier transforms of distributions, and the paper also gives a new proof of the theorem of Makai, Martini, and Ódor for star bodies. In their second paper [360] on this topic, Makai, Martini, and Ódor gave an easier approach to this. The authors of [396] prove several characterizations of balls by symmmetry properties of their sections or projections. E.g., they call a star body K in d-dimensional space completely symmetric if it has its centroid at the origin and if every ellipsoid whose symmetry group contains that of K must be a ball. If K is a star body with a continuous radial function in d-space, $d>2$, whose central sections are all completely symmetric, then K is a centered Euclidean ball. The paper [448] gives an overview on questions dealing with the unique determination of convex or starshaped bodies with projections or sections having some symmetries, such as bodies with congruent projections and sections, translations only, directly congruent projections, and other groups of symmetries.

The notion of centroid body of a given (convex or) star body was introduced by Petty (see [414] and also our Sect. 13 for further related tesults) and became a useful tool regarding affine isoperimetric inequalities and various other problems in geometric convexity; like projection bodies, also centroid bodies are zonoids, i.e., limits of vector sums of segments. The name “centroid bodies” comes from the fact that for the subcase of a given centered convex body K the boundary points of the corresponding centroid body give the locus of the centroids of the halves of K obtained by cutting K with hyperplanes passing through the origin. An early reference is [338] in which (among other things) a survey on dual mixed volumes of starshaped bodies with positive continuous radial functions and the relations between centroid bodies and projection bodies was given. In [339], $L_{p}$ analogues of centroid bodies were defined and used to extend analogously the related Busemann–Petty centroid inequality (which proves that the ratio of the volume of the given body to that of its centroid body is maximal precisely for ellipsoids). Further on, in [342] this notion was extended to Orlicz centroid bodies of star bodies (see our Sect. 14). The corresponding analogue of the Busemann–Petty centroid inequality was established for convex bodies, and the authors conjectured that this inequality can be extended to star bodies and that the ellipsoid is again extremal (as proved for convex bodies) among all star bodies with respect to the origin. In [563] the Orlicz centroid inequality for convex bodies was indeed extended to star bodies, and in [539] the authors established equality conditions for the Orlicz centroid inequality for certain types of star bodies; one equality condition generalizes Zhu’s equality condition and yields again, as expected, ellipsoids as extremal bodies. In the papers [249, 250], polar $L_{p}$ centroid bodies were studied within the framework of valuations, e.g. also regarding their relation to $L_{p}$ intersection bodies.

A function Z on a space of convex bodies, or star bodies (or other subsets of ${\mathbb {R}}^{d}$) is a valuation if $ZK+ZL=Z(K\cup L)+Z(K\cap L)$ holds, whenever $K,L,K\cup L$ and $K\cap L$ are still from that space. E.g., the paper [334] shows clearly that there are deep connections between intersection bodies (see above) and valuations, where intersection bodies of polytopes play an essential role. In [293] the author found an approach to homogeneous valuations on sets starshaped with respect to the origin of ${\mathbb {R}}^{d}$, with an $L_{p}$-function as radial function. Particularly, also the rotation invariant ones are investigated. First it is shown that dual mixed volumes can be defined so that they possess the same basic properties as usual mixed volumes, and then all continuous star valuations which are homogeneous with respect to dilatations are classified. These results yield a characterization theorem for dual mixed volumes of pairs of starshaped sets and a “dual analogue” of Hadwiger’s theorem classifying homogeneous valuations which are rotation invariant (see the related references in [293]). Continuing this, in [294] a classification of rotation invariant valuations on starshaped sets without the requirement of homogeneity is given. Although it is not focused on valuations, we mention here the paper [9]. Its authors derived a dual analogue, namely for two star bodies within the dual Brunn–Minkowski theory, of Shephard’s classification of quermassintegrals of two convex bodies. They showed that their characterization of dual quermassintegrals yields new determinantal inequalities among these dual quermassintegrals, and that it will be useful for the investigation of structural properties of the set of roots of dual Steiner polynomials of star bodies. In [542] a complete classification of all continuous GL(n) contravariant star body valuations on $L_{p}$-spaces is given. These results allow a new characterization of polar symmetric centroid bodies. The authors of [518] studied radial valuations of starshaped sets with continuous radial functions, obtaining an integral representation of the radial continuous valuations. They showed that every such valuation can be written as a certain integral over the unit $(d-1)$-sphere, and based on this they were able to characterize, in different ways, all continuous radial valuations which arise from a measure on the Borel sets of d-dimensional space.

As already mentioned in our discussion of intersection bodies, in [248,249,250] generalizations of intersection bodies within the dual $L_{p}$ Brunn–Minkowski theory were investigated. The authors established many fundamental results on $L_{p}$ intersection bodies, e.g. Busemann–Petty type results, injectivity results for $L_{p}$ intersection body operators, results on $L_{p}$ centroid bodies (related to intersection bodies), and a deep characterization of $L_{p}$ intersection bodies via special nontrivial $L_{p}$ radial valuations on convex polytopes. Schuster [459] extended results on projections and intersection bodies to a large class of valuations by studying the problem whether $\Phi K\subseteq \Phi L$ implies $V(K)\le V(L)$, where $\Phi$ is a homogeneous, continuous operator on convex or star bodies, being an SO(n) equivariant valuation. The nice survey [335] on classification results concerning valuations on function spaces (real valued, matrix valued and convex body valued valuations), both in Lebesgue and Sobolev spaces, also covers interesting analogous results on star bodies.

18 Extensions and generalizations

The geometry of starshaped sets is developed mainly in Euclidean spaces, but many results have generalizations in topological vector spaces. Starshaped sets are also considered in metric spaces (see [408]). A metric space $(X,\rho )$ is said to be starshaped at a point $a\in X$ if for every $x\in X$ the points x and a can be joined by a metric segment, that is, a subset isometric to an interval of length $\rho (a,x)$.

Another generalization of starshapedness was considered in convexity spaces (see [114, 312, 471]). A convexity space is a pair $(X,{\mathcal {C}})$, where X is a nonempty set and ${\mathcal {C}}$ is a family closed under arbitrary intersections which includes X and the empty set. A set $A\subset X$ is ${\mathcal {C}}$-starshaped at a point $a\in A$ whenever, for every $x\in A$,

$$\begin{aligned} \bigcap \{S\in {\mathcal {C}}\ :\ \{a,x\}\subset S\}\subset A. \end{aligned}$$

A natural generalization of convex and starshaped sets in terms of visibility functions can be found in papers by Beer (see [31, 32]). Let A be a measurable set in ${\mathbb {R}}^{d}$. The visibility function $v_{A}$ of A assigns 0 to every point of ${\mathbb {R}}^{d}\backslash A$, and to every $x\in A$ the Lebesgue outer measure $\mu _{d}$ of the star $\mathrm {st}\left( x:A\right)$. Beer proved

Theorem 68

Let E be a compact set in ${\mathbb {R}}^{d}$. If $x\in E$, the set of endpoints of all maximal segments in $\mathrm {st}(x:S)$ with one endpoint being x forms a measurable set and has measure 0.

Theorem 69

Let E be a compact starshaped set in ${\mathbb {R}}^{d}$ such that $\mathrm { int}E\ne \varnothing$ and $\dim \ker E\ge d-1$. Then the visibility function $v_{E}$ is continuous in $\mathrm {int}E$.

Let E be a measurable set in ${\mathbb {R}}^{d}$. In [32] Beer defined the pseudo-kernel of E as the set $P\ker E=\left\{ x\in E:v_{E}\left( x\right) =\mu _{d}\left( E\right) \right\}$, and he said that E is pseudo-starshaped if $P\ker E\ne \varnothing$.

Theorem 70

Let $K\subset {\mathbb {R}}^{d}$ be a compact set such that $\mu _{d}\left( K\right) >0$ and $P\ker K\ne \varnothing$. Then $K=S\cup F$, where $\mu _{d}\left( F\right) =0$ and S is a compact starshaped set with convex pseudo-kernel $P\ker K$. Moreover, $\mu _{d}\left[ \mathrm {st}\left( x:F\right) \right] =0$ for every $x\in F$.

Forte Cunto [190] characterized the points of discontinuity of the visibility function in the boundary of a planar Jordan domain, and Piacquadio, Forte Cunto, and Toranzos extended this characterization to compact subsets of ${\mathbb {R}}^{d}$ in their papers [191, 415], which also include several interesting examples of “pathological stars”. They introduced the set of restricted visibility of a point p in S as the set $\mathrm {rv}\left( p:S\right) =\mathrm {st} \left( p:S\right) \backslash \mathrm {nova}\left( p:S\right)$, that is, the set of all points of S that see p via S but do not see it clearly. If $S\subset {\mathbb {R}}^{d}$ is a compact set and $x\in \mathrm {bd}S$, they say that $\mathrm {st}\left( x:S\right)$ is healthy if either $\mu _{d} \left[ \mathrm {rv}\left( x:S\right) \right] =0$ or $\mathrm {int}\left[ \mathrm {rv}\left( x:S\right) \right] \ne \varnothing$. Otherwise, $\mathrm { st}\left( x:S\right)$ is pathological.

Theorem 71

Let $S\subset {\mathbb {R}}^{d}$ be a compact set and let $x\in S$ be such that $\mu _{d}\left[ \mathrm {st}\left( x:S\right) \right] =\mu _{d}\left[ \mathrm { nova}\left( x:S\right) \right]$. Then the visibility function $v_{S}$ is continuous at x.

Theorem 72

Let $S\subset {\mathbb {R}}^{d}$ be a compact set and $x\in S$. Then the visibility function $v_{S}$ is continuous at x if and only if $\mu _{d} \left[ \mathrm {rv}\left( x:S\right) \right] =0$.

Another generalization, almost starshapedness, was investigated by Breen [109] and Cel [130]. A set A is almost starshaped at $a\in A$ if the set $\{x\in A\ :\ \left[ a:x\right] \not \subset A\}$ is nowhere dense in A. Moreover, Breen (see [109]) defined staircase starshaped sets for families of boxes in ${\mathbb {R}}^{d}$, and she studied their properties. Cel introduced quasi-starshapedness in [130]: a set $S\subset {\mathbb {R}}^{d}$ (or a real topological linear space) is quasi-starshaped if there is a point $q\in \mathrm {cl}S$ such that the subset of points of S visible via S from q is everywhere dense in S and contains $\mathrm {int}S$. The set of all such points q is called the quasi-kernel of S, and it is denoted by $\mathrm {q}\ker S$. Following Valentine [527], a point $s\in \mathrm {cl}S$ is said to be a point of weak local convexity of S if there is some neighborhood N of s such that for each pair of points $x,y\in S\cap N$, $\left[ x:y\right] \subset S$. If S fails to be weakly locally convex at q in $\mathrm {cl}S$, then q is called a point of strong local nonconvexity (slnc point) of S. The set of all slnc points of S is denoted by $\mathrm {slnc}\,S$. Also following Valentine, a point $s\in \mathrm {cl}S$ is said to be a point of strong local convexity of S if $S\cap N$ is convex for some neighborhood N of s. If S fails to be strongly locally convex at $r\in \mathrm {cl}S$, then r is called a point of mild local nonconvexity (mlnc point) of S. The set of all mlnc points of S is denoted by $\mathrm {mlnc}\,S$. Finally, for each point z, ${\hat{A}}_{z}$ is the set of all $s\in \mathrm {cl}S$ such that z is clearly visible from s via S. Cel [130] proved

Theorem 73

If $S\subset {\mathbb {R}}^{d}$ (or a real topological linear space) is a connected set such that $\mathrm {slnc}\,S\ne \varnothing$, then

$$\begin{aligned} \bigcap \nolimits _{z\in \mathrm {slnc}\,S}\mathrm {conv}{\hat{A}}_{z}\subseteq \mathrm {q}\ker S, \\ \bigcap \nolimits _{z\in \mathrm {mlnc}\,S}\mathrm {conv}{\hat{A}}_{z}\subseteq \mathrm {q}\ker S. \end{aligned}$$

The notion of starshapedness was also developed in fuzzy-set settings. Starshaped fuzzy sets were introduced by Brown in [115], and then used by many authors (see [160, 162, 419, 543]).

19 Applications and further topics

In this section we will survey applications of the notion of starshapedness in various mathematical disciplines. Here, it is impossible to be complete and define all the mentioned notions. However, all non-defined notions are certainly “folklore”, at least for those specialized in the respective fields. Also, we are additionally selective by concentrating on references which refer mainly to geometric properties of starshaped sets. Due to the broad spectrum of fields from pure and applied mathematics that are seriously affected by starshapedness, we restrict ourselves to fields where the notion explicitely occurs.

19.1 Discrete and computational geometry

We will start with applications of, or results on, starshaped sets in the framework of discrete and computational geometry, almost everywhere in the Euclidean setting (otherwise we explicitly describe the relevant type of non-Euclidean geometry).

It is natural that under this headline starshaped polygons and polyhedra, and also notions from the combinatorial geometry of convex bodies, are discussed.

Krasnosel’skii’s theorem, presented in our Sect. 6, was clearly inspiring for many problems in discrete and computational geometry that are based on notions like visibility or illumination (see, e.g., the problem book [148], in particular its subsections A5 and E2, and regarding the large variety of different visibility and illumination concepts as well as related notions also the papers [367, 368] might be taken into consideration). One of the most famous types of problems in this direction is usually summarized under the name art gallery problems, for which we also refer to our Sect. 14. It was V. L. Klee who suggested in 1973 (at a conference in Stanford) the systematic study of the original problem, namely to combine an art gallery (which is in most situations assumed to be a simple, closed polygon P) with the minimum number of guards (all guards being points from P) who together can observe the whole gallery. Important summaries of this type of problems are [381, 402, 525]. It is obvious that such problems are closely related to notions like unions of starshaped sets and kernels. The paper [324] computationally deals with the kernel of starshaped polygons P with n vertices, i.e., the (non-empty) intersection of appropriate half-planes determined by prolongations of the edges of P; the authors present an optimal O(n)-time algorithm for finding the kernel of P. Via the notion of spindle starshapedness, where the usual linear visibility notion is replaced by so-called spindle visibility (replacing segments suitably by spindles), the authors of [52] established spindle analogues of well-known theorems by Krasnosel’skii, Carathéodory, and Klee and of more recent results by other authors. They also gave a spindle version of the art gallery problem, characterizing planar galleries for which one guard is sufficient. In [458], the authors called the family of k guards necessary to control a gallery P the k-kernel of P, and they gave a characterization of usual starshapedness in terms of 2-kernels. One can also modify the shape of guards, e.g. allowing them to be line segments; see [56], where the authors presented a linear time algorithm to find a segment-shaped guard. Similarly, an open or closed edge of the given polygon can be defined as a guard (cf. [517]), also yielding some type of segment starshapedness. In [517] the authors simplified the proof of a result from [406]: every simple non-starshaped polygon admits at most three closed guard edges; and it admits at most one open guard edge. In addition, open guard edges are characterized by using a special type of kernel, and also results on polygons with holes are derived, where at most six guards are possible. In [27] a fractional d-dimensional Helly-type result was proved, assuming that many $(d+1)$-tuples of a family of convex sets have a starshaped union and concluding that many of the sets have a common point. Also related art gallery results on polygons with a bounded number of holes were obtained, completing previous results on galleries without holes.

Now we come to other types of results from discrete and computational geometry, starting with triangulations of polygons and polytopes which are related to starshapedness. In [177] a linear-time algorithm for triangulating a starshaped polygon was derived. Regarding the analogous question in 3-space (using then tetrahedra), the authors of [447] verified the NP-completeness of the triangulation problem for polyhedral sets. They also showed that this statement remains true even for given starshaped polyhedra. In [167] it was proved that finding empty triangles (i.e., triples in a given finite point set F forming vertex sets of triangles with no other point from F in their interiors) is related to the problem of determining pairs of vertices that see each other in a starshaped polygon. A linear-time algorithm for the latter problem, being of independent interest, yields an optimal algorithm for finding all empty triangles. Possible extensions to higher dimensions are also mentioned in [167]. The problem of polygonal separation in the plane is to find a convex polygon with the minimum number of sides separating two given finite sets $S_{1}$ and $S_{2}$ with $k_{1}$ and $k_{2}$ points, respectively. Besides other results, the authors of [173] found an algorithm which solves this problem subquadratically in k, where $k=k_{1}+k_{2}$. In the preprocessing task, the convex hull of the internal set ($S_{1}$, say) and a nested starshaped polygon determined by the outer set $S_{2}$ play the key role (the separating polygon is contained in the annulus between both boundaries). A polygonization of k points in the plane is a connection of the points yielding finally a simple polygon. Since the number of possible polygonizations of k points is exponential in k, the problem of finding polygonizations into starshaped polygons with kernels having nonempty interiors was studied in [159]. It is proved that the number of distinct polygonizations into nondegenerate starshaped polygons is $O(k^{4})$, and that one can get them in $O(k^{5})$ time. The proof uses kernels of nondegenerate starshaped polygons derived suitably from the given set. In [244] a weighted Erdős-Mordell inequality for starshaped n-gons was proved, where the case of equality remained open; a geometric characterization of the n-gons satisfying the case of equality was given in [439]. A further notion very important in computational geometry as well is that of Voronoi diagrams. The study of Voronoi diagrams in spaces with a metric $d^{*}$ is strongly based on the property that Voronoi regions are always $d^{*}$-starshaped (see Chapter 1 of [302] and Sect. 15 of [59]) and therefore connected. Such properties have to be taken into consideration for investigations of abstract Voronoi diagrams and, on the other hand, for related structures created by so-called nice metrics; see again [302] and also [14]. In [267] results of the following type are presented: Given a starshaped polygon P in the plane and a vertex a of P, a point b is said to be accessible to a if, when a is moved to b along a linear path while all the other vertices of P are fixed, the polygon remains starshaped and isomorphic to P throughout the entire process of motion. The author proves that the set of points accessible to a is always an unbounded, open, starshaped set whose boundary consists of straight line segments. The authors of [466] developed an algorithm for inscribing convex polygons (which are extremal regarding certain quantities) suitably into starshaped figures, where starshaped polygons are also used in the intermediate steps. A useful list of related results is also given.

In [136] an efficient algorithm for sampling a given starshaped body K was derived. The respective complexity grows polynomially in the dimension and inverse polynomially in the fraction of the volume taken up by the kernel of K. The approach uses a new isoperimetric inequality, and the authors also obtained a polynomial algorithm for computing the volume of such a set. (In contrast it should be noted that linear optimization over starshaped sets is NP-hard.) Similar results are derived in [266], where it was also shown that these results cannot be extended to polyhedral spheres in 3-space. In [201] the question was investigated whether it is possible to recognize starshaped polygons in the plane when the one-dimensional measures of their sections parallel to one or more directions are known. One of the main results says that polygons starshaped at the origin cannot be determined this way by k directions, for any k.

Three-dimensional isohedra are polyhedra whose facets are equivalent under symmetries. It seems that not too much is known about isohedra with non-convex facets. Grünbaum and Shephard [239] showed that such polyhedra are starshaped of genus 0, and that their facets are starshaped pentagons with one concave vertex. McMullen [377] investigated regular honeycombs with starshaped cells or starshaped vertex figures, concentrating on those having five-fold rotational symmetries. He gave a description of such honeycombs from the viewpoint of abstract regular polytopes and presented discrete realizations in higher dimensions, discussing also certain quasiperiodic tilings with them.

It is not surprising that within the context of lattice points and convex bodies visibility notions also attracted attention; see, e.g., Chapter 13 of [179], where it is underlined that they are also hidden under the disguise of various illumination and transversality concepts. It was Hermann Minkowski who initiated (in about 1910) the geometry of numbers which studies convex bodies and integer vectors in n-dimensional spaces. This field has close connections to other fields of mathematics, especially functional analysis and Diophantine approximation; cf. the basic reference [236]. This book already showed that starshaped sets play an essential role within the geometry of numbers, see particularly Chapters 1, 4, and 5 there (e.g., considering notions like reducible star bodies). The so-called first theorem of Minkowski says that any origin-symmetric convex set in d-space having volume greater than $2^{d}D(L)$, where D(L) is the determinant of a d-dimensional lattice L, contains a non-zero lattice point. A generalization of this result was given by van der Corput, assuming positive numbers of pairs of lattice points (instead of the origin only) contained in the convex set. Rogers [438] conjectured some analogue of van der Corput’s result for star bodies, and he confirmed this for certain (for example, all prime) numbers. There are earlier results in this direction. For a starshaped region K, a lattice is said to be K-admissible if its only point interior to K is the origin. Let $\delta (K)$ be the lower bound for the determinants of all K-admissible lattices. If the determinant of a lattice equals $\delta (K)$, the lattice is said to be critical. Mordell (see [387]) developed a general method for studying lattice determinants and critical lattices for starshaped sets with rectangular symmetry, showing that nonconvex lattice point problems form a promising area of research in the geometry of numbers. And Bambah [26] continued with the study of star-regions with hexagonal symmetry and starshaped dodecagons in this framework. Mahler (see [346, 348]) assumed that the boundary of a starshaped set K consists of a finite number of analytic arcs and determined a process whereby $\delta (K)$ may be found in a finite number of steps. He proved that a critical lattice contains at least two independent boundary points $a_{1},a_{2}$ of K, and that there is only a finite number of K-admissible lattices containing both $a_{1}$ and $a_{2}$. In further papers (see, e.g., [347]) he applied his methods also to so-called “infinite starshaped sets”. In a sequence of related papers, many further results on starshaped sets in lattices were derived by Mahler, see [349,350,351,352,353,354]. Later Mahler [355] proved the following nice theorem: For any dimension d and any $\varepsilon >0$ there is a bounded starshaped body K such that $V(L)<\varepsilon \delta (K)$ holds for the volume V(L) of any convex body L contained in K.

A subset of the plane is called a universal cover if it can be moved (not only translated) to cover any plane set of diameter 1; Lebesgue asked for the convex universal cover of least area. This question is still not settled (see, e.g., D15 and D16 in [148]), and such questions can also be posed for non-convex figures. In this framework Kovalev [317] proved that every cover of minimum area has to be starshaped, and he also derived certain conditions on the radial functions describing the boundaries of such extremal figures.

Makeev [362] showed that for any cyclic quadrilateral and any closed starshaped $C^{2}$ Jordan curve there is a circle intersecting this curve in exactly four points which form the vertex set of a quadrilateral similar to the given one.

19.2 Inequalities

Various inequalities mainly coming from convex geometry were already discussed in our Sects. 13 and 17. We continue this now with a broader view. It is clear that results related to this headline are widespread and partially also discussed in other parts of this paper (for example, in our Subsection 19.1 on discrete and computational geometry or in our Sect. 14). Thus, the following selection completes these other results already given. Even more, the authors cannot give a complete picture of the field “inequalities and starshapedness”, since there are simply too many (also recent) results in this direction. Exemplary keywords for respective further research are: centroid bodies, dual mixed volumes, intersection bodies, Orlicz-type extensions, starshaped functions, and valuations.

Gueron and Shafrir [244] proved a weighted version of the Erdő s-Mordell inequality for starshaped n-gons, and they asked for a geometric characterization of the n-gons reaching equality. In [439] this characterization is given.

The following rather surprising theorem is due to Mahler [355]: For every $\varepsilon >0$ there is a d-dimensional bounded starshaped body S whose volume V(S) satisfies $V(S)<\varepsilon \delta (S)$, where $\delta (S)$ denotes the lattice determinant of S (see Subsection 19.1). In [295] a new geometric tool (by which planar starshaped sets are dissected into four parts and, by a so-called cyclic rearrangement technique, suitably glued together around some rhombus, to get a new set of the same perimeter, but with larger area) is introduced and geometrically analysed. Applying this, the author could give a new proof of the planar isoperimetric inequality and also obtained new approaches to Bonnesen-style error estimates for the isoperimetric deficit of starshaped figures centered at the origin. Fang proved in [182] a reverse isoperimetric inequality for planar starshaped curves in terms of the perimeter and the enclosed area. The Chernoff inequality, also holding for planar convex curves, estimates their enclosed area in terms of the width function, with circles as extremal curves. In [554] a so-called k-order radial function $\varphi _{k}(\theta )$, “replacing” the width function and depending on k equi-distributed support function values, is introduced for starshaped planar curves to establish a geometric inequality involving $\varphi _{k}(\theta )$ and the area enclosed by a starshaped curve. Certain bounds are obtained, and it is shown that the exact lower bound for the area is reached when $\varphi _{k}(\theta )$ is a constant. We refer once more to [271]; this paper contains an interesting inequality bounding the length of a (non-convex) curve of bounded curvature in terms of the enclosed area. Also related is [405] which contains a new, elegant proof of an inequality estimating the maximal curvature of a smooth Jordan curve against the enclosed area (starshapedness plays an interesting role in the argumentation). Here we also mention the deep paper [30] which refers to generalized distance functions whose indicatrix (gauge) is starshaped with respect to the origin. In this framework, the authors derived a very general form of the isodiametric inequality for measurable sets. Keogh [290] found sharp upper bounds for the curvature radius and the length of level curves of convex regions if the function which maps onto the unit circle is normed in 0, and if one considers the curve corresponding to the circle of a fixed radius r. These bounds are given in terms of r, and analogous results were derived replacing the convex regions by starshaped ones.

The paper [340] deals with the Legendre ellipsoid L of a convex body K in Euclidean d-dimensional space having o as center of mass; this is the ellipsoid centered at o whose moment of inertia about any axis passing through o equals the corresponding moment of inertia of K. Relations to isoperimetric-like inequalities are discussed, and this definition is extended so that K is allowed to be starshaped. It is shown that L always contains another ellipsoid $L^{*}$, being somehow dual to L,  where equality characterizes ellipsoids among starshaped sets, and that this inclusion is the geometric analogue of the so-called Cramer–Rao inequality, which is very important in information theory. The main result of the paper [240] extends the Alexandrov–Fenchel quermassintegral inequality for a convex domain to starshaped domains satisfying a smoothness condition. Continuing [240] by using the inverse mean curvature flow, the authors of [113] extended the Minkowski inequality, which treats the (inward) mean curvature H of convex closed hypersurfaces, to starshaped closed hypersurfaces which are mean-convex (i.e., $H\ge 0$).

The notion of general mixed chord integrals of star bodies was extended in [329] to the general $L_{p}$-mixed chord integrals of star bodies. Their extremum values, some type of Aleksandrov–Fenchel inequality, and a cyclic inequality for general $L_{p}$-mixed chord integrals of star bodies were studied. Furthermore, two Brunn–Minkowski type inequalities for $L_{p}$-radial bodies were derived. Also in [537] Brunn–Minkowski type inequalities for star bodies were studied. Inspired by the notion of $L_{p}$-mixed geominimal surface area of multiple convex bodies, in [328] the concept of $L_{p}$-dual mixed geominimal surface area for multiple star bodies was investigated; also here several inequalities related to this concept were obtained. In [553] the notion of $L_{p}$-mixed intersection body (for any $p\ne 0$) is introduced, and a Minkowski-type and a dual Aleksandrov–Fenchel inequality for such bodies were deduced.

For suitable sets $A\subset B$ in Euclidean d-space, let o be an interior point of A. The authors of [186] proved that if A and B are compact sets starshaped with respect to o, where o is their common barycenter, then there is a positive number k such that for every $0<\lambda \le k$ the set $\lambda A$ is convexely majorized by B (i.e., the inequality $v_{A}\le v_{B}$ holds for each real continuous convex function v defined on the closed convex hull of $A\cup B$). If, furthermore, B is a convex set with $-B=B$, then there is a universal positive constant $k_{d}$ (depending only on the dimension d) such that the following holds: every symmetric convex set A satisfying $A\subset k_{d}B$ so that A and B have the same barycenter is convexly majorized by B.

19.3 Starshapedness in differential geometry

In this subsection we want to survey selected results from differential geometry using or directly yielding geometic properties of starshaped sets.

We start with results on types of curvatures and geometric flows which are usually understood as gradient flows associated with a functional on a manifold which has a geometric interpretation (e.g., associated with some extrinsic or intrinsic curvature notions). We start with the planar case. In the paper [271] an extensive collection of differential geometric techniques was used to investigate plane curves with bounded signed curvature. The main results also concern bounds for the length of the boundary of a closed planar disk in terms of its area; in particular, one interesting result on such disks is the necessity of their being starshaped (with respect to the origin), with boundary between the unit disk and its homothet of radius three.

In [544] the asymptotic behaviour of starshaped closed curves

$$\begin{aligned} \left( x\left( \vartheta ,t\right) ,y\left( \vartheta ,t\right) \right) \in {\mathbb {R}}^{2},~\vartheta \in S^{1},0\le t<\infty , \end{aligned}$$

in the plane, following the equation $V=1-K$, was studied, where K denotes the curvature and V the outward-normal velocity. E.g., if the curvature is smaller than 1, then asymptotic shapes (as t is running to infinity) exist and determine the primary curve (when $t=0$) uniquely. Also related, in [520] the author investigated the evolution of planar closed starshaped curves which move in the direction of their outer unit normal vectorfield with speed given by a suitable function f of their curvature. It turns out that for very general functions f the curves expand to infinity in infinite time, and combined with a suitable rescaling they converge to a circle. Somewhat continuing [520], the authors of [138] showed that for a certain class of closed embedded initial curves (more general than starshaped ones) the solutions become starshaped, then (after a finite length of time) convex, and then asymptotically round. A little survey on this evolution of embedded plane curves by functions of their curvature, considering expansion flows, is [521]. Vassiliou [533] discussed the interrelations between the technique of moving frames and a more general scheme to implement group-theoretical techniques in the framework of local differential invariants of curves. Based on this, he derived an explicit expression for the equi-affine curvature of a plane starshaped curve.

Continuing studies of Foland and Marr [189], Beltagy and Shenawy [46] investigated starshaped sets whose kernels consist of a single point and which are embedded in complete, simply connected $C^{\infty }$ Riemannian 2-manifolds without conjugate points.

Recall that a point p is called an equireciprocal point of a planar curve C if C is starshaped at p and if every chord xy of C passing through p satisfies the relation $\left\| p-x\right\| ^{-1}+\left\| p-y\right\| ^{-1}=\alpha$ for some constant $\alpha$. (For example, the foci of ellipses are such points.) In [181] it was shown that, except for certain cases, any curve with two such points must have the same constant at each point. Also, any twice-differentiable curve with two such points must be an ellipse, but in general there exist non-elliptical convex curves with two such points.

The Korteweg–de Vries (KdV) equation is a mathematical model of waves on shallow water surfaces, integrable and invariant under the Mö bius transformation. Well known in PDE and of great interest both in physics and mathematics, it is a non-linear partial differential equation whose solutions can be exactly specified. In [416] a geometric interpretation of the KdV equation as an evolution equation on the space of closed curves in the centroaffine plane was provided. With this background, an example of a soliton equation coming in a natural way from a differential geometric problem was presented. The importance of starshaped curves in centroaffine geometry was shown, and continuing these considerations, the authors of [122] showed that projectivization induces a map between differential invariants of starshaped planar curves and a bi-Poisson map between Hamiltonian structures. It was also verified that a Hamiltonian evolution equation for closed starshaped planar curves, discovered in [416], has the Schwarzian KdV equation (involving the Schwarzian derivative) as its projectivization.

The locus of points from which a strictly convex curve C in the plane is seen from outside under the same angle $\alpha <\pi$ is called the $\alpha$-isoptic of C. The authors of [139] proved that if C is of class $C^{2}$ with nonvanishing curvature, then its $\alpha$-isoptic is also of class $C^{2}$ and a starshaped curve. Starshaped curves also occur in the study of the group-theoretical structure created by reflections suitably defined in normed planes and their combinations. Namely, in [369] cycles in strictly convex normed planes were introduced as the loci of points that are images of a given point x under the set of all left reflections in lines through a second point y. It turns out that cycles are starshaped closed Jordan curves that can be used to characterize smooth normed planes, Radon planes or the Euclidean subcase; differentiability conditions also play a role there. In [226], planar closed $C^{1}$ curves were investigated which are starshaped with respect to the origin and can be seen as “analogues” of curves of constant width in the sense that they “intersect the rays from the origin transversally”.

Before switching to higher dimensions, we still want to mention a nice result on curves in three-dimensional space. Namely, Ghomi [217] proved the following extension of the classical four vertex theorem: Let there be given a simple closed $C^{3}$ immersed curve with nonvanishing curvature in 3-space. Suppose that this curve is starshaped and locally convex with respect to a point in the interior of its convex hull. Then its torsion changes sign at least four times.

Of course, starshapedness also plays a role in higher dimensional, differential geometric settings. Again we start with results related to curvature notions. E.g., higher dimensional analogues of the above mentioned planar results from [138, 520] related to geometric flows were derived in [166, 184, 215, 216, 275, 452, 524]. More precisely, Gerhardt [215] studied closed starshaped hypersurfaces in Euclidean d-space which expand in the direction of their exterior unit normals. The speed of the surfaces is given by the inverse of a certain function of the principal curvatures satisfying some additional conditions. The resulting flow is in some sense complementary to the inward flow by the mean curvature (which was investigated in [468], also regarding the development of certain singularity types) or the Gauss curvature of convex surfaces and allows the dropping of the convexity assumption; so any starshaped initial surface will flow for infinite time and converge to a round sphere after appropriate rescaling. Urbas [524] investigated smooth, closed starshaped hypersurfaces expanding analogously in the direction of their normal vectorfield and showed that during the procedure the hypersurfaces remain smooth and starshaped forever and become asymptotically round. Similar results were given in [166, 184, 275, 365]. The latter paper deals also with Gerhardt’s investigations from [208]: The author considers classical solutions to the inverse mean curvature flow in the case where the initial hypersurface is a starshaped surface with strictly positive mean curvature. In contrast to Gerhardt’s work, he studies hypersurfaces possessing a boundary which meets the cone perpendicularly. The cone can be viewed as a supporting hypersurface for the evolving surface, and is not moving itself. It is proved in [365] that this flow exists forever, and that the surfaces converge to a piece of the round sphere. As a consequence of such results, the authors of [184] recovered the existence result for Weingarten hypersurfaces (see also [121]). In [216], analogous results on the inverse curvature flow of starshaped hypersurfaces in the hyperbolic d-space with asymptotic estimates on the rate of convergence to spheres were established; see also [452]. In [330] a new modified mean curvature flow of complete embedded starshaped hypersurfaces in hyperbolic d-space with prescribed asymptotic boundary at infinity was investigated. Under some geometric conditions on the initial hypersurfaces, the existence, uniqueness and convergence of such a flow was shown. Extending results from [215, 524], Ivochkina et al. [279] studied flows with non-homogeneous speeds applied to starting hypersurfaces which are compact and starshaped. The authors proved the long-time existence of solutions for such flows, and the convergence (of appropriately rescaled hypersurfaces) to a sphere was confirmed with some additional assumptions.

The existence of closed starshaped hypersurfaces with prescribed curvature properties was studied by several authors from differential geometry. In [2] the following was shown: A hypersurface F in d-dimensional Euclidean space which is starshaped with respect to a fixed point o and has constant value $rH_{1}$, where $r=r(p)$ is the distance between o and $p\in F$, and $H_{1}=H_{1}(p)$ is the first mean curvature of F in p, is a hypersphere around o. Using two suitable consecutive mappings (namely a hyperplane reflection and a homothety), this was elegantly reproved in [491]. Also the following results characterize balls (or spheres) among starshaped sets. Namely, the authors of [373] obtained isoperimetric estimates relating the Lebesgue measure of a bounded domain with smooth boundary and the Levi curvatures of the boundary (that is, elementary symmetric functions of the eigenvalues of the normalized Levi form). They proved that the only bounded smooth starshaped domains whose classical mean curvature is bounded from above by a positive constant Levi curvature are balls. The paper [241] contains existence results on smooth starshaped hypersurfaces whose curvature measures are prescribed via the radial map. In the context of convex geometry, the studies from [241] can be seen as a counterpart for curvature measures of the Christoffel–Minkowski problem, concerning area measures. It turns out that in a natural way the topic that we discuss here is also interesting from the viewpoint of our subsection on PDEs! For the case of prescribed mean curvature we refer furthermore to [25, 121, 254, 519], where also the intermediate cases of k-th mean curvatures were investigated, as well as cases of more general curvatures. Related extensions to the hyperbolic space can be found in [28, 282], and the elliptic case (with mean curvature types) is particularly discussed in [327]. Replacing the mean curvature by the Gauss curvature, analogous problems are presented in [156, 400], and regarding the Weingarten curvature we refer to [137, 184].

Extending Cohn–Vossen’s classical rigidity theorem, the authors of [242] proved that any two $C^{2}$ compact starshaped hypersurfaces in a complete, simply connected space form $N^{(d+1)}(K)$ for curvature $K=-1$ or 0, with the normalized scalar curvature strictly larger than K, are congruent. Moreover, if the ambient space is the unit $(d+1)$-sphere, then any two $C^{2}$ compact starshaped hypersurfaces with normalized scalar curvature strictly larger than 1 are congruent if the hypersurfaces are contained within (possibly different) hemispheres.

Beltagy (see [39, 40, 42]) studied starshaped sets in complete, simply connected Riemannian manifolds without conjugate points, also in the d-dimensional hyperbolic space. Using the Beltrami (or central projection) map from d-dimensional hyperbolic to d-dimensional Euclidean spaces, he verified that many geometric properties of starshaped sets in the Euclidean case hold in the hyperbolic space as well; for the spherical case see [40]. He also derived results of Krasnosel’skii-type and on the dimension of the kernel of a starshaped set in the hyperbolic case, and he investigated the reason for the importance of the non-existence of conjugate points, too.

In [43] local starshapedness plays an important role: An embedding of a compact, connected, and smooth d-manifold M without boundary in a complete, simply connected, and smooth Riemannian $(d+1)$-manifold W without conjugate points bounds a convex subset of W if and only if the inner component of the embedding is locally starshaped. Halpern [253] proved the following: Take the subset of the Euclidean $(d+1)$-space formed by the union of all tangent hyperplanes at points of an immersed, compact, closed, connected and smooth d-manifold M. If this point set is a proper subset of the considered $(d+1)$-subspace, then M is diffeomorphic to a sphere, the image of the immersion is the boundary of a unique open starshaped set S, and the set of points not belonging to any of these tangent hyperplanes forms the interior of the kernel of S. A converse statement was verified, too. Furthermore, the set of points not on any tangent hyperplane forms the interior of the kernel of this starshaped set, and a converse statement holds, too. In [41] the results of Halpern were generalized: Namely, let $f:M\rightarrow W$ be a smooth immersion of a compact connected d-dimensional smooth manifold M into a smooth, complete and simply connected $(d+1)$-dimensional Riemannian manifold W without conjugate points. Beltagy proved that if the subset of W swept out by the geodesics of W which are tangential to f(M) is not the whole W, then M is diffeomorphic to a sphere and f is an embedding onto the boundary of a unique open starshaped subset of W. Also the paper [45] discusses starshaped sets and locally starshaped sets in Riemannian manifolds without conjugate points. E.g., it was proved there that if the closure of a connected open set with a smooth hypersurface as boundary is locally starshaped at each boundary point, then this set is convex. In [326] closed (Weingarten) hypersurfaces embedded in warped product manifolds, based on compact Einstein manifolds equipped with the Riemannian metric, were investigated. The authors proved an analogue of Aleksandrov’s theorem for such hypersurfaces, and among the occuring geometric conditions starshapedness of the considered hypersurfaces plays the key role. Finally we mention here the papers [178, 291] where (further) properties of starshaped sets embedded into Riemann manifolds were presented, as well as [113] concerning starshaped sets in a class of asymptotically hyperbolic manifolds with boundary.

A notion well known in the geometry of symplectic spaces and connected with isomorphisms there is that of Lagrangian subspaces (or shortly Lagrangians). Using variational methods, Guo and Liu [246] proved that if F is an arbitrary $C^{2}$ smooth, compact, symmetric hypersurface starshaped with respect to the origin of a (2d)-dimensional space, then for every Lagrangian subspace L the hypersurface F has infinitely many L-Lagrangian orbits on F. Similarly, in [247] multiplicities of Lagrangian orbits on starshaped hypersurfaces in the same framework are studied. Investigating the existence of periodic solutions of Hamiltonian systems (i.e., mathematical formalisms developed by Hamilton to describe the evolution equations of physical systems) of ordinary differential equations, Rabinowitz proved in [420] properties of starshaped level sets of Hamiltonian functions in the standard symplectic $\left( 2d\right)$-dimensional Euclidean space, referring to periodic orbits of the occuring vector fields. Berestycki et al. [51] continued these investigations by studying the existence of periodic orbits of Hamiltonian systems on a given starshaped energy hypersurface. Similarly, the paper [218] deals with the case that the starshaped energy hypersurface is symmetric with respect to the origin, and Viterbo [535] considered the problem of finding closed orbits on special types of starshaped hypersurfaces. It turns out that for such generic surfaces, either there are infinitely many closed orbits, or they are all hyperbolic (the latter cannot occur for even d). In [331] the authors got analogous results, and they showed that at least one hyperbolic closed characteristic exists if the Maslov-type mean index of every closed characteristic is larger than 2 when d is odd, and larger than 1 when d is even. (Note that the Maslov index is a tool for determining in $\left( 2d\right)$-space the nature of intersections between two evolving Lagrangian subspaces.) A Maslov-type index was also used in the strongly related paper [274], and the iteration formula of Viterbo [535] for non-degenerate starshaped Hamiltonian systems was generalized. In the expository paper [409] on the so-called Weinstein conjecture (asserting that the characteristic line bundle of a compact, contact type hypersurface in a Finsler manifold has a closed integral curve) the above topics were discussed, too.

19.4 Starshaped sets and PDE

Starshaped sets also play an explicit role in several papers in the field of partial differential equations (PDE). In the following we cite examples. Additionally, the reader is referred to our subsection on differential geometry, since many results there are obtained via methods related to this subsection. In particular, the discussion on compact Weingarten hypersurfaces is interesting for this interplay.

In [285], Kawohl proved that if the geometry of the data in the so-called “obstacle problem” or the “capacitory potential problem” yields a starshaped set, then the corresponding level sets are starshaped. The proofs are based on appropriate maximum principles. Similar results on the level surfaces of solutions of nonlinear Poisson equations are derived in [332].

Useful for the field of PDE, but also geometrically interesting in itself, a so-called radial symmetrization for starshaped sets was introduced in [488]. E.g., for a planar smooth curve starshaped with respect to the origin, presented by $r=r\left( \varphi \right)$ and bounding the region G, a natural number $n\ge 2$ defines a symmetrization $S_{n}$ which transforms r to r’, where r’ is the geometric mean of the n radii $r\left( \varphi +2k\pi /n\right)$, $k=0,\ldots ,n-1$. (This symmetrization decreases the area of G.) The author also considered three-dimensional analogues for smooth starshaped surfaces. With applications in function theory in mind, generalizations of such symmetrizations were presented in [363]. Kawohl [287] continued these investigations showing that via this procedure functions f are transformed into new functions $f^{*}$ with starshaped level sets, therefore yielding the notion of starshaped rearrangement. This is applied to certain variational and free boundary problems, also leading to new results on the geometrical properties of solutions of these problems. In the nice booklet [286], methods of rearranging functions as a valuable tool for investigating interesting geometric properties of solutions of PDEs are presented. Studying the shapes, information on level lines, critical values or certain symmetry properties can be obtained. The author gave a unified treatment of various methods of rearrangements, such as starshaped, Steiner, and Schwarz symmetrization (which are scattered in the literature), their interrelations and applications.

We now come to some special types of functions. Francini [195] studied the starshapedness of level sets created by nonlinear parabolic equations, considering the angle between the normal direction to the level surface and the radial direction. He verified that a maximum principle holds for this angle. In [163, 164] parabolic problems on a convex or starshaped ring-shaped domain were studied (ring-shaped means that the boundary is enclosed in a suitable generalization of an annulus). It was shown that if the initial data have convex or starshaped level sets, then the solutions have analogous properties. An interesting early survey of results concerning the properties of level sets of solutions of boundary-value problems for elliptic equations is [288]. Important techniques for getting related results, such as radial symmetrization and starshaped rearrangement, are nicely presented. In [198] special elliptic equations were investigated, showing that certain solutions in an exterior starshaped domain have starshaped level sets, and in [197] properties of starshapedness of level sets of a special rotationally invariant and strict elliptic equation in a starshaped ring with constant boundary values were studied. Also Salani [451] studied how geometric properties of the investigated domain are inherited by level sets of solutions of elliptic equations. In particular, solutions of elliptic Dirichlet problems in starshaped rings were shown to have starshaped level sets, and the results of this paper can be applied to a large class of operators. We refer also to [196], where related results on the starshapedness of level sets for solutions of elliptic and parabolic equations were exposed. The authors of [142] studied geometric properties of level sets of positive solutions of a semilinear elliptic equation in a bounded domain in d-dimensional space, satisfying homogeneous boundary conditions and having a certain symmetry property, and they proved that the level sets of any positive solution are starshaped with respect to the origin. The authors of [540] generalized the direct method of lines for elliptic problems in a starshaped domain C under the assumption that the boundary of C is a closed Lipschitz curve parametrized angularly, so that an appropriate transformation of coordinates can be introduced. Then the elliptic problem can be reduced to a variational-differential problem on a semi-infinite strip in the new coordinates; this method yields an effective way to solve a wide range of elliptic problems. In [214] it was verified that for the Green function g(P) of a region D in 3-space, starshaped with respect to the origin as pole, all regions $D_{k}=\{P:g(P)>k\}$ are also starshaped with respect to the origin. Stoddart [485] continued this with corresponding results for harmonic functions, replacing the pole at the origin with a starshaped continuum.

It is well known that the wave equation is an important second-order linear partial differential equation for the description of waves. There are also results combining the wave equation explicitly with starshapedness. The authors of [141] studied the wave equation in a bounded, starshaped domain with a nonlinear dissipative boundary condition. Under various assumptions on nonlinearity they proved decay estimates on the energy of the solutions. In [50] traveling waves for a nonlinear diffusion equation with bistable or multistable nonlinearity were investigated. The goal was to study how a planar traveling front interacts with a compact obstacle that is placed “in the middle” of the space. In particular, starshaped obstacles were taken into consideration. As a first step, the existence and uniqueness of an entire solution that behaves like a planar wave front approaching from infinity and eventually reaching the obstacle, was shown. This causes disturbance on the shape of the front, but the solution gradually recovers its planar wave profile and continue to propagate in the same direction, leaving the obstacle behind. Whether the recovery is uniform in space was shown to depend on the shape of the obstacle. In [426], the authors verified the almost global existence of solutions to quasilinear wave equations in the complement of starshaped domains in 3-space which satisfy a certain boundary condition.

19.5 Starshapedness in fixed point theory

It is well known that starshaped sets also play a role in fixed point theory. We want to present some results in this spirit, also selected due to their strong relation to the geometry of starshaped sets. These results mainly occur in (certain types of) Banach spaces and, more generally, in topological vector spaces. We note that in this subsection some results interesting from the viewpoint of approximation theory are cited; thus it is natural to refer here also to our subsection on approximation theory.

Dotson [168] showed that if C is a compact starshaped subset of a Banach space X and $T:C\rightarrow C$ denotes a nonexpansive mapping (i.e., $\left\| Tx-Ty\right\| \le \left\| x-y\right\|$ for $x,y\in X$), then T has a fixed point in C. A related result on weakly compact starshaped subsets of Banach spaces was derived, too. Closely related, certain well-known fixed point theorems for (compact, condensing, nonexpansive) mappings defined on convex sets were extended to starshaped subsets of certain linear topological Hausdorff spaces in [425]. E.g., a nonexpansive mapping defined on a bounded closed starshaped subset C of a Hilbert space that maps the boundary of C into C has a fixed point. Similar results (also treating special linear topological Hausdorff spaces) were derived in [423], see also the announcement [424]. Akkouchi [5] investigated asymptotically nonexpansive mappings defined on unbounded starshaped sets, confirming the existence of fixed points and hence generalizing similar results holding for bounded convex sets; the papers [1, 174] contain results in the same direction. Continuing [168], in [134] the following was proved for a closed, starshaped subset C of a Banach space and a nonexpansive mapping $T:C\rightarrow C$: if there exists a subset of C, an attractor for compact sets, then T has a fixed point in C. An analogous result for certain self-maps of bounded closed starshaped subsets of convex metric spaces can be found in [38]. In [64] it was shown that a convex subset K of a Banach space X is closed iff every contraction of X leaving K invariant has a fixed point in K. This yields the result that a normed space is complete iff each contraction on the space has a fixed point. It was also proved that these results fail if the convexity property is replaced by starshapedness. The authors of [395] showed that for closed starshaped subsets of finite dimensional Banach spaces compactness and the fixed point property for nonexpansive mappings are equivalent. Leaving the concept of self-mappings, Kuhfittig [321] showed the following result. Let C be a starshaped closed subset of a Banach space, and K be a subset of C. If K is a closed [compact and starshaped] subset of C, and C is also starshaped with respect to K, then the following implication holds. If $T:K\rightarrow H$ is contractive [non-expansive], also mapping the relative boundary of K back into K, then T has a fixed point in K. Among other related results, Carbone [124] proved that for a nonempty closed starshaped subset C of a normed linear space X and a nonexpansive map $f:C\rightarrow X$ with f(C) compact and $f(\mathrm {bd}C)\subset C$, f has a fixed point. In [374], fixed point theorems for nonexpansive multivalued mappings are obtained. E.g., let C be a nonempty, closed and starshaped subset of a Banach space X. If T maps C into the set of all compact subsets of X and is nonexpansive so that T(K) is bounded and satisfies a few further elementary conditions, then T has a fixed point. Yanagi [546] considered weakly compact starshaped subsets of uniformly convex Banach spaces and nonexpansive fixed-point mappings therein. The authors of [260] called a subset C of the separable Banach space X of all real sequences convergent to 0 coordinatewise starshaped (c.s.s.) with respect to $x\in C$ if, for any $y\in C$ and $z\in X$, z being coordinatewise between x and y implies that $z\in C$. They proved that weakly compact c.s.s. subsets of X have the fixed point property for nonexpansive mappings, and that the fixed points of such mappings can be constructed in an efficient way. Continuing [260], Dowling and Turett [169] showed that, for closed, bounded c.s.s. sets, the fixed point property is equivalent to weak compactness. In [422], several classes of nonexpansive set-valued self-mappings of bounded, closed, starshaped sets in Banach spaces were considered. Using the Baire category approach (see our subsection 19.8 below), it was shown that in these classes most mappings are contractive. In particular, this holds for a certain class of compact-valued mappings, implying that a generic mapping in this class has a fixed point. The paper [284] contains similar results on set-valued mappings in view of Brouwer’s theorem and defined over starshaped sets, and in a related spirit the Borsuk–Ulam antipodal theorem is also extended to the starshapedness framework. In [243] the existence of fixed points for nonexpansive and equicontinuous mappings in convex and starshaped metric spaces (including the case of Banach spaces and their starshaped subsets) was proved, thus also generalizing various results presented above.

Now we present some results on Hilbert spaces. The author of [219] proved some fixed point theorems for nonexpansive mappings on starshaped sets in a Hilbert space. The main result is as follows: Let C be a starshaped, closed subset of a Hilbert space and $T:C\rightarrow C$ a nonexpansive mapping such that some bounded set $C_{0}\subset C$ is mapped into itself; then T has a fixed point in C. Using the concept of attractive points of a nonlinear mapping, the authors of [3] obtained a strong convergence theorem for nonexpansive mappings, also concerning starshaped subsets of Hilbert spaces; these results implied a new fixed point theorem. In [397], attractive points of a class of generalized nonexpansive mappings on starshaped sets in a real Hilbert space were studied, and strong convergence theorems of iterative sequences generated by these mappings were established. Furthermore, the approximation of common attractive points of generalized nonexpansive mappings was investigated, and a strong convergence theorem by a new iteration scheme for these mappings was obtained.

Taylor [490] established variants of fixed point theorems for nonexpansive mappings on starshaped sets holding even in linear topological vector spaces. Zhang [559] showed that a closed subset C of a topological vector space is a starshaped subset with center $x_{0}$ if and only if $\left\{ x+\lambda (y-x):y\in C,\lambda \ge 1\right\}$ with $x\in C$ is a family of starshaped subsets with a common center $x_{0}$. In [407] Birkhoff–Kellogg type theorems (which generalize Brouwer’s famous fixed point theorem) for starshaped subsets of topological vector spaces and compact mappings were studied. A generalization of Ky Fan’s fixed point theorem (which extends traditional fixed point theorems to set-valued mappings) for compact starshaped subsets of topological vector spaces was proved in [48]. The authors of [273] derived an interesting related extension of the Markov–Kakutani fixed-point theorem to compact starshaped sets.

As announced, we finish this subsection by mentioning results close to approximation theory. Schu investigated in [455] the iterative approximation of a fixed point of a nonexpansive mapping with starshaped domain in certain reflexive Banach spaces. More precisely, for a subset C of a Banach space, a mapping $T:C\rightarrow C$ is said to be asymptotically nonexpansive if T is Lipschitzian and the Lipschitz constants $l_{n}$ of the iterates of T converge to 1 as $n\rightarrow \infty$ (the case of nonexpansive mappings is given by $l_{n}=1$). For starshaped domains C, in [457] some iteration schemes leading to the construction of fixed points of T were derived. The Browder–Göhde–Kirk theorem says that if T is a nonexpansive self-mapping of a nonempty, bounded, closed convex subset C of a uniformly convex Banach space, then T has a fixed point in C (see also [423]). A new version of this theorem, valid for starshaped subsets of a reflexive space with the Kadec–Klee property, was obtained in [456], together with some applications and construction principles to get the corresponding fixed point. In invariant approximation theory, occasionally results from fixed point theory are used. In this spirit, several theorems on nonexpansive mappings, particularly also defined on the class of starshaped sets, were proved in [135], thereby extending known results in this field.

19.6 Starshaped sets in approximation theory

Also in approximation theory starshaped sets play an interesting role. In most cases where they occur, the corresponding results are related to Banach space theory, fixed point theory, and convexity.

In particular, methods and results from fixed point theory (see the respective subsection) are also used in approximation theory, and our first paragraph here gives examples in this direction. Slightly generalizing [168], Habiniak [251] proved that the following holds for a closed starshaped subset S of a normed linear space: if $T:S\rightarrow S$ is nonexpansive and the closure of T(S) is compact, then T has a fixed point. This is then suitably applied to the problem of invariant approximation; namely, if T is a nonexpansive operator on a normed linear space with a fixed point a, leaving a subspace M invariant, and if T|M (the restriction of the operator to M) is compact, then a has a best approximation b in M, which is also a fixed point of T. However, we have to mention that earlier Singh [465] got a slightly more general result on starshaped, compact sets. Also generalizing results of Dotson [168] and Taylor [490], in [277] a common fixed point theorem for two self-maps R, S on a set M in a Hausdorff locally convex space (M having starshapedness properties) was derived, and its applications to best approximation operators were given. In [398] two self-mappings R, S on a weakly compact subset M of a Banach space were considered, where M has starshapedness properties, and R, S satisfy certain affine conditions. In [11] also pairs of self-maps were studied. Let D be a closed subset of a normed linear space, let R and S be self-maps of D with $R(D)\subset S(D)$, and x be a fixed point of S. If D is starshaped with respect to x, the closure of R(S) is compact, S is continuous and linear, S and R are commuting and R is S-nonexpansive, then R and S have a common fixed point. Based on this theorem, some known results on fixed points and common fixed points of best approximations are generalized. Also the following result of Ganguly [199], using fixed point theorems for best approximation, touches starshapedness explicitly: Let T be a generalized nonexpansive mapping on a normed linear space X, M a T-invariant subset of X, and x a T-invariant point. If the set of best M-approximants to x is compact and starshaped, then it contains a T-invariant point. This generalizes the result obtained in [465] for T being nonexpansive in the usual sense. The same author (see [200]) generalized [465] in another direction, namely replacing the usual starshapedness of M with some modified notion of starshapedness. Here we also refer to [403].

Toranzos [508] proved that, in the sense of Hausdorff metric, compact starshaped sets can be uniformly approximated by starshaped polytopes and starshaped smooth sets.

Let A be a nonempty closed proper subset of a reflexive Banach space. A point $x_{0}$ in the starshaped and bounded set S is called a solution of the minimization problem $\left( d_{A},S\right)$ if the distance $d\left( x_{0},A\right)$ equals the infimum of $\left\{ d\left( x,A\right) :x\in S\right\}$. The problem $\left( d_{A},S\right)$ is said to be well posed if it has a unique solution and if every minimizing sequence converges to the solution. The authors of [154] proved that the family of starshaped sets S such that $\left( d_{A},S\right)$ is well posed forms a dense $G_{\sigma }$-subset of the metric space (under the Hausdorff distance) of starshaped closed and bounded subsets of the considered Banach space which are at a positive distance from A. Analogous results for maximization problems are verified under the additional hypothesis that A is convex, as well as for approximation by elements of convex sets; when S is assumed to be convex then reflexivity can be dropped.

In [36] it was shown that for a normed linear space X the following holds: if $\mu$ is a regular (i.e., finite on compact sets, and compact inner regular and outer regular on Borel sets) Borel measure on X, then the nowhere dense closed bounded starshaped sets of measure zero are $\sigma$-dense in the closed bounded starshaped sets. The authors of [383] derived approximation results in normed spaces generated by certain starshaped cones. They first introduced these starshaped cones (which have nonempty kernel and are representable as unions of closed convex pointed cones whose intersection has interior points), and then they derived best approximation results with respect to closed sets in such normed spaces.

A famous Weierstrass theorem asserts that every continuous function on a compact set in d-dimensional space can be uniformly approximated by algebraic polynomials. The study of the same question for the important subclass of homogeneous polynomials containing only monomials of the same degree yields the conjecture that every continuous function on the boundary of convex centered bodies can be uniformly approximated by pairs of homogeneous polynomials. In [320] the recent progress on this conjecture is reviewed, and a new unified treatment of the same problem on starshaped domains is investigated. It is proved that the boundary of every centered non-convex starshaped domain contains an exceptional zero set so that a continuous function can be uniformly approximated on the boundary of the domain by a sum of two homogeneous polynomials if and only if the function vanishes on this zero set. Hence this approximation problem amounts to the study of these exceptional zero sets, and intersections of starshaped domains concerning this framework are studied, too.

More applied in nature, in [158] the approximation of starshaped surfaces in Euclidean 3-space in the spirit of certain spline functions was studied. Given a finite number of points in such a starshaped surface M, related minimization and triangulation procedures were used to get a starshaped surface $M^{*}$ suitably approximating M; the surface $M^{*}$ was constructed by means of a certain scalar-valued interpolant, which also gives rise to some error estimates. The authors of [23] studied the “consistent approximation” of a starshaped set S from a random sample of n points from S, e.g. with respect to the Hausdorff metric. They used an “estimator” defined as the union of balls centered at the sample points with a common radius which can be chosen in such a way that this estimator is also starshaped. These results are also related to statistical image analysis.

19.7 Applications of starshapedness in optimization

The notion of starshapedness also occurs in many papers discussing optimization and control theory, e.g. combined with suggestive examples or as a basic property of the considered domains or used functions. It is impossible to give a complete picture where starshapedness explicitely occurs in these disciplines. Thus, again we only select some results directly related to geometric properties of starshaped sets. Starshapedness of the considered domains is discussed first, followed by a brief discussion of a few results on (approximately) starshaped functions used in this framework.

Many practical problems can only be modelled as non-convex optimization problems so that it is natural and interesting to weaken convexity assumptions. Clearly, starshapedness means that in the definition of convexity one of the two variable endpoints of an arbitrary contained segment is fixed, and precisely this weakening is important for the relations between variational inequalities and optimization theory. E.g., Crespi et al. (see [145, 146]) made explicit use of the concept of starshaped domains to study certain scalar variational inequalities, scalar optimization problems and suitable generalizations, where also the existence of solutions lying in the kernels of the considered starshaped domains plays an important role.

A quasiconvex function is a real-valued function defined on an interval or a convex subset of a real vector space so that the inverse image of any set of the form $(-\infty ,a)$ is a convex set. This concept generalizes that of convexity and arises almost naturally when functions of one variable are involved, but differences occur when functions of several variables are studied. For this case it was shown in [147] that the existence of a solution of the investigated variational inequalities does not necessarily imply the quasiconvexity of the considered function but that the level sets of the function must be starshaped at a point which is a solution. Similar assumptions were made in [140]. In [15], quasiconvex mathematical programming problems having equilibrium constraints with locally starshaped constraint regions were studied. While usual necessary conditions of optimality become sufficient when the feasible region is starshaped and the ojective function has the pseudoconvexity property (which is in general stronger than quasiconvexity), the usual necessary condition may not be sufficient for the weaker quasiconvex case. The authors introduced a new necessary condition of optimality using the normal cone to the sublevel set instead of the subdifferential of the objective function, and it turned out that this is also sufficient in the case of quasiconvexity; local starshapedness of the constraint set is an important geometric tool in that paper. In [394] it was shown that locally starshaped domains can play an essential role in multiobjective programming, since semilocally convex functions have certain convex-type properties. Vector optimization problems with semilocally convex objective functions defined on locally starshaped sets were studied, the authors derived respective saddle point and Kuhn–Tucker conditions, and a related duality theory was developed.

In the paper [10] approximate starshapedness was investigated in connection with several basic types of subdifferentials. Based on vectorial definitions of the studied topics, their relations were considered also using results from [412]. In [10] the authors extended the concepts of approximate starshapedness and equi-subdifferentiability to the case of vector functions. They established relations between approximate efficient solutions of multiobjective optimization problems and solutions of associated vector variational inequalities for approximate starshaped vector functions.

Fang and Huang [183] used starshaped sets to investigate the well-posedness of vector optimization problems. Also related to vector optimization and using a generalized domination property, in [272] the lower convergence of minimal sets in sequences of starshaped sets in related optimization problems was discussed.

In [212] a class of abstract parabolic variational problems is studied, where the set of all admissible elements is closed and starshaped with respect to a ball. Due to this starshapedness type, the corresponding variational formulation is no longer a variational inequality, and the authors proved the existence of corresponding solutions and gave further results. The starshapedness assumption enables the use of a discontinuity property of the generalized Clarke subdifferential of the distance function.

All well known extremal principles for conformal mappings of simply connected regions yield mappings onto disks D. Using sub-norms it was shown in [401] that given an arbitrary starshaped region D as range, a corresponding extremal principle is valid if one replaces the ordinary modulus with a suitable positively homogeneous functional. In [118] this method, to approximate the conformal mapping of D onto the interior of a starshaped region, was generalized considering univalent harmonic mappings of D.

As announced, we only briefly mention a few results on starshaped functions. We recall that a function $f:[0,1]\rightarrow {\mathbb {R}}$ is said to be starshaped if for all a, $0\le a\le 1$, $f(ax)\le af(x)$ for $0\le x\le 1$. Hummel [276] presented a systematic, fundamental investigation of the definition of a multivalent starlike function. Six possible definitions were derived, each of which reasonably leads to a class of such functions. In [411] the author introduced a starshaped conjugate of a function f in such a way that the biconjugate of f coincides with the greatest lower semicontinuous starshaped function which minorizes f. Properties of this interesting duality relation were studied, and in [412] different types of subdifferentials of the difference of two approximately starshaped functions were investigated. Ubhaya [523] investigated a curve fitting problem involving the minimization of the distance from a funtion to a convex cone of functions whose domain is a partially ordered set. He applied his results to the approximation of starshaped functions. The author of [487] considered starshaped subsets of the complex plane and univalent functions on the unit disk which are geometrically starlike. The purpose of this paper is to exhibit a broad collection of such functions, where the geometrically interesting notions of annular and geometric starlikeness occur.

19.8 Further topics

In this subsection we want to mention some further isolated topics and results which refer to or use the concept of starshapedness.

As it was done with convex bodies, it is natural to also consider starshaped bodies with fractal boundaries as objects, which are interesting from the viewpoints of convex geometry and fractal geometry. In [263], the notion of fractal star body was introduced and studied regarding basic geometric properties. Also operations within the family of fractal star bodies were investigated, and it was shown how the fractal property limits the possible values of the topological dimension of their kernels. In view of fractal limit sets, the author of [213] studied an algorithm that modifies the well-known inversion in a circle for sets that are starshaped. He presented examples including the change of centers of the inversion circles, the change from inversion circles to squares and also to sets which are properly starshaped.

A planar set K is called a Kakeya set if a segment of length 1 can be rotated continuously inside K to return to its original position with its endpoints reversed. The Kakeya problem asks for the smallest area that Kakeya sets can have, and Besicovitch has shown that this area can be arbitrarily small (see the discussion in subsection G6 of [148]). Cunningham [149] continued Besicovitch’s investigations of the Kakeya problem, and one of his results says that any Kakeya set that is also starshaped cannot have area less than $\pi /108$; however, it is known that this is not the best value.

Inspired by the concept of random convex sets, for which the recourse to tools like support functions and Minkowski addition is common, the authors of [421] propagated the use of radial functions instead, and the study of the more general concept of random starshaped sets. They introduced the analogous concepts of expected value and variance, various further notions (such as mean directional length) and suggested also some comparative measures for centered starshaped sets. With a link to fixed point theorems (see our respective subsection), the authors of [37] proved some random fixed point theorems for asymptotically nonexpansive random operators defined on starshaped subsets of Banach spaces, hence obtaining a stochastic generalization of comparable results on convex sets. Also in the fields of probabilistic modelling and stochastic representations, the geometry of starshaped sets can be explicitly used. In [427], such representations were studied for geometrically described distributions. Based on known results, the author established a geometric disintegration method for deriving even starshaped distributions, whose basic properties and applications are discussed, as well as some necessary background from metric non-Euclidean geometry. Further on, in view of stochastic representations of correspondingly distributed random vectors, in [428] the authors investigated starshaped distributions whose shapes are based on topological boundaries of polyhedral sets.

Baire category results deal with the typical behavior of convex bodies in the following sense. We say that most convex bodies have a certain property if those convex bodies which fail to have that property form a meager or first category subset of the space of all convex bodies (i.e., a countable union of nowhere dense sets in this Baire space, see [233]). In many cases, the typical behavior of convex bodies turns out to be counterintuitive. Also the notions of starshaped sets and Baire spaces can be successfully combined; see, e.g., [34] treating kernels and [484] regarding the nearest point mapping. From [550] we learn that the set of all compact starshaped sets in Euclidean d-space, endowed with the Hausdorff metric, is a Baire space, that for most compact starshaped sets their kernels consist of precisely one point x, and that the directions in which they extend from x are dense in the unit sphere (see also below, the discussion of [155]). In [237] results from [550] were reproved, and the following was shown: Typical compact starshaped sets have non-$\sigma$-finite 1-dimensional Hausdorff measure, but they are still of Hausdorff dimension 1. The three surveys [233, 549, 552] discuss Baire categories in convexity and geometry, and they also discuss related results on starshaped sets and compact sets (see, in particular, Sect. 13 of the Handbook article [233]). In addition, we mention [422] (see our subsection on fixed point theory) and the papers [152, 153] (cited in [233] as manuscripts). [155] completes the work in the latter ones, and contains the following: In a real Banach space E endowed with the Hausdorff metric, let S(E) be the family of all nonempty, compact, starshaped subsets of E. Let $pr_{(X)}$ be the metric projection onto the set X which associates to each $a\in E$ the set of all points in X closest to a, and let A(X) be the ambiguous locus of X, i.e., the set of all $a\in E$ whose projection $pr_{X(a)}$ has positive diameter. In a complete metric space (that we have here) the complement of any set of the first Baire category is called a residual subset, and its elements are then called typical. The authors proved that in a strictly convex separable Banach space E of dimension at least two, a set A (as the union of all $X\in S(E)$ with ambiguous locus A(X) uncountable everywhere) is a residual subset of S(E). It was also proved that a typical element of S(E) has a kernel consisting of a single point and a set of extension directions dense in the unit sphere of E.

Also in potential theory the concept of starshapedness occurs explicitly. Novikov [399] called the following problem the “inverse problem of potential theory”: given a positive mass distribution $\phi$ on a bounded region M, try to determine M if the potential of $\phi$ in the neighborhood of infinity is known. In general, this problem has no unique solution, but if $\phi$ has density 1 on M and M has smooth boundary and is starshaped with respect to the origin, then M is uniquely determined by the exterior potential of $\phi$. These considerations were continued in [495], see also Chapter 3 of the monograph [278]. Margulis [364] proved that for the Newtonian potential of a starshaped domain of constant density $\varepsilon$, there is, for any $\varepsilon _{0}<\varepsilon$, a starshaped domain of density $\varepsilon _{0}$ creating the same outside Newtonian potential. Results of this type were also comprehensively discussed in [278].

Bifocal curves in the Euclidean plane are defined by having, with respect to two focal points, constant sums (ellipses), absolute differences (hyperbolas), products (Cassini curves) or ratios (Apollonius circles) of distances. These concepts naturally extend to higher dimensions, real Banach spaces and, in the case of constant sums or products, to finitely many focal points. The authors of [280] studied such multifocal curves and surfaces in real vector spaces defined by gauges (i.e., by “generalized norms”, not necessarily satisfying the symmetry axiom). In the case of multifocal Cassini curves and surfaces starshapedness plays a significant role. E.g., a multifocal Cassini curve is starshaped in this framework if its radius is sufficiently large, and for sufficiently small radius it consists of starshaped components around the focal points.

The work [384] interprets symmetric stable laws using also starshaped sets and recent results from convex geometry to come up with new probabilistic results for multivariate symmetric stable distributions.

Finally, we mention a few references from infinite dimensional analysis, where starshaped sets were explicitly taken into consideration (However, we note that such results have already been given in other parts of our paper, e.g. in the subsections on fixed point theory and approximation theory.). Starshaped bodies are also interesting in analysis because, among other things, they are related to polynomials and smooth bump functions as well as for their geometrical properties, also yielding topological observations. E.g., Klee first gave a topological classification of convex bodies in a Hilbert space, and this was generalized for every Banach space with the help of Bessaga’s non-complete norm technique. In [18] the question to what extent known results on the topological classification of convex bodies can be extended to starshaped bodies was studied; one of the results follows the mentioned Bessage–Klee scheme (see [18] for a historical discussion) regarding the topological classification of convex bodies, and another one gives a new classification scheme in terms of the homotopy type of the boundaries of starshaped bodies (holding in full generality provided the considered Banach space is infinite-dimensional). Since every convex body is starshaped, one may ask whether the famous James’ theorem (characterizing reflexivity) remains true if the word “convex” in this theorem is replaced by “starshaped”. The authors of [17] disproved this conjecture with a Hilbert-space construction.

It is well-known that in a finite-dimensional Banach space there is no continuous retraction from the unit ball onto the unit sphere. This is no longer true in infinite dimensions; for every infinite-dimensional space there exists a Lipschitz retraction from the unit ball onto the unit sphere. The authors of [16] showed that this result can be sharpened. They proved that the boundary of a smooth Lipschitz bounded starlike body in an infinite-dimensional Banach space is smoothly Lipschitz contractible; furthermore, the boundary is a smooth Lipschitz retract of the body. In the expository paper [19], the authors contributed to an intriguing problem: provide a characterization of those infinite-dimensional Banach spaces which admit diffeomorphism deleting points with a bounded support. Several sufficient conditions for a Banach space to have diffeomorphism deleting points were provided. All of these conditions are of geometric flavour and involve the existence of certain families of smooth starshaped bodies. Also the “Four Bodies Lemma” was obtained: given four smooth and radially bounded starshaped bodies with the same characteristic cone and such that every body is contained in the interior of the following one, there is a diffeomorphism carrying the second body onto the third one and being the identity inside the first one and outside the fourth one.